OCFL Object Forking

[lnielsen]

In Zenodo we have a use case where we have two layers of versioning. A user can publish a dataset on Zenodo which will get a DOI. A new version of the dataset can be published by the user, which will get it a new DOI. This way a DOI always point to a locked set of digital files. Occasionally, however, we have the need to change files of an already published dataset with a DOI (e.g. user accidentally included personal data in the dataset and discovered 2 months later). Essentially this means we have two layers of versioning in Zenodo, which I'll call

• Versioning (each version get's a new DOI - at the repository level each version is a separate record)
• Revisions (edits to a single version - at the repository level this a single record).

In the Zenodo of case, our need for deduplication is essentially between versions, because' that's where a user may only add 1GB to a 100TB dataset.

They way we have thought about mapping Zenodo to OCFL is that each DOI is essentially an OCFL object. Because OCFL object only supports deduplication within an OCFL object, and not between OCFL objects, nor does OCFL allow symlinks, then we cannot do this deduplication.

Example

Imagine these actions:

1. Publish first version 10.5281/zenodo.1234 with two very large (let's just say 100TB to exaggerate) files: data-01.zip and mishap.zip
2. Publish new version 10.5281/zenodo.4321 with one new file: data-02.zip (files is thus: data-01.zip and data-02.zip).
3. Remove mishap.zip from 10.5281/zenodo.1234

The OCFL objects would be:

[10.5821/zenodo.1234]
    ├── 0=ocfl_object_1.0
    ├── inventory.json
    ├── inventory.json.sha512
    ├── v1
    │   ├── inventory.json
    │   ├── inventory.json.sha512
    │   └── content
    │       ├── data-01.zip
    │       └── mishap.zip
    └── v2
        ├── inventory.json
        ├── inventory.json.sha512
        └── content


[10.5821/zenodo.4321]
    ├── 0=ocfl_object_1.0
    ├── inventory.json
    ├── inventory.json.sha512
    └── v1
        ├── inventory.json
        ├── inventory.json.sha512
        └── content
            ├── data-01.zip (duplicatied 100TB of data!!!)
            └── data-02.zip

What I would like is not having to duplicate data-01.zip in 10.5821/zenodo.4321 OCFL object?

Is there a solution for this in OCFL, or a different way to construct our OCFL objects that could support this?

[ahankinson]

Depending on the underlying storage system you have in place your disks may be doing this deduplication transparently.

How do you handle this case at the moment?

[lnielsen]

Our storage system doesn't handle it (it's http://eos.web.cern.ch with some 400PB of disk space). Essentially if e.g. hard symlinks where allowed, some system operating on the OCFL objects probably even wouldn't know that it's deduplicated.

The problem is with either the requirement on not using hard links:

Hard and soft (symbolic) links are not portable and must not be used within 
OCFL Storage hierarchies. A common use case for links is storage deduplication. 
OCFL inventories provide a portable method of achieving the same effect by using 
digests to address content.

or with the assumed linear versioning.

Note, I've been discussing this with @neilsjefferies IRL as well.

[zimeon]

I think this is a special case of reference to a file/datastream external to an OCFL object. This is partly discussed in #27 but I created #35 to separate out the idea of an external file. IMO this is out-of-scope for v1 but we should revisit when considering scope of v2.

[lnielsen]

A reference to file/datastream in another OCFL object could solve the issue. My general thinking here is that a reference to file/datastream anywhere is not a good idea, but that instead it should be constrained to the OCFL storage root.

I fully understand that you want to get v1 out the door. Just know that this is kind of a show stopper for using OCFL for us, so a quick v2 release afterwards would be much appreciated. We have 1.4 million OCFL objects and 300TB of data to write, so I'd prefer not having to rewrite them :-) Obviously, I'm happy to help out, in case there's anything I can do to accelerate it.

[awoods]

Thanks, @lnielsen.
Taking a step back, for clarification, what is the rationale for your decision of:

They [sic] way we have thought about mapping Zenodo to OCFL is that each DOI is essentially an OCFL object.

It is conceivable that separate versions of a single OCFL Object could have their own DOIs.

[lnielsen]

@awoods It's related to the two levels of versioning that I call versioning and revisions and that they can happen in different sequences (e.g. v1.0, v2.0, v1.1 or v1.0, v1.1, v2.0).

I'll try to see if I can give a clear example 😄 and of course don't hesitate to let me know if there's something obvious that I just haven't seen.

If I change my initial example to use a single OCFL object it would look like this (after the three actions):

[multi-doi-object]
    ├── 0=ocfl_object_1.0
    ├── inventory.json
    ├── inventory.json.sha512
    ├── v1               # 10.5821/zenodo.1234
    │   ├── inventory.json
    │   ├── inventory.json.sha512
    │   └── content
    │       ├── data-01.zip
    │       └── mishap.zip
    ├── v2               # 10.5821/zenodo.4321
    |   ├── inventory.json
    |   ├── inventory.json.sha512
    |   └── content
    |       └── data-02.zip
    └── v3               # 10.5821/zenodo.1234
        ├── inventory.json
        ├── inventory.json.sha512
        └── content

So far so good. I've managed to represent the changes in an OCFL object.

Now let's switch the order of actions from 1, 2, 3 to 1, 3, 2. My OCFL object would instead look like this:

[multi-doi-object]
    ├── 0=ocfl_object_1.0
    ├── inventory.json
    ├── inventory.json.sha512
    ├── v1               # 10.5821/zenodo.1234
    │   ├── inventory.json
    │   ├── inventory.json.sha512
    │   └── content
    │       ├── data-01.zip
    │       └── mishap.zip
    ├── v2               # 10.5821/zenodo.1234
    |   ├── inventory.json
    |   ├── inventory.json.sha512
    |   └── content
    └── v3               # 10.5821/zenodo.4321
        ├── inventory.json
        ├── inventory.json.sha512
        └── content
            └── data-02.zip

So far so good as well. I've achieved deduplication of the big file.

The problem I see with this structure is that it's non-trivial/non-intuitve to find the latest state of a specific DOI, and thus requires interpretation on top of OCFL in order to be understandable. The reason for using OCFL in the first place, is to have an self-evident structure that requires no other knowledge than OCFL.

Similarly, I could also imagine hacks to make things work like writing a completely new OCFL object and deleting the old one. But then performance would be an issue.

[julianmorley]

Hi @lnielsen! We have this issue at Stanford ("In the Zenodo of case, our need for deduplication is essentially between versions, because' that's where a user may only add 1GB to a 100TB dataset.") and don't have a perfect solution, but have approached it in two ways:

1. If a user inadvertently accessions personal info in an object, we have to purge the entire object from SDR and re-accession it with the same identifier and cleaned content. It's a pain to do (deletes are hard by design!) but it's the only way to truly purge sensitive data from an existing object.

2. For incremental additions to large datasets, we try to break the dataset into smaller logical pieces (still in zip files, but not one big zip for the entire dataset). This also requires some curatorial intervention but we've found that it provides a slightly better user experience, especially for downloading that dataset. It also gives us a chance that future dataset changes impact only a handful of prior zips (or maybe even non at all!), allowing us to leverage the incremental diff feature of Moab (which OCFL also implements).

[neilsjefferies]

Copies from use-cases...general musing...so not completely thought out,

I can imagine a minor modification to the inventory that adds "inherits from ObjectID" type sections to the manifest. The digests that follow identify paths in other OCFL object(s). Other than that nothing else needs to change. When copying an object, parsing the manifest tells you which additional objects it has dependencies on. It would permit version forking and inter-object deduplication. this does mean that if object versions are not stored as a single units then each version has a new ID - this is not necessarily a bad thing.

...this might also be adapted to include "Inherits from external_storage_path" in some form.

[neilsjefferies]

• In spec, must be explicit that a newly created object is “inheriting” from an existing object within the same storage root (thus while we violate completeness for an object, we will still have completeness for a storage root)
  -- However, if in v2, we come up with a mechanism for multiple storage roots making one repo, we should support inheritance from any of the roots making up the repo (which might mean one has a general root reference mechanism.... But with a dire warning to be prudent)
• We do not want to support an OCFL object referencing files from across multiple other objects. This is to prevent validation loops. Thus:
  -- Inheritance can only happen when a new object is instantiated, an entire manifest block of the source object is inherited
  -- We will only support inheritance from a single object
  -- The new object must clearly reference the version of the original object from which it is inheriting
• Implementation notes:
  -- Validation failure if original object subsequently is purged
  -- Validation of new object must also validate the original (parent) object. This is a reason why we have the same storage root requirement.
  -- validator must have guard rails to catch loops.