Package per version storage

[zimeon]

In cases where there are many small files in an object or where the storage infrastructure is not efficient at handling many files, it is useful to package files using a technology such as ZIP. This is addressed for the whole object in #10. However, packaging the whole object as a ZIP/Tar etc. breaks the idea of immutability of version data. One could instead package the inventory and content for each new version as a new ZIP file.

[zimeon]

This could be something along the lines of:

[object root]
    ├── 0=ocfl_object_1.0
    ├── inventory.json
    ├── inventory.json.sha512
    ├── v1.zip
    ├── v2.zip
    └── v3.zip

this still leaves three potentially small files per object (though inventory.json might not be) but avoids any small files in the object's contents appearing alone in storage, while each v#.zip is immutable.

[rosy1280]

potentially a sub-use case of #39

[ThomasEdvardsen]

Hello everybody! The National Library of Norway is in the process of installing a new bit repository (HPSS) that can hold 44 PB of data. In this context, we are considering using OCFL to organize our data packages.

So far, OCFL looks very good, but we are dependent on ZIP per version storage #33 being resolved to be able to use OCFL. This is because we want to limit the number of files so that it becomes more efficient to store/retrieve data from HPSS.

I reckon this needs to be solved using an object extension? Do you have any thoughts on how this can be implemented?

[ThomasEdvardsen]

We have begun to think about how this can be implemented based on our needs. This is a very immature first proposal for a new object extension.

We would like to discuss the following:

1. Whether or not to include a full path to the archived files.
2. Whether the archive files should be placed on the object's root, or in separate version folders.
3. Whether a version can consist of more than one archive file.

Arguments for allowing more than one file for each version:

• Split very large files into smaller files
• Make it easier to transfer large objects, so you do not have to restart completely if a file fails while uploading.

What are your initial thoughts?

[object root]
├── 0=ocfl_object_1.0
├── extensions/
│   └── nnnn-archived-versions/
│       ├── archived-versions.json
│       └── archived-versions.json.sha512
├── inventory.json
├── inventory.json.sha512
├── v1/
│   ├── v1-1.zip
│   ├── v1-2.zip
│   └── v1-3.zip
├── v2/
│   └── v2-1.zip
└── v3/
    ├── v3-1.zip
    └── v3-2.zip  

Example content of archived-versions.json

{
  "id": "zipped_updates_three_versions_one_file",
  "versions": {
    "v1": {
      "created": "2019-01-01T02:03:04.000Z",
      "archiveAlgorithm": {
        "mime": "application/zip",
        "pronomId": "x-fmt/263"
      },
      "digestAlgorithm": "sha512",
      "files": {
        "0675bdf376e92e9994612c33ea255b12f7": {
          "filePath": "/hpss/storage-root-01/1ec1/8fe/5cd/zipped_updates_three_versions_one_file/v1/v1-1.zip",
          "fileSize": 133410430
        },
        "0675b1ff76e92e9994612c33ea255b12f7": {
          "digestHex": "/hpss/storage-root-01/1ec1/8fe/5cd/zipped_updates_three_versions_one_file/v1/v1-2.zip",
          "fileSize": 520430330
        },
        "067ab1f376e92e9994612c33ea255b12f7": {
          "digestHex": "/hpss/storage-root-01/1ec1/8fe/5cd/zipped_updates_three_versions_one_file/v1/v1-3.zip",
          "fileSize": 8353634100
        }
      }
    },
    "v2": {
      "created": "2020-02-02T02:03:04.000Z",
      "archiveAlgorithm": {
        "mime": "application/zip",
        "pronomId": "x-fmt/263"
      },
      "digestAlgorithm": "sha512",
      "files": {
        "5b23ffdf2709bf393a7d8883fcdf583980": {
          "filePath": "/hpss/storage-root-01/1ec1/8fe/5cd/zipped_updates_three_versions_one_file/v2/v2-1.zip",
          "fileSize": 42644244
        }
      }
    },
    "v3": {
      "created": "2021-03-03T02:03:04.000Z",
      "archiveAlgorithm": {
        "mime": "application/zip",
        "pronomId": "x-fmt/263"
      },
      "digestAlgorithm": "sha512",
      "files": {
        "88492082026f1a3a1c0637d6bd02214dd6": {
          "filePath": "/hpss/storage-root-01/1ec1/8fe/5cd/zipped_updates_three_versions_one_file/v3/v3-1.zip",
          "fileSize": 8743244
        },
        "3a1c0637d6bd02214dd62c5c19ee8d4bbf": {
          "digestHex": "/hpss/storage-root-01/1ec1/8fe/5cd/zipped_updates_three_versions_one_file/v3/v3-2.zip",
          "fileSize": 892345
        }
      }
    }
  }
}

[zimeon]

I support the idea that any solution for packaged content should include support for multiple packages in a version (v1-1.zip, v1-2.zip etc) so that it could address right-sizing both groups of small files and segmenting large files #40)

I think the biggest question is where one describes the logical files vs the physical files (packages). I lean toward having the inventory describe the physical files and thus providing the infrastructure for preservation/fixity/transfer, and then create some new way to describe the logical object content in a way that doesn't make those other processes too cumbersome in the case of objects with large numbers of files. This would potentially mean significant changes in the state ideas that currently map physical to logical files in an object version.

[pwinckles]

I support the idea that any solution for packaged content should include support for multiple packages in a version (v1-1.zip, v1-2.zip etc) so that it could address right-sizing both groups of small files and segmenting large files #40)

If the spec adds support for zipped versions, does it necessarily need to make special mention of split zips, which are already part of the zip spec?

[julianmorley]

If the spec adds support for zipped versions, does it necessarily need to make special mention of split zips, which are already part of the zip spec?

I'd lean towards 'yes', based on our experiences doing something similar with Preservation Catalog. At the end of the day OCFL tracks files and their checksums. It doesn't know, for example, that a .zip file contains information that points to other zip segments, and we want a human reading the manifest to be able to see that the version directory should contain 10 files (file.zip, file.z01, file.z02, ... etc) without having to wonder if the single file.zip in the directory is meant to be just one file or the first in a series of zip segments.

My early guess is that, in OCFL v2, we'll expand inventory.json to be able to say "this version of this physical representation of this object is stored as a zip archive with these parameters", and list out all the zip parts and their checksums, together with a sidecar file that lists all the files in those zips (and their checksums).