irmin-pack: extend GC to unlimited history stores

[metanivek]

A list of things to-do for adding a lower layer that can be used to archive data instead of deleting it during a GC.

Configuration

• add lower_root configuration value @metanivek irmin-pack: add lower_root config for lower layer #2177
• V5 version and any updates to upper control file @metanivek irmin-pack: add V5 version #2184
  • add volume_num to control file

Volumes

• sparse file abstraction @art-w irmin-pack: introduce Sparse_file abstraction #2172
  • appendable sparse file @art-w irmin-pack: appending into the sparse file #2192
• control file for volumes @metanivek irmin-pack: introduce volume control file #2180
• index volumes in lower layer when opening a store @metanivek irmin-pack: introduce lower layer + volumes #2183
  • only needs control file
  • basic code for volumes and volume management
  • rw/ro volumes; appendable volume
• create empty volume @metanivek irmin-pack: introduce lower layer + volumes #2183
  • empty: zero byte mapping and data; no control file.
• load lower/volumes when opening store @metanivek irmin-pack: initial integration of lower into store api #2188
• add store API for creating empty volume @metanivek irmin-pack: initial integration of lower into store api #2188
• update GC to archive to a volume (https://hackmd.io/jRgzxlL4Qry6SD3k4A_Z2A?view#Updates-to-GC-process)
  • update GC API @Firobe irmin-pack: introduce shallow GC API changes, V0 GC compatibility #2190
  • lower archival @Firobe irmin-pack: GC adds data to lower #2202
• update read to use volumes @metanivek irmin-pack: implement reading from lower #2198
  • https://hackmd.io/jRgzxlL4Qry6SD3k4A_Z2A?view#Updates-to-data-read-process
  • limit open file descriptors
• handle cleanup scenarios for crashes @metanivek irmin-pack: add gc cleanup for lower layer #2222
  • https://hackmd.io/jRgzxlL4Qry6SD3k4A_Z2A#Handling-crashes-during-GC
• remove orphaned commits during GC @art-w irmin-pack: archive only reachable objects #2215

Migration

• GC supports V0 objects in reachability algorithm @Firobe irmin-pack: introduce shallow GC API changes, V0 GC compatibility #2190
• Actual migration (move prefix as a big chunk, automatically create new empty volume) @art-w irmin-pack: fast migration to lower by copy #2205

Miscellaneous

• from irmin-pack: introduce volume control file #2180 (comment)

  • update volume to V5 when we bump the pack store version @metanivek irmin-pack: add V5 version #2184
  • make a more specific error for Corrupted_control_file so that we can know which control file is at fault. This can likely be as simple as giving a string that we put a message into. @metanivek irmin-pack: add context for corrupted control file #2224
  • add specific error for when the length is longer than a page size. Currently, this is also Corrupted_control_file but it seems better to be specific with this error. Also, we should add path information to the error.
    • would also be solved by irmin-pack: Use temp control files upon writing #2206 if its benchmarks are ok @clecat

• from irmin-pack: initial integration of lower into store api #2188 (comment)

  • to properly support fresh, we need to recursively delete the lower root's volumes irmin-pack: implement fresh for lower layer #2238
  • should we return an error when add_volume is called an no lower exists? @metanivek irmin-pack: adding a volume w/o lower is an error #2199
  • there is a placeholder for a future test when we have proper control file writing for volumes @metanivek irmin-pack: test store re-open after add volume #2207

• add classification for volume files @metanivek irmin-pack: add classification for volume files #2201

• (tentative) switch from using mmap to lazy loading data for mapping file @art-w irmin-pack: no mmap syscall #2232

Follow-up PRs

• @metanivek irmin-pack: test data locality for volumes #2212
• @metanivek irmin-pack: fix nested lower root creation #2219
• @adatario irmin-pack.unix: Use asynchronous unlink when cleaning up chunks in Gc. #2221

[metanivek]

Shipped as a part of 3.7.