Skip to content

Feature: Check if a bag exists in a local storage #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HafeezOJ merged 44 commits into from
Aug 22, 2025
Merged

Feature: Check if a bag exists in a local storage #121

HafeezOJ merged 44 commits into from
Aug 22, 2025

Conversation

@HafeezOJ
Copy link
Collaborator

@HafeezOJ HafeezOJ commented Aug 7, 2025 •
edited
Loading

Description

This PR enhances ReBACH to check the path to local storage for duplicate bags. It also handles where to check for duplicate bags for different workflow settings, which could be to upload or not to upload to a remote storage. Configuration fields that hold different storage locations are updated to the current specifications in line with OAIS model.

See #118

Documentation Update

  • I have updated README.md and other relevant documentation

Implementation Notes

  • Special functions are created in Utils.py to handle checking paths that are local to ReBACH for duplicate bags.
  • Counts of already preserved items in archival staging storage is reported in logs message
  • A flag (--check-staging-storage) can now be set to check Wasabi for duplicate bags. Wasabi will also be checked for duplicate bags if the workflow is configured to upload to a remote storage location.
  • Folder name of a package will be reused if it exists in the ingest staging storage from a previous run and no new version is generated.
  • Configuration fields for storage locations are updated to the current specifications in line with OAIS model
  • AP Trust configuration has been moved from .env.ini configuration file in ReBACH to .toml configuration file in bagger.
  • Terms for storage locations are updated in log messages
  • Updated ReBACH and Bagger READMEs

HafeezOJ added 24 commits July 31, 2025 17:09
@HafeezOJ
Check local path, Retrieve package name from local path, Format version
ab58820
@HafeezOJ
Use format version, reuse folder name if exists
7df7657
@HafeezOJ
Reuse folder name if exists
26e0025
@HafeezOJ
Format collection version number
6063ee3
@HafeezOJ
Always have a folder name for package
81341d1
@HafeezOJ
Add flag to turn on checking remote staging storage
e7f23e1
@HafeezOJ
Document levels of storage check
59e62a4
@HafeezOJ
Consider check-remote-staging flag when checking storages
bf4c8fc
@HafeezOJ
Add a function to check if packages are uploaded to remote storage
637a2e7
@HafeezOJ
Skip item if upload_to_remote is true
1416b20
@HafeezOJ
Remove quotes from path
769b4fe
@HafeezOJ
Do not check duplicate bags if no upload
343e9c5
@HafeezOJ
Skip duplicate items based on workflow
22d82d0
@HafeezOJ
Fix double counting for versions
dfceea5
@HafeezOJ
Change storage nomenclature
362b6dc
@HafeezOJ
Add count of item versions preserved in archival staging storage
904b8dd
@HafeezOJ
Report counts in alternative archival staging storage depend on flag …
fbdfd54 
…and workflow
@HafeezOJ
Use OAIS term for storage configuration
9531599
@HafeezOJ
Update Bagger README
bb78ba7
@HafeezOJ
Update ReBACH README
52dd060
@HafeezOJ
Update Description of ReBACH Log Messages
3301ec6
@HafeezOJ
Change output_dir to archival_staging_storage in all files
856dc69
@HafeezOJ
Update ReBACH and Bagger READMEs with behaviour of remote staging sto…
d1dd67c 
…rage check
@HafeezOJ
Style flags
23c0d57
@HafeezOJ HafeezOJ linked an issue Aug 7, 2025 that may be closed by this pull request
Feature: Check if a bag exists in a local storage #118
Closed
4 tasks
@HafeezOJ HafeezOJ changed the title Feature check if a bag exists in a local storage Feature: Check if a bag exists in a local storage Aug 7, 2025
@HafeezOJ HafeezOJ self-assigned this Aug 7, 2025
@HafeezOJ HafeezOJ marked this pull request as ready for review August 7, 2025 06:27
@HafeezOJ HafeezOJ requested a review from zoidy August 7, 2025 06:29
zoidy
zoidy requested changes Aug 11, 2025
View reviewed changes
Copy link
Collaborator

@zoidy zoidy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a bug in the bagger code that erroneously detects that a bag is already in archival staging storage or archival storage. Integration.py L123

@HafeezOJ
Copy link
Collaborator Author

HafeezOJ commented Aug 14, 2025 •
edited
Loading

The bug is due to using the same storage location for both ingest_staging_storage and archival_staging_storage. I enforced using separate locations from both storage locations.

@zoidy
Copy link
Collaborator

zoidy commented Aug 19, 2025

There is a bug in the bagger code that erroneously detects that a bag is already in archival staging storage or archival storage. Integration.py L123

Resolved #121 (comment)

@zoidy zoidy self-requested a review August 19, 2025 22:37
zoidy
zoidy requested changes Aug 20, 2025
View reviewed changes
zoidy
zoidy previously approved these changes Aug 22, 2025
View reviewed changes
@zoidy zoidy dismissed their stale review August 22, 2025 14:05

#121 (comment) remains unresolved

@zoidy zoidy self-requested a review August 22, 2025 15:54
@HafeezOJ HafeezOJ merged commit 81e9ab6 into main Aug 22, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zoidy zoidy zoidy approved these changes

Assignees

@HafeezOJ HafeezOJ

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feature: Check if a bag exists in a local storage

3 participants

@HafeezOJ @zoidy