-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Description
Why It Matters
- Multi-cloud migrations, scheduled backups, and CI/CD deployments need a “make dest look like source” primitive instead of manual
putloops. - Post-transfer validation (hash / size comparison) is table stakes for preventing silent corruption; competitors like rclone expose
sync+checkas core commands. - Having both commands lets users follow a reliable “sync → check → promote” workflow inside one CLI.
Feature Requirements
storify sync <source> <dest>
- Walk source and destination simultaneously, skip identical files (size+mtime or hash), upload/overwrite changed files, and optionally delete extra destination files (
--delete=never|during|after). - Optional knobs:
--dry-run,--concurrency N,--include/--exclude pattern,--track-renames,--create-empty-src-dirs. - Example:
storify sync ./datasets oss://team-data/datasets --delete=after --dry-run
storify check <source> <dest>
- Compare source/dest without modifying either side. Report matches, differences, and missing entries; support
--one-way,--download(byte-for-byte when hashes are unavailable), and report files (--differ,--missing-on-src, etc.). - Example:
storify check ./datasets oss://team-data/datasets --differ differ.log --missing-on-dst missing.log
Implementation Sketch
- CLI wiring: add
sync/checkmodules undersrc/cli, parse flags, and reuse the existingContext+Operatorsetup. - Dual traversal: build a comparer that iterates source/dest listings (via OpenDAL
Lister), producing diff records (Match,SrcOnly,DstOnly) to feed sync or check pipelines. - Sync executor: for
Matchentries, decide whether to copy; queue deletes based on--deletemode; reusestorage::operations::{get, put, delete}for actual transfers. - Check reporter: implement a lightweight reporter (stdout or file) emitting
= / + / - / *markers or JSON; optionally fall back to streaming comparison when hashes are missing. - Testing: extend
tests/behaviorwith fixtures covering added/modified/deleted/missing cases on the localfsbackend and at least one S3-compatible backend.
Success Criteria
storify synccan mirror a directory tree, respecting delete modes and skipping unchanged files.storify checksurfaces differences with clear exit codes;sync+checkare documented in README/CLI help.- Behaviour tests prove that new commands work end-to-end for local + remote backends.
Metadata
Metadata
Assignees
Labels
No labels