ACQ ID package_images.py, package_rsync.py, and transfer_rsync.py #56
Conversation
|
new streams folder in destination not being created and stream files not being moved. |
| return parser.parse_args() | ||
|
|
||
|
|
||
| def run_rsync(source: Path, dest: Path, quiet: bool = None) -> None: |
There was a problem hiding this comment.
currently rsync seems to be unable to transfer some files. This will throw an error but won't indicate what files aren't transferred or why. There may be additional rsync flags to add to this function.
There was a problem hiding this comment.
add a logger to check the log for what didn't transfer
| args = parse_args() | ||
|
|
||
| base_dir = pb.create_package_dir(args.dest, args.carrierid) | ||
| pb.create_bag_in_objects(args.payload, base_dir, args.log, "rsync") |
There was a problem hiding this comment.
Currently the script creates a packages with no metadata folder, an object folder with an empty data folder. payload is not being moved but manifest is generated. Note in testing I used the rsync log generated by transfer_rsync.py.
Issues may be connected to an OS Error thrown by os_rename in move_files in move_data_files in package_base.py, this error is thrown because of an .fseventssd file.
There was a problem hiding this comment.
Currently the script creates a packages with no metadata folder, an object folder with an empty data folder. payload is not being moved but manifest is generated. Note in testing I used the rsync log generated by transfer_rsync.py.
Issues may be connected to an OS Error thrown by os_rename in move_files in move_data_files in package_base.py, this error is thrown because of an .fseventssd file.
Current error:
% poetry run python3 src/digarch_scripts/package/package_filetransfer.py --payload '/Volumes/BJF MEMOS' --carrierid ACQ_5181_541623 --dest '/Volumes/DigArchDiskStation/Staging/ingest/fileTransfers' --log '/Users/djonathan/Desktop/ACQ_5181_541623_rsync.
log'
/Volumes/BJF MEMOS/.fseventsd /Volumes/DigArchDiskStation/Staging/ingest/fileTransfers/ACQ_5181/ACQ_5181_541623/objects/data/.fseventsd
Traceback (most recent call last):
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_filetransfer.py", line 34, in
main()
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_filetransfer.py", line 28, in main
pb.create_bag_in_objects(args.payload, base_dir, args.log, "rsync")
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 260, in create_bag_in_objects
create_bag_in_dir(
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 236, in create_bag_in_dir
move_data_files(paths, bag_dir)
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 214, in move_data_files
return move_files(data_paths, pkg_dir, "data")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 197, in move_files
move_file(file_path, pkg_dir, dest)
File "/Users/djonathan/Documents/GitHub/digarch_scripts/src/digarch_scripts/package/package_base.py", line 190, in move_file
file_path.rename(new_file_path)
File "/usr/local/Cellar/python@3.12/3.12.5/Frameworks/Python.framework/Versions/3.12/lib/python3.12/pathlib.py", line 1363, in rename
os.rename(self, target)
OSError: [Errno 18] Cross-device link: '/Volumes/BJF MEMOS/.fseventsd' -> '/Volumes/DigArchDiskStation/Staging/ingest/fileTransfers/ACQ_5181/ACQ_5181_541623/objects/data/.fseventsd'
No description provided.