Major inefficiency when using snbk over SSH

[tyami94]

Greetings,

When using snbk to transfer backups over SFTP/SSH, it seemingly creates a new ssh session for every snapshot contained on the remote storage device. This makes snbk take an eternity when you have a large amount of snapshots (~1000+). Local transfers are extremely fast in comparison.

Is there a way to let snbk scan the snapshots without creating a new SSH session for each one?

Thanks,
tyami94

[aschnell]

I am aware of that problem. Unfortunately snbk needs the "Received UUID" for all snapshots and that is only available in the output of "btrfs subvolume show" which only works for a single snapshot.

I see two solutions:

1. Somehow run several commands within one ssh connection. That is ugly as far as I remember.
2. Extend "btrfs subvolume list" to include the "Received UUID". This likely requires the JSON output for that command which has been experimental for years by now.

[aschnell]

See also kdave/btrfs-progs#941. Whether that is all that would be required I cannot say. The implementation in snbk has changed meanwhile since the path handling with the btrfs subvolume list command is not straightforward.

[tyami94]

By sheer coincidence, today I saw that incus accomplishes something like this when doing a incus copy --refresh by doing btrfs subvolume list -u -R <mountpoint>. Would that work here? Some parsing may be needed, but it prints all subvolume uuids in one go.

Sample output snippet:

ID 3398 gen 55292 top level 259 received_uuid e9fbcfca-5bf8-8947-aa32-0ac0570c6fe1 uuid 99e3c8ef-4574-bd42-ae29-478b8afb1eb4 path @snapper/pc/home/25/snapshot
ID 3399 gen 55300 top level 259 received_uuid 6d03ba35-22ee-4f4f-bd5b-411aebf66f70 uuid 47cc999e-8c66-364b-9ad3-1e1de3e2f68b path @snapper/pc/home/1041/snapshot
ID 3400 gen 55313 top level 259 received_uuid 17878407-6973-7344-8e24-d909b3807d86 uuid 1e737e95-5f54-cb40-b896-02ef6fcfabb7 path @snapper/pc/home/2385/snapshot
ID 3401 gen 55316 top level 259 received_uuid f275aff4-4965-324f-ab7a-ff98b38ac831 uuid b4f4f1ac-7449-6343-a788-ed4b8bb50a1d path @snapper/pc/home/2529/snapshot
ID 3402 gen 55319 top level 259 received_uuid 886041a9-fde3-9e49-a949-d16b2403d3d6 uuid 23bae4f1-85a3-3445-b26b-41fa829a488e path @snapper/pc/home/2721/snapshot
ID 3403 gen 55322 top level 259 received_uuid db55be02-23a9-0b41-b7b8-55b9c0cfbbb1 uuid ef7b41b0-ae38-c648-ba87-76352c422792 path @snapper/pc/home/3057/snapshot

[aschnell]

That output does not include the read-only flag (for which I made the PR one year ago). Apart from that the path handling using btrfs subvolume list is strange and caused several bugs (so btrfs subvolume list is currently not used by snbk.