CLI cannot process multiple passed files with the same name

[bmwiedemann]

Version information:

go-ipfs version: 0.9.0
Repo version: 11
System version: amd64/linux
Golang version: go1.16

Description:

For https://github.com/bmwiedemann/ipfs-iso-jigsaw/blob/prehashnodirs/prehash.py#L53 I am trying to add files without directories into ipfs and noticed that directories are not printed in output, but what is worse, files are silently not added (or at least not reported)

mkdir -p /tmp/x/{1,2}
echo x > /tmp/x/1/x
echo y > /tmp/x/2/x
ipfs add -n --raw-leaves -- /tmp/x/1/x /tmp/x/2/x
added bafkreib3wkv3nhv3e7574y6hmolcjrxmlyzrxba2lpemh26bbojil2iio4 x

[aschmahmann]

Thanks for the report, it looks like the CLI command parser (https://github.com/ipfs/go-ipfs-cmds/blob/4ade007405e5d3befb14184290576c63cc43a6a3/cli/parse.go#L239) bundles up all the files into a map keyed by the file name (not path, inode, etc.) of the files leading to the described issue.

Perhaps just keying on path name instead of file name (i.e. removing https://github.com/ipfs/go-ipfs-cmds/blob/4ade007405e5d3befb14184290576c63cc43a6a3/cli/parse.go#L335) would do the job.

[bmwiedemann]

I tested that L335 patch and indeed, it helps to add both files and also helps to get distinct path names in the output:

ipfs add -n --raw-leaves -- /tmp/x/1/x /tmp/x/2/x
added bafkreidtzm4frjuhvbeuzizsgbjqcyuc6pnnhhkcz5rmuttz3wrkvr6zvq /tmp/x/1/x
added bafkreib3wkv3nhv3e7574y6hmolcjrxmlyzrxba2lpemh26bbojil2iio4 /tmp/x/2/x
cd /tmp/
ipfs add -n --raw-leaves -- x/1/x x/2/x
added bafkreidtzm4frjuhvbeuzizsgbjqcyuc6pnnhhkcz5rmuttz3wrkvr6zvq x/1/x
added bafkreib3wkv3nhv3e7574y6hmolcjrxmlyzrxba2lpemh26bbojil2iio4 x/2/x

Thanks.

[bmwiedemann]

It seems, the patch broke something else:

ipfs --upgrade-cidv0-in-output dag put --pin --input-enc raw --format dag-pb /path/to/openSUSE-Tumbleweed-DVD-i586-Snapshot20210714-Media.iso.dag-pb
Error: expected a regular file

And the other problem I encountered was exceeding the number of open files (1024 by default). The DVD has 5260 files.

[aschmahmann]

It seems, the patch broke something else

@bmwiedemann are you sure? Does that work without the patch?

I tried this and it worked for me:

C:\Users\adin\workspace\test\example>echo "a" | ipfs add -q
QmXNiF7qYu82aXfuVsmAyNd7Eku7psgBu7eXKMNhcGZjfN

C:\Users\adin\workspace\test\example>ipfs block get QmRJAPrEuBZyezSe5zKGsfh9KT91v2Dyj6DqHMRiFTXoS4 > block.dagpb

C:\Users\adin\workspace\test\example>ipfs --upgrade-cidv0-in-output dag put --pin --input-enc raw --format dag-pb block.dagpb
bafybeibl57rc754ayntwyrj52oolpnmc472d5x4x73r3r5sprhmsponvou

I'll put up a PR to go-ipfs-cmds and go-ipfs to see how tests look.

---

A couple of side notes that we can discuss further on discuss.ipfs.io if you have additional questions (just tag me):

ipfs dag put is meant for putting individual blocks of data and all blocks of data should be kept under 1MiB or they will not be transferable by Bitswap.

So it seems pretty strange to me that:

1. How you're getting a single dagpb block that's an ISO (a UnixFS DAG is what you'd get from ipfs add myIso)
2. Why you are using ipfs dag put at all here. Perhaps you're looking for ipfs dag import/export for moving around DAGs as files.
3. Why you'd need multiple file descriptors to open a single file being used for import (ISOs like zip and tar files are just one file to the file system no matter how many internal objects they represent internally)

[bmwiedemann]

Why you are using ipfs dag put at all here

The idea is to have https://github.com/bmwiedemann/ipfs-iso-jigsaw/ create a custom UnixFS PB object such as openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210715-Media.iso = bafybeiby7fd6dp37qaqvre2dypadzg2yxlw7voeoxno7cwe3qt32seu5xe (537533 byte) that re-uses file CIDs that I already store independently plus approx 20 MB for non-file data for iso bootloader and directory data.

https://www.reddit.com/r/ipfs/comments/ohbqgv/efficiently_store_daily_iso_images/ explains it a bit.

Why you'd need multiple file descriptors to open a single file being used for import

I use fuseiso to mount the iso and call ipfs add with a list of those 5000 files in the iso.
One workaround would be to call it multiple times with a part of the file list, but would be nicer to have it working out of the box.

[Zanda256]

Hullo team,
Can I pick this up? If no PR is in review currently or no one is working on it. @aschmahmann