Timeline/reccomended way to use IPFS from Python

[PJ-Finlay]

I'm the developer of Argos Translate, an open source neural machine translation library, and currently have models available for IPFS download. I'd like to add support for automatically downloading from IPFS at some point (currently you need to manually download and install the models). What's the recommended timeline/strategy for doing this? My current plan is to wait for this project to be production ready at which point adding IPFS support would be pretty straightforward. Is there any sort of timeline estimate for when this will be? Is there a better strategy like trying to use the Go implementation?

Thanks

[ntninja]

Sounds like an interesting project!

Unfortunately py-ipfs still hardly exists (and has been effectively on hold for the last year), so I wouldn't recommend waiting for it. Instead, use py-ipfs-http-client to first probe if there is an IPFS API Server already running on localhost and, if that isn't the case, initialize your own private go-IPFS instance (run these three commands once using subprocess.run or similar):

# Store go-IPFS data in private location
export IPFS_PATH="${YOUR_APPDATA_DIR}/ipfs"

# Initialize “embedded” configuration
#
# Meaning of profiles:
#   * lowpower: reduces network traffic
#   * flatfs: simpler filesystem implementation using less memory and disk
#   * randomports: changes the default 4001 swarm port to something else
ipfs init --empty-repo --profile=lowpower,flatfs,randomports

# Disable IPFS gateway port, since the API is sufficient
ipfs config --json Addresses.Gateway '[]'

# Expose API on Unix domain socket, instead of Localhost TCP
ipfs config Addresses.API "/unix/${YOUR_APPDATA_DIR}/ipfs.sock"

Start go-IPFS and communicate with it (pseudo-codeish, IPFS_PATH must still be set):

daemon = subprocess.Popen(["ipfs", "daemon", "--enable-gc", "--migrate"])
try:
    # Wait for control socket to appear (I'm not a aware of any better way unfortunately)
for _ in range(20):
        try:
            client = ipfshttpclient.connect(f"/unix/{YOUR_APPDATA_DIR}/ipfs.sock")
        except ipfshttpclient.exceptions.ConnectionError:
            time.sleep(1)
            continue
        else:
            break
    else:
        raise ipfshttpclient.exception.TimeoutError(None)

    # Do something with `client`

    # Ask daemon to exit
    client.shutdown()
    daemon.wait()
finally:
    # Ensure daemon doesn't stick around
    daemon.terminate()

[ShadowJonathan]

py-libp2p is pretty much dead, i worked on it for a few months, but realised that the maintainers only had interest in it due to them requiring it for Ethereum's Trinity client, which made the library extremely opinionated imo

now it rarely receives updates, and the supposed "community manager" is nowhere to be seen to look for another maintainer, or even strip the current "maintainers" off of their membership roles.

[amogorkon]

I'm working on justuse, aiming to complement pip and classical imports and was researching into ways to take load off of PyPI via P2P networking. Could IPFS in its current state help with the vision of hosting the Python ecosystem with PyPI as backbone and fallback? If yes, how hard would it be to integrate it?

[ShadowJonathan]

No, I would not recommend py-ipfs as a solid backbone for hosting PyPi blobs, I wouldn't recommend ipfs for pypi in general anyways, but that's out of scope for this issue.

[amogorkon]

No, I would not recommend py-ipfs as a solid backbone for hosting PyPi blobs, I wouldn't recommend ipfs for pypi in general anyways, but that's out of scope for this issue.

I'd like to hear why you'd think IPFS isn't suitable for hosting python packages, if not in this thread, I'm free to discuss it elsewhere :)