Implement a streaming ls api

[Stebalien]

Currently, when listing large directories, the file list needs to be buffered in memory. This is fine for ~50K files but likely won't work well for ~1M files (in a single directory).

Unfortunately, ipfs ls accepts multiple arguments. If we want to stream back results, we'll have to somehow group them by directory (hopefully without sending too much extra data).

[ajbouh]

You can also say that streaming is only supported for a single directory input, no?

[ajbouh]

Just added some performance measurements for a big 1.2M file sharded directory over at tesserai/iptf#2 (comment)

[Stebalien]

You can also say that streaming is only supported for a single directory input, no?

I'm currently thinking:

1. Spit out the directory "object" with a Links: "streaming" (or something like that) field.
2. Follow it with the stream of links.

That way, we don't change the output format based on the number of inputs.

[ajbouh]

I'm not deeply familiar with this interface, but what you've outlined certainly sounds reasonable

[Stebalien]

(fixed in #5611)