Add `userdata` endpoint?

[Fuuzetsu]

Currently tailsrv allows input in a form of a seek location. I would like to add a special case. Allow tailsrv to accept userdata as the first thing on the socket which would then send a user-defined blob followed by a newline. Or an u64 followed by bytes of data. Whatever works.

So for a real example, I might start tailsrv like

tailsrv ... /tmp/foo/bar --userdata '{ "filename": "bar" }'

This lets me communicate to the client (on demand) that it's streaming bar and the client can then for example save the file locally with the same name. metadata is arbitrary bytes.

I could then send userdata/n0/n to tailsrv to get the user data followed by the actual data stream and I can happily use the information to, for example, replicate to the same filename, or whatever I want.

This allows to pass through information on the same connection instead of requiring a side-channel to pass through data which is a hassle to set up and race-y and error prone or requires format changes to underlying data stream to smuggle the data and so on.

[Fuuzetsu]

I guess there's a more general method which is to allow multiple named streams. So for example we can do

tailsrv ... /tmp/foo/bar --stream userdata:path/to/userdata/file --stream other-file:path/to/other/file 

One can ask tailsrv to change streams by giving a name of the named stream and it'll spit out data from those.

It's up to the caller to determine when stream is over.

I think this might be better. It does require making a file for even small userdata but that's OK. This allows a single tailsrv to serve lots of things, saving ports and not having to worry about synchronisation as one might do if we were emulating it with multiple tailsrv instances.

Yeah, I think I like it.

The base usecase of serving 1 thing is completely unaffected: we always have a base stream and unless name is given, it's what's given out.

[asayers]

tailsrv is all about streaming, but this metadata looks like a download-then-process kind of situation; ie. HTTP is a better fit.
I'm guessing the only advantage of using tailsrv to distribute this metadata is that tailsrv is already installed; but python3 -m http.server is probably available too.

[asayers]

Oh, you did mention another advantage: port depletion. But I'm sure you can spare one more port.

[asayers]

Re. multiple streams: I'm not totally against adding this feature. But the only thing it adds compared to running multiple tailsrvs is that you can refer to streams with a string, rather than a number. I'm not sure this is worth the added code.

[Fuuzetsu]

Port depletion also matters when the other side is behind a restrictive firewall. One can hack around with vhosts and proxies but it's a hassle.

One can always start up a tailserv to that only distributes metadata but you have to keep manage way more processes. One must always stop metadata endpoint then other streams then start new streams then start new metadata endpoint. Clients must always poll for metadata on connect and reconnect and then poll for stream with timeouts etc. It's just simple if there is one connection and nothing to wait for and recheck.