LocalFiles remote and XenonRemoteFiles local?

[LourensVeen]

You may want to use an existing WebDAV on a different server, e.g. BeeHub, as the local storage. Vice versa, you may want to run the service and a WebDAV on a staging node, with the WebDAV saving directly to a filesystem that's accessible from the compute nodes.

The former would require a LocalFiles that understands WebDAV, the latter a XenonRemoteFiles that does nothing? But LocalFiles then also should not read in the files, if XRF isn't saving them. So LocalFiles should then also do nothing?

Needs thought.

[LourensVeen]

For the second option, the user will upload stuff to the WebDAV, but we will still want to have a directory per job so that they can't interfere with each other. So I think we'll want to make a copy of the input files anyway. That just means that XRF gets a local:// adapter for the remote system, with a given jobs directory, and LocalFiles gets pointed to the local WebDAV directory.

For large files and zero-copy, you can always upload them out-of-band (or more likely, you've calculated them before and they're already on the remote resource), and pass their location on the resource as a string to the workflow, rather than as a File. That's effectively manual staging, which then gives you full control.

[LourensVeen]

For the former, LocalFiles needs to be able to download from http, which I think it can already do. But it also needs to publish results by uploading them to a WebDAV, for which it needs to be a WebDAV client. In that case it also needs credentials for the WebDAV, which it may need anyway for downloading as well. Question is how that will work, I guess the service will need to use the same credentials as the user.

So should we start thinking about automatically starting this service somewhere, e.g. on a staging node? Perhaps that's beyond the scope of this repository.

[LourensVeen]

Also, iRODS support would be nice, instead of the WebDAV.