WebDAV/SMB Integration for Next Version [For Large Files]

[praeviadev]

We work with large images, usually north of 20GB. Uploading these images might take days. As a solution, what if OpenRelik was configured to access a WebDAV or SMB share?

[hacktobeer]

Hi, Relik works with normal file paths so if you can mount the folder on the host where the docker containers are running (and map them into the container as a volume) you can use any filesystem you want. We currently use NFS (Filestore on GCP) in some projects. But Relik is not filesystem type dependend.

Let me know if you were thinking of a different use case.

[LucaKuechler]

Hi, I came across this thread and I am still a bit unsure of how this would work.

I often receive multiple large images from my clients exceeding 100GB, such as *-flat.vmdk or .E01 files. These files are stored internally and can be accessed via an SMB share.

If I understand your instructions correctly: When I mount either the entire SMB share or a specific image as a volume inside the containers, how does OpenRelik recognize these files and make them available for starting a workflow through the GUI?

Thank you in advance for your help—please let me know if I missed anything.

[hacktobeer]

You would need to do 2 things:

1. mount your files into the container (eg you mount your SMB share on the host in /mnt/our-data and you mount that folder as a volume into every container so they can access the data).
2. create file database objects in OpenRelik so OpenRelik can use them. See here for an example function and here on how to use it.

This way you do not need to upload those files.

@berggren This use case has popped up before. (where files are available on a different filesystem and there is no need to upload them). We have the concepts of importers where we currently created a GCS importer. Maybe a local file system importer would make sense as a feature request.

[LucaKuechler]

Hi,

I’ve recently had some time to look into this idea and started experimenting with what you referred to as the “local file system” importer or as I’d describe it, a remote file link.

While testing, I noticed a few challenges when storing files that don’t reside on the OpenRelik shared file system. As to my understanding the GCP importer downloads the files to the shared file system and therefore doesn't face the same problems.

One example is the create_file_in_db function, which uses the magic.from_file function to determine a file’s type before storing it in the database.

The function generates a file path that would normally point to a file stored on OpenRelik's shared file system. This path always follows the pattern /usr/share/openrelik/data + uuid + extension. Since the base directory /usr/share/openrelik/data is fixed, it will cause an error because the file at that location does not exist unless it is downloaded from the remote location, which we want to avoid.

There were also questions around how the importer should be triggered, since the GCP importer uses a Pub/Sub model that might not be necessary in this case. A small subpage or even a button in the main interface would suffice.

I’ve documented quite a bit more on these findings and even put together a small proof of concept showing how “remote file link” support could be added to the OpenRelik server with only minimal changes.

If there’s interest, I’d be happy to collaborate on moving this feature forward. Would you prefer continuing the discussion here, on Discord, or in a dedicated PR?

Thank you in advance for your help and let me know if you have any further questions.

[berggren]

Thanks for the details, much appriciated. Good news is that this is on the roadmap to support additional read-only data paths. I have a PR in the works for this use-case.

• Tracked in: Support for External Read-Only Data Stores openrelik-server#120
• PR (wip): Storage Provider Support openrelik-server#121

@LucaKuechler and @dadamnmayne Do you think that functionality would cover your use-cases?

[LucaKuechler]

Hi,

Thank you for the update! I actually came across that branch after posting my initial message and had a chance to experiment with it a bit. From what I’ve seen so far, the Storage Provider support should definitely cover the most critical parts of our use case. I’ll do some further testing once it’s officially released.

If the PR is already in the final testing stage, I can also take a look at it directly.

Appreciate the effort on this!

[berggren]

Yeah, it's almost done :)

I'm doing some final testing and then it will go out for review.

(@hacktobeer for awareness)