Alternative data upload mechanisms #4
Description
Currently, data can be uploaded to an S3 data import bucket for each Local Node, from where it will be synced to the node EFS file storage. S3 allows for web-based data upload or command-line upload/sync using AWS IAM credentials.
Enhancements or alternative mechanisms could be deployed, for example
- Use an AWS lambda to trigger synchronisation immediately when data are added to the S3 bucket
- Install an sftp server on the Local Node, accessible when the VPN is connected. This would require users to upload data using the command line or install a local sftp client (as most web browsers do not support sftp upload)
- Build a custom data upload website which would be accessible through the VPN. This could allow for customised authorisastion mechanisms so that, for example, data providers could use their FBM credentials instead of requiring an AWS IAM user account. AWS provide a bare-bones example, but this would require some work (for example, it does not support batch upload): https://aws.amazon.com/solutions/implementations/simple-file-manager-for-amazon-efs/
- Build a custom upload tool to install on the client side (which might integrate with a PACS system, for example)