Submitting workflows to Omics with large inputs is limited to 50,000 bytes

[golharam]

botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the StartRun operation: Workflow parameters must be less than 50000 bytes

[golharam]

Can we provide inputs from a file?

    --parameters file://workflow-inputs.json

This results in a unauthorized exception.

[golharam]

I think I can upload an input file to S3:

import boto3

s3_client = boto3.client('s3')
healthomics_client = boto3.client('healthomics')

# Upload large input data to S3
s3_client.upload_file('large_data.txt', 'your-bucket-name', 'large_data.txt')

# Start a run with the S3 URL as input
response = healthomics_client.start_run(
    WorkflowId='your-workflow-id',
    InputParameters={
        'data_url': 's3://your-bucket-name/large_data.txt'
    }
)

This doesn't either. InputParameters is not recognized..

[golharam]

Currently the recommend approach on AWS is to put the list of inputs in a file, then have that step of the workflow parse the file for the input parameters.

I'm not a fan of this because now the workflow is responsible for moving files around on a given platform and not just crunching on those files.