Preprocessing: support metadata produced from generated metadata

[anna-parker]

At the moment preprocessing generates each metadata field independently from the input metadata submitted by the user or obtained from the nextclade run. However, it is not configured to use generated metadata to create new metadata fields.

For example the concatenate takes country, date and accession as input and joins them to create a displayName field. The processed country field is processed by process_options to validate it is a valid country. We do not perform this processing in concatenate and concatenate thus accepts invalid countries. The only way to add this validation would be to create a new function with the necessary input of both the concatenate and process_options, this is what we had to do with the build_display_name and my new assign_custom_lineage function.

Should we alternatively restructure the pipeline to allow metadata fields to be created from processed metadata fields? i.e. allow concatenate to take the processed country field as input instead of the non-processed input field. This raises a question of how many layers of fields can/shoudl we allow. Given input metadata field I1 and I2, we now support output field A produced from I1 and I2. If we allow B produced from A and I2 we might still want to create C from B and I1...

[maverbiest]

Thanks for opening the discussion! This is something I was also thinking about while working on prepro, however it's not something I have experience with.

I guess a naive approach would be to have some sort of priority system where functions run in sequential groups in an order configured in the values.yaml. But I don't have a clear idea of how to not make this super error-prone when configuring it.

As far as I'm aware, a way similar problems are solved by software like Nextflow, Snakemake etc. is to create a DAG based on dependencies between different entities and use this to determine the order in which things need to run. For our usecase, each metadata field would declare which other metadata fields it depends on, which we'd use to build a graph. I can imagine there are libraries with existing implementations for this.

It may be overengineering things in our case and is not something I'm too familiar with, but would be happy to look into it a bit :)

[anna-parker]

Yeah totally! "layers of dependency" does scream using a priority DAG of sorts. We need prepro to be a continuously running service that can constantly get more data to process so as you mentioned sth like snakemake wouldn't really fit the requirements. Ideally you would define the DAG in advance based on the config using a python library for this like https://networkx.org/documentation/stable/reference/algorithms/dag.html.

But as you said I do fear that this gets a bit overengineered - Im also not sure how to implement this without totally rewritting preprocessing.

So I am curious if anyone has any suggestions on either how to do this without a DAG - or an "easy way" to switch to using a DAG for this.

[maverbiest]

I would indeed determine the 'order of metadata calculation' only once when the config is parsed, and then pass this on to prepro. Naively, my minimal approach would be to add a field to each metadata that declares it's dependencies (if it has any), parse this into a DAG, and then extract the order of execution from the graph into a list that prepro uses to know the order in which to calulate metadata fields.

According to Gemini there's python libraries for this:

from graphlib import TopologicalSorter

# Define your dependencies: Field: {Dependencies}
# Here, B needs A, and C needs A and B
dependencies = {
    "B": {"A"},
    "C": {"A", "B"},
    "D": {"C"},
    "E": {"A"}
}

ts = TopologicalSorter(dependencies)
order = list(ts.static_order())

print(order) 
# Output: ['A', 'E', 'B', 'C', 'D'] (or similar, A will always be first)