Collect workflows

[makkus]

Description

A catch-all issue to remind everyone that we want to collect as many potentially useful workflows as possible. As with the 'Collecting datasets' issue, 'potentially useful' is hard to define, but basically if we come across a workflow we think might be a good candidate for our project, we should record it.

This does not mean we need to create a fully-fledged Jupyter notebook for it, in fact, there does not need to be any coding work done. As a minimum, we should describe the problem the workflow would solve, as well as the input data it would require and the output data it would produce. Obviously, in as much detail as possible, but this can be improved over time.

Also, if possible, it would be great to come up with a draft segmentation of the workflow into modules. Again, this does not need to be perfect as it can be refined later, but having a table of descriptions of potential steps (incl. each of their inputs and outputs) will help us a great deal when it comes to deciding which modules to create, in which order, and how they should be designed so they can cover the widest range of use-cases.

Task list

• decide on a format and place to record those workflows
• create metadata schema for information we want to record along the workflows
• collect workflows (open ended)

[makkus]

So, a few more (random) thoughts on this:

I think in general, what we are trying to do here is to collect workflows in a form that is not dissimilar to what users would see in our application when we present to them the options they have in terms of what they can do with their data at hand:

• we'd have to have a good title, that makes it intuitively clear what the workflow does (if that is possible)
• a good, concise first paragraph that summarizes the workflow in a bit more detail, while not requiring the user to have in-depth knowledge about any of the things that are involved
• (optionally) more documentation on the concepts involved, what characteristics the input data would need to be 'eligible' for the workflow, what to expect in terms of output data and its characteristics, details about how things are done if necessary, links and references to tools, methodologies, etc.

We also want to describe the (internal) steps of the workflow, using more or less the same format that is used for the overall workflow itself. This is what users would see once they selected a workflow, and are working on a specific step. Basically, the 'code-view' plus/minus some details.

I don't know if it's helpful, but one way of coming up with a list of workflows would be to start with the datasets we collected, and try to think of different ways those can be wrangled into something that could potentially answer research questions. At this stage, we are not really looking for super-interesting workflows that have never been done before, but more for ones that mirror more or less common, established practices (digital or analog).

As I've said before, it would also help if we categorize the workflows we come up with into 'pre-processing' and 'research-question' (or whatever word we want to use here) ones. The main reason why this will be helpful is technical, because categorically the pre-processing ones have a variety of possible input data shapes, and always a well-spec'ed output data schema. The 'research-question' ones have well-spec'ed input and output data schemas. The result of pre-processing workflows would always feed into the other type of workflow, so their schemas would ideally match up as well as possible.