Record provenance in a standardized way

[sdruskat]

@led02 suggested using W3 PROV-O for this, which seems like a good idea.

There is an initial basic implementation in #288, but this needs more conceptual work.

This issue tracks this.

• Related: Refactoring of data model [PARENT ISSUE] #153, Rename tags to provenance #336

TODOs

• Check with PROV pros for missing processes & correct implementation (@sdruskat)

• Add curation activity

• Fix typo (duplicate curation data)

• Find solution for recording provenance from deposit substeps

• Design post-process command

• Where are settings recorded? (probably in the plugin agents)

[sdruskat]

As far as I can see, the requirements for this feature include:

1. Provenance should be recorded using the PROV-O ontology
2. All agents, activities and entities in creating and editing the metadata sets in hermes should be recorded.
3. The recording of provenance should be handled by the hermes framework, and should not be implemented by, e.g., developers of plugins. This means that hermes needs to implement hooks for provenance recording at least in the plugin base classes and the API (SoftwareMetadata). Other necessary hook points may exist and need to be identified.
4. Whenever hermes is run, each run should create and/or update provenance data and cache it, so that the cached provenance data always summarily reflect the most recent run. This means that, when I run command A twice, at time t0 with plugin X and time t1 with plugins Y and Z, the cache at (currently) .hermes/A/result/prov.json after t1 contains provenance data for both runs and for all plugins.
5. For deposition, the inclusion of the complete provenance file in the deposit should be possible.
6. In post-processing, it should be possible to post-process provenance data.
7. Provenance is implemented using immutable data structures.

@led02 & @zyzzyxdonta, do you agree? Have I forgotten anything?

[led02]

I think 3. is nearly impossible to achieve, at least for harvesting plug-ins (e.g., how should the framework know where the data was sourced). Also post-processing and deposition have similar problems…

[sdruskat]

Re 7. (immutability), this could be achieved using Python dataclasses, or a pydantic model that implements PROV-O.

[sdruskat]

Re

I think 3. is nearly impossible to achieve, at least for harvesting plug-ins (e.g., how should the framework know where the data was sourced). Also post-processing and deposition have similar problems…

That's a good point and one for consideration for the implementation. One option would be to use metaprogramming techniques to catch file operations, web API calls, etc. Alternatively, hermes base plugins could provide their own data access methods that wrap common data access patterns and hook provenance recording into these.

[led02]

Immutability is best achieved using a block chain… (sorry, lots of trolls here in Norway) Using monkey patching is not guaranteed to work and might bring strange side effects (e.g., if a method is already monkey-patched).Hence, we need an API and a “Hermes Full Provenance Seal of Approval”-Program (or at least review guidelines, good practice guide, or the like)

[sdruskat]

Hence, we need an API ...

Yes, API was my way of thinking as well. If we have API for

• file access
• API access (via requests, which is already a dependency)

we can intercept the file names and contents, requests and responses.

... review guidelines, good practice guide

Yes. Or perhaps a generic end-to-end test in the plugin template repo(s) that check basic provenance recording.

[sdruskat]

We also discussed today the provenance info to be recorded, and how to record them in terms of PROV-O objects & properties, and this is what we came up with.

Source: hermes-harvest-prov.drawio

This assumes:

class MyPlugin(HarvestPlugin):

  def __call__(self):
      data = SoftwareMetadata(...)
      raw = self.load(toml.load, "pyproject.toml")
            
      data["key"] = raw["key"]
      
      return data
  
  
class HarvestPlugin():
    
    def load(func, str, ...):  # json.load
        # TODO: handle byte formats
        with open(str) as fi:
            data = func(fi)
        prov.record(load, str, func, raw)
        return data  # dict