Enhancement: Add APIS integration to SFA post-storage workflow

[fiver-watson]

Background and context

SFA's long-term solution for digital preservation involves many disparate systems across multiple different departments and agencies coordinating. However, the Archival Information System (AIS) is intended to be source of truth for all archival metadata. Given that various users will be creating and submitting metadata that at times may be in conflict with existing records or other submissions, SFA has developed an intermediary application, called APIS, to negotiate the review and synchronization of metadata prior to import into the AIS.

During preprocessing, Enduro and APIS will resolve any potential metadata conflicts and deliver the initial SIP metadata.xml file. In return, APIS will supply an ID for the request. This ID can then be used in post-storage to deliver a copy of the AIP METS.xml file for the AIS to index and store. Details of this earlier preprocessing integration can be found here:

• Feature: Add APIS integration to SFA preprocessing workflow #173

To support the second phase of this new integration, ASI will need to modify the existing SFA post-storage child workflow so that it can use the ID delivered during preprocessing and send the METS file to the AIS via APIS.

Implementation proposal

The following workflow diagram describes the proposed initial post-storage workflow modifications at a high level:

As usual, during preservation Enduro will continue to poll the preservation engine for updates, until it can confirm that preservation has completed successfully and the PIP has been transformed into an AIP that is now stored. Once this is confirmed, the post-storage workflow can begin. Enduro will need to:

• Fetch a copy of the AIP METS.xml file
• Using the ID value returned from APIS during preprocessing, initiate a POST to /api/ImportTasks/{id}/importRuns
• This API request expects 3 required parameters to be included:
  • id: the ID value returned from APIS during preprocessing, via the importTasks request
  • file the METS.xml file from the resulting AIP
  • importBehavior: this pram expects one of two enum values:
    • AppendOnly: This is the default value if the file param is not included in the request (it's the only param here that is optional), and will be used in most cases when no metadata conflict is identified eariler during preprocessing via APIS
    • OverwriteAndAppend: In rare cases where a metadata conflict between the submitted SIP and the AIS is found, preprocssing will be modified via issue Feature: Add APIS integration to SFA preprocessing workflow #173 to include a decision point, allowing an archivist to either cancel the ingest, fix the metadata in the SIP, and then start a new ingest, or else continue and treat the SIP metadata as the source of truth overwriting what is currently in the AIS. This enum value will be used in that second case. Meaning: IF a user chose the "CONTINUE & OVERWRITE" option during the preprocessing workflow, Enduro will need to remember that and use this enum value when submitting the ImportRuns request.

Once submitted, APIS will respond with a runId value.

Enduro can use the original ID returned from the ImportTasks request performed during preprocessing to query for updates - the same GET /api/ImportTasks/{id}/status request API request used in #173 can also be used to return information about the ImportRuns process - this request can return the following eleements:

       "status": "Neu",
       "analysisResult": "AlleGleich",
       "analysisProgressInPercent": 0,
       "importResult": "Erfolgreich",
       "importProgressInPercent": 0,
       "processedDocumentCount": 0,
       "totalDocumentCount": 0

The status property of this response will use the following enums:

• WirdImportiert if the import is still in progress, or
• Importiert if the import run has completed

When still running the importProgressInPercent will return a progress percentage number that can be used to provide progress updates in the Enduro UI.

When the import is completed, the importResult property will tell us the outcome:

• Erfolgreich = success
• Fehler = error

If the response is an ERROR, Enduro should:

• Update the status of the relevant task to ERROR
• Display a WARNING admonition on the SIP ingest page, indicating that the post-storage workflow has failed, but the AIP is still successfully stored.
  • This same message should be included in the task details when the ERROR status is applied, so that there is a persistent record of the post-storage workflow failure, even if the notification banner is dismissed.
• Perform any required cleanup and end the workflow

If the response is a SUCCESS, then enduro can simply update the workflow as completed, perform any cleanup etc.

Open considerations

Currently post-storage workflow failures are silent, with no indication in the Enduro user inteface as to what has occurred. We will need to consider the wording of any admonitions or notifications shown, and how to clarify that the metadata delivery failed but the AIP is still stored.

This may be the moment for us to consider breaking the worklfows into multiple smaller workflows in the user interface, as that way we can better indicate that some parts succeed and others fail. Further internal discussion and design work will be needed for this part.

Additional context and resources

A copy of the most recent APIS API specification can be found here in Drive. It can be loaded into a site like SwaggerEditor for a human-readable version if desired.

Meeting minutes about the APIS integration with SFA can be found here in Drive.

[fiver-watson]

2026-02-27 edit of the issue ticket description:

• Updated the workflow diagram to better indicate how the prior outcome of the preprocessing workflow determines exactly how the ImportRuns API call is submitted

[fiver-watson]

2026-03-03: updated based on new client feedback. The exact updates from Christa:

I just talked to Jörg to get a better understanding for the import statuses and I have some additional points as feedback:

For Issue 174
We would suggest that you use /api/ImportTasks/{id}/status instead of /api/ImportTasks/{id}/importRuns/{runId}/status since it should give back all of the information you need and there we can control the vocabulary whereas the importRuns status just gives back whatever ACTApro Desk provides as a status. If you use the ImportTasks Status, this is the relevant vocabulary:

• The "Status" is "WirdImportiert" while the work is in progress. With "importProgressInPercent" you can see how far along we are.
• The "Status" is "Importiert" when the work is finished. Then you can get the "importResult" which will either be "Erfolgreich" (= success) or "Fehler" (=error).