Skip to content

Add ugly configuration functionality #118

@pooleycodes

Description

@pooleycodes

Overview

In order to move to using the Manage Service for all datasets and not have to manually configure in Github to make things work, we need to get to the point where all pipeline and collection files can be tweaked within the Manage Service. These changes can then be reprocessed via the async to confirm the data is correct before being passed through to the Add Data step to assign lookups, entity-organisations etc... It will then all go into the Github Action to be added to the Pull Request with all the changes.

To move quickly on this, we will start by allowing each configuration file to be edited as a CSV to mirror the current Github functionality. At a later date, we will work through each of these files to see if a "prettier" version can be created that doesn't rely on a working knowledge of CSVs and how the pipeline files are constructed.

Detail
We will want to add the following pipeline files to a Configure Data page that is accessed via the Check results. For all of them, we should also confirm that the current CSVs across all the collections have a uniform order of fields in the header row:

  • column.csv
  • combine.csv
  • concate.csv
  • convert.csv
  • default-value.csv
  • default.csv
  • filter.csv
  • patch.csv
  • skip.csv
  • transform.csv

NB: the following files should not yet be included via the Configure Data page:

  • entity-organisation.csv (this is completed as part of the Lookups process)
  • expect.csv (this is a separate process that will need to eventually be mapped into Manage, but is not concerned with data adding)
  • lookup.csv (this is completed as part of the Lookups process)
  • old-entity.csv (this will need to be added to add data at a later date when we tackle duplicate entities)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions