Skip to content

Add ADR048: Immutable Form Versioning #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
theseanything wants to merge 4 commits into
base: main
Choose a base branch
from
+83 −0
Open

Add ADR048: Immutable Form Versioning #248

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3cd31b8
Add ADR048: Immutable Form Versioning
theseanything Mar 24, 2026
d00263c
Update ADR048 about hard submission deadlines
theseanything Apr 1, 2026
8220af5
Update ADR048 about linking submissions to form versions
theseanything Apr 1, 2026
51df71a
Update ADR048 about schema versioning
theseanything Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions ADR/ADR048-immutable-form-versioning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# ADR048: Immutable Form Versioning

Date: 2026-03-24

## Status

Accepted

## Context

GOV.UK Forms currently uses a mutable document-based system for managing form lifecycle states. The `Form` model has an associated `FormDocument` model, with documents tagged as `draft`, `live`, or `archived`. The current API (v2) exposes these via:

- `GET /api/v2/forms/:form_id/draft`
- `GET /api/v2/forms/:form_id/live`
- `GET /api/v2/forms/:form_id/archived`

The `FormStateMachine` manages transitions between states (`draft`, `live`, `live_with_draft`, `archived`, `archived_with_draft`), and the `FormDocumentSyncService` synchronises the JSON content into `FormDocument` records when these transitions occur.

This approach has several limitations:

1. The `/live` endpoint is mutable. The content behind `/api/v2/forms/:form_id/live` changes each time a form is re-published, so consumers must always re-fetch. This prevents effective caching.
2. No explicit link between a submission and the form version it was submitted against. When a form is updated and re-published, there is no reliable way to identify which version of the form a given submission relates to. This makes it difficult to group submissions by form version or detect when a form changed between batch submission deliveries.
3. Mid-journey disruption. If a form creator publishes a new version or archives a form while a user is part-way through filling it in, the form can change or disappear mid-journey. There is no mechanism for in-progress users to continue with the version they started.

## Decision

We will introduce an immutable versioning model for published forms, exposed through a new v3 API. The key changes are:
Copy link
Copy Markdown
Contributor

@lfdebrux lfdebrux Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a way we could achieve the same aims and outcomes of this ADR but without needing a new API version; would you be interested in that or would you prefer to have a new API version?

Copy link
Copy Markdown
Contributor Author

@theseanything theseanything Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooo! Yes always interested in hearing ideas!

Copy link
Copy Markdown
Contributor

@lfdebrux lfdebrux Apr 1, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if we added the /versions endpoints you're suggesting to the v2 API, and added a version link to the form documents, that would cover the same functionality more or less?

So for a form document that's been made live, regardless of where it's retrieved from, you'd see something like:

{
  form_id: "aB123x",
  name: "Test form",
  ...
  links : {
    version: "https://api.forms.service.gov.uk/api/v2/forms/aB123x/versions/9"
  }
}

Copy link
Copy Markdown
Contributor Author

@theseanything theseanything Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah thats a nice idea.

I think my main concern with doing this in v2: we’d end up with a slightly confusing API shape where the old state-based endpoints still exist alongside the new version-based ones.

For example, it becomes unclear what /api/v2/forms/:id/archived is meant to represent

Copy link
Copy Markdown
Contributor

@lfdebrux lfdebrux Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it still represents the form document as it was when it was archived? We have that so that it's easy for forms-runner to preview the form after its been archived but changes have been made to it. With endpoints for versions we could alternatively do that by sending them to a preview for that version, which I guess what you're proposing currently, but both still seem valid to me?

It doesn't help that I've thought about this so much that it's hard to imagine looking at it for the first time, so I'm unsure about what you mean about it being unclear 😅

When I was thinking about the v2 API I thought about the state endpoints as branches/tags in a git repository, or symbolic links in a filesystem, so that's where I'm coming from.


### New API endpoints


| Endpoint | Description |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `GET /api/v3/forms/:form_id/draft` | Returns the current draft form document JSON (mutable, changes as the form creator edits) |
| `GET /api/v3/forms/:form_id/versions/:form_version` | Returns an immutable, versioned form document. Once created, this content never changes. |
Copy link
Copy Markdown
Contributor

@stephencdaly stephencdaly Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want live form documents to be actually immutable - or immutable from an end user perspective? The main issue we might encounter here is when we make internal changes to form structure such as adding/removing/changing attributes.
It would make it harder to make changes to forms-runner and hard to support users reverting changes if we don't also update the structure of historic form documents when we make breaking changes to the format.

There might be some things we can do to lessen the need for altering existing form documents. Currently when we add attributes to a form, we always back-fill the attribute on form documents. Potentially we can change our approach to this and fall back on default values when an attribute isn't present rather than updating it in the database.

Copy link
Copy Markdown
Contributor Author

@theseanything theseanything Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

V good point — I think ultimately we want the semantic content to be immutable.

We could handle changes in schema in a few ways:

  1. Include a schema version in the form document. This makes it explicit to consumers how to interpret the content. It may mean we'd need to support multiple schema versions, but in practice, making sure most changes are backwards-compatible minimises the complexity around this. (Could reserve breaking schema changes for a new API version.)

  2. Weaken the immutability by keeping the semantic content the same but allowing the schema to be updated. i.e. migrate old documents to newer schemas. We wouldn't be able to cache "forever", but schema changes are probably infrequent enough that caching would still be useful.

Copy link
Copy Markdown
Contributor Author

@theseanything theseanything Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd probably lean towards 1. as it gives stronger guarantees for the API, in way that is less likely to break forms-runner.

Copy link
Copy Markdown
Contributor Author

@theseanything theseanything Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the ADR with a section on API changes.

| `GET /api/v3/forms/:form_id/latest` | Returns the latest live version of the form (a redirect or alias to the most recently published version) |


### Lifecycle

- Draft state: A form being edited has a draft available at `/api/v3/forms/:form_id/draft`. This behaves similarly to today.
- Publishing (making live): When a form is made live, a new immutable version is created and assigned an incrementing version identifier (e.g. `1`, `2`, `3`). It becomes available at `/api/v3/forms/:form_id/versions/:form_version`. The `/api/v3/forms/:form_id/latest` endpoint points to this new version.
- Archiving: When a form is archived, `/api/v3/forms/:form_id/latest` and `/api/v3/forms/:form_id/draft` return `404` (or `410 Gone`). However, all previously published versions remain available at `/api/v3/forms/:form_id/versions/:form_version` because they are immutable.

### Linking submissions to form versions

Each submission will store the `form_version` it was made against, and that version information will be exposed to downstream form processors. This makes the version of the form explicit at the point of processing, instead of requiring processors to infer changes from the submission payload shape.

### Schema changes

Consumers should always be able to treat `/api/v3/forms/:form_id/versions/:form_version` as a stable representation of the questions, structure, and behaviour that were published at that point in time.

To handle changes in how form documents are represented, the form document should include an explicit `schema_version`. This makes it clear to consumers how to interpret the document while allowing the published content itself to remain immutable.

In practice, schema changes should usually be backwards-compatible so consumers can continue to handle older and newer documents. Breaking schema changes should be reserved for the introduction of a new API version.

### Hard submission deadlines

For legal or policy reasons, some forms may need a strict cutoff time after which no new submissions are permitted. Archiving would no longer act as a way to cut off in-progress journeys and prevent any future submissions.

This behaviour should be re-implemented. For example, this could be a deadline timestamp attribute on the form that `forms-runner` checks before displaying the form or accepting a submission. This is more explicit and reliable than using archiving as a proxy for a hard stop. It would also allow form owners to schedule a cutoff in advance.

### Content removal

Immutability prevents deletion as part of normal operations. A separate process will be needed for exceptional cases where published content genuinely must be removed (e.g. GDPR erasure).

## Consequences

### Positive

- Cacheable published forms. Versioned form documents at `/api/v3/forms/:form_id/versions/:form_version` can be cached indefinitely by consumers (e.g. forms-runner, CDNs), significantly reducing load on the API and improving latency for form rendering.
- Submissions linked to form versions. Each submission can explicitly reference the `form_version` it was submitted against. This enables grouping submissions by version and helps people processing submissions to know exactly which questions were asked.
- Graceful publishing. Users who have already started filling in a form can continue submitting against the version they began with, even if the form creator publishes a new version in the meantime.
- Graceful archiving. When a form is archived, new users can be prevented from starting the form while users who have already started can finish and submit against the version they are on.
- Reverting to previous versions. Preserving all published versions makes it easier to implement future features allowing form creators to revert to a previous version of a form.
- Audit trail. The full history of published form versions is preserved and addressable.

### Negative

Copy link
Copy Markdown
Contributor

@lfdebrux lfdebrux Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another negative is increased complexity for form processors; making it so that an old version of a form can be submitted after the new version has been made live means that submission processors need to be able to handle both old and new version submissions for a time.

Copy link
Copy Markdown
Contributor Author

@theseanything theseanything Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, but I think form processors already have this problem today. Unless a form owner can update the form and their processing system at exactly the same time, there's always a window where submissions come in against a version the processor wasn't expecting. The current setup just makes that invisible.

Explicit version numbers actually make it easier to handle. A processor can see which version a submission was made against and branch accordingly (e.g. if version 2, do X, otherwise do Y), rather than having to infer what changed from the shape of the data.

Copy link
Copy Markdown
Contributor

@lfdebrux lfdebrux Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so as part of the changes for this ADR we'll expose the version information to form processors, and inform them of the change?

Copy link
Copy Markdown
Contributor Author

@theseanything theseanything Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup - added info that submissions would reference the form version. And yup, would be good to inform them.

- Migration and data model complexity. Existing consumers (primarily forms-runner) will need to be updated to use the v3 API, requiring a transition period running both APIs in parallel. The `FormDocument` model (or a new model) will need to support version numbering alongside or in place of the current tag-based system (`draft`, `live`, `archived`).
- Archiving no longer acts as a way to cut off in-progress journeys and any future submissions. Some forms may rely on this behaviour for legal or policy reasons. We would need to re-implement this behaviour in a new way.
- Schema compatibility discipline. We would need to keep schema changes backwards-compatible within the API version wherever possible, and treat breaking schema changes as part of a future API version.