Data scenarios following organisation changes

[psd]

How do we represent changes to local government in our data?

[stevenjmesser]

What’s in scope here? Is it changes to administrative geographies and the hierarchies, types and names?

Given the title I’m assuming that changes to legislation that confer a power or duty are out of scope.

[psd]

Very much a placeholder discussion at this stage. We need to capture the scenarios for how data is collected and assembled into entities, and document them as patterns.

[greg-slater]

Adding some thoughts as we recently made some fixes in the brownfield-land dataset for entities belonging to terminated organisations which have been superceded by newer ones. We found one of two scenarios:

1. We have entities belonging to the old organisation as well as the new one (e.g. Aylesbury and Buckinghamshire entities)

In this scenario we redirected entities from the old orgs to entities for the new orgs where they could be matched on both reference and site-address fields, and retired any entities which couldn't.

2. We only have entities belonging to the old organisation but no new endpoint for the new organisation (e.g. entities for Allerdale but no new endpoint for Cumberland)

It's less clear what's best to do in this case, and we haven't yet made any changes. In the Allerdale example, the endpoint is still on the Allerdale site despite the fact that the ONS organisation was terminated in March 2023.

To update the organisation we could possibly patch or default the organisation to Cumberland and add new mappings to lookup.csv.

The main questions I have on this topic are:

• Do users expect to be able to search for data using new organisations as soon as they exist?
• Should we wait until we get data from new organisations before redirecting entities, or cutover to them sooner using manual configuration?

[stevenjmesser]

In my view, a re-organisation of local government – whether that’s a merging or breaking of authorities, or a change in boundaries – is an official change, and the facts associated with any entities should change too. For scenario 2, that likely means updating the organisation, as you’ve suggested.

Even though there is no new endpoint from the new authority, we can infer that the old endpoint is good enough for a portion of the new authority. Authorities may not migrate or merge websites in tandem with official changes to authorities, so we can’t expect there will be a new endpoint so soon. It’s an lag in operations after an official change.

[psd]

1. I'm unclear what you're saying here. There's no need to create new entities due to an organisation change, so should be no need for any redirection.

You should definitely not be retiring entities just because of an organisation change. We only mark entities as being old to rectify a mistake, when the are a duplicate entities for the same "thing", or an entity has been added in error.

The way we "retire" entities is to add an end-date. We need to keep old entities wherever possible for historical reasons as they can still be used as material evidence.

By "belong" I take it we mean the change the entity-organisation value from Aylesbury to Buckinghamshire. That's what I'd expect as in this case the brownfield site now falls in the jurisdiction of Buckinghamshire and is the responsibility of the new organisation.

I think we need to craft some tests and documentation to ensure people don't remove entities or overuse redirection.

2. This will often be the case during a reorganisation. Our experience shows it can take several years for a new Unitary Authority to move endpoints and data from old district websites and merge them into a new system.

Without changing any data, the new organisation should see existing data within their boundary through the providers service.

Assigning the entity-organisation to the new authority will result in the data from the existing districts will no longer be marked as "authoritative".

We then need to prompt the LPA for them to provide a new endpoint or link to the existing endpoint from their source page so we can mark the endpoint as belonging to the new LPA.

Don't patch the data in this case. Patching is again only to be used to rectify mistakes, not change data.

Let's work on documenting these patterns.

I am concerned if we are removing or redirecting entities unnecessarily.

[greg-slater]

To add some clarity:

1. This is a scenario where new entities had already been erroneously created for the new district, so the changes were made to try and correct these errors. In situations like these it's often not completely clear whether the type of error fits the criteria for retiring or not, so I completely agree on the need to document these patterns.

Given that, is the following an appropriate solution?

• where they can be matched on reference and site-address redirect Allerdale entities to Buckinghamshire entities (this is what we did already)
• where no match can be made from an Allerdale entity to a Buckinghamshire entity it should be given an end-date (these were retired but we can change that)