Non unique codes - standard/codelists /geoCodeType.csv

[michaelwood]

Unlike other codelists the list of geoCodeTypes is non unique - there are multiple codes with multiple differing titles/descriptions. I'm not sure how to choose the correct one when trying to display the non-code version when looking at Grant data.

https://github.com/ThreeSixtyGiving/standard/blob/main/codelists/geoCodeType.csv

[mariongalley]

@michaelwood There are a couple of issues:

1. The geoCode type CSV needs to be updated
2. The geoCode type codelist is not currently validated and so contains data that isn't drawn from the codelist, which means the codes can't always be looked up.

[neelima-j]

Why does geoCodeType have duplicates?

geoCodeType is not constrained in the schema and can contain any string.

History

#29 is how we chose this codelist.
The duplication question was raised then (exactly 10 years ago) in #29 (comment)

However, I fear this has:
(a) Duplication - I see 'WD' used as the abbreviation for a number of different ONS Codesets - so not sure the abbreviations > are appropriate to use as our codes;

This was not addressed and the codelist was added.

Looking at the megalist from Register of Geographic Codes uploaded to drive as (RGC_DECEMBER_2020_UK v2)and the GSS wikipedia page ,
It appears that we have used a non unique 'Entity abbreviation' as the code. The source uses 185 unique Entity codes. The current codelist is a subset of the source, except for 6 Codes/ Entity abbreviations.

Duplicates

There are 46 unique values with 18 duplicates.

Duplicate Codes / Entity abbreviations occur because they are repeated for different Entity coverages (One of the 9 - Wales, Channel Islands, Isle of Man, Scotland, England and Wales, England, United Kingdom, Northern Ireland, Great Britain)

Here is a table showing the 3 character GSS code prefix, the code the codelist uses (which is an non unique entity abbreviation), the geographic coverage of the code, and the title of the code (name of the entity)

Code	Entity abbrev. / Code in codelist	Entity coverage	Entity Name / Title in codelist
E41	CMLAD	England	Census Merged Local Authority Districts
W40	CMLAD	Wales	Census Merged Local Authority Districts
J03	WD	England and Wales	1961 Census Wards
N08	WD	Northern Ireland	Electoral Wards
S13	WD	Scotland	Electoral Wards
W05	WD	Wales	Electoral Wards
E05	WD	England	Electoral Wards/Divisions

Deduplicating the codelist

If we deduplicate the codelist, there should be no issue, because the geoCode field is unique in itself within the GSS system.
However, if publishers have used other geoTypes and populated the geoCode field as they see fit, it is conceivable that there may be duplicates.

Governance - As per https://standard.threesixtygiving.org/en/latest/about/governance/#versions this will be a PATCH

Updating the codelist

Ideally, the codelist woud use what the GSS calls 'code', the unique 3 character prefix - however, making this change would be a MAJOR change. Updating the codelist within the realm of PATCH means only cleaning up the list.

Work to be done before we know how to update the codelist:

• Why was only a subset of the codes included in this codelist?
• What codes are currenty used?
• How many publishers use the field?
• What is a comprehensive source?
• These codes have changed with time, and may change again - how does this affect the tools using this field? How are publishers getting this data - from the ONS?

[mariongalley]

It seems to me that we have a few options moving forward:

• Update/maintain geocode types list - deduplicate terms, add new ones (on what basis?), remove out-of-date ones
  • If we do this we can lookup the human-readable names associated with the codes in GrantNav like we do for the other codelists
  • We could also validate the codes in this case (backwards-incompatible) - although this would only guarantee that they are drawn from the codelist, not necessarily that they match the geocode provided
• Stop maintaining geocode types list and refer to ONS list instead (as we do with ISO country codes)
  • This also would allow us to validate the codelist, with the same caveats as above
• Deprecate the field entirely
  • If the geocodes themselves allow us to unambiguously determine the type, there's an argument that this field adds no value
  • This also resolves any possible ambiguity from geocode types that do not match the codes provided

[mariongalley]

@michaelwood Katherine agrees that the field is a candidate for deprecation - hence not worth investing effort in updating the list (also this would likely be backwards-incompatible)

[neelima-j]

@mariongalley I'll get a branch ready which deprecates the field and updates the description appropriately.

How should the Data Quality Tool report on deprecation if this field is used in the data - is it a yellow question mark?

@michaelwood @R2ZER0

[mariongalley]

@neelima-j I don't think we're ready to pull the trigger on deprecating the field yet, but yes it's a good question if we do this, how will the DQT represent it

[neelima-j]

Deprecation & governance

The process to follow to deprecate a field can be set in motion as one would propose a PATCH change. Deprecating a field is a less drastic change than a full removal, and it's reasonable to propose it as a PATCH. The governance policy requires the following steps:

1. Draft a proposal for a PATCH version upgrade
2. Make the proposal on the 360Giving forum
3. Notify the Stewardship Committee and technical team via email
4. If no objections are received within one week, the PATCH change will be considered approved

As per the deprecation policy, once we decide to deprecate a field, the next release will mark the field as deprecated. Note the definition of deprecation - While a deprecated feature remains, its use may raise warning messages recommending alternative practices. Deprecated status may also indicate the feature will be removed in the future.
Data using the deprecated field will receive warnings from the 360Giving Data Quality tool.

The following MAJOR release must rename or remove the term from the schema, making it obsolete. the Data Quality tool will now give errors for data with this field.

Related to issues #155 #94

[mariongalley]

It seems to me that we have a few options moving forward:

• Update/maintain geocode types list - deduplicate terms, add new ones (on what basis?), remove out-of-date ones

  • If we do this we can lookup the human-readable names associated with the codes in GrantNav like we do for the other codelists
  • We could also validate the codes in this case (backwards-incompatible) - although this would only guarantee that they are drawn from the codelist, not necessarily that they match the geocode provided

• Stop maintaining geocode types list and refer to ONS list instead (as we do with ISO country codes)

  • This also would allow us to validate the codelist, with the same caveats as above

• Deprecate the field entirely

  • If the geocodes themselves allow us to unambiguously determine the type, there's an argument that this field adds no value
  • This also resolves any possible ambiguity from geocode types that do not match the codes provided

Added a bonus 4th option after discussing with David. We could turn this codelist into a more meta codelist that specified if the geocode is drawn from the GSS (ONS) codes, or another type of geocode (such as non-UK postcodes)