Invalid WKT issue raised for suspected valid WKT

[Ben-Hodgkiss]

Overview
An LPA has reported (via Jira Service Desk) that they are getting invalid WKT issues showing in their LPA Dashboard and the Check tool.

We need to:

• Confirm whether this data is valid - the points provided look fine on first inspection
• We therefore expect that we need to modify the conditions for the invalid WKT issue being raised as it is creating false positive issues.

[Ben-Hodgkiss]

Looking at the column mappings - https://datasette.planning.data.gov.uk/tree/column_field?_sort=rowid&resource__exact=22822b88922f36c1956de3b58363f04f9822c1a3aa3bdb3c4c3d569154c03091 - it appears point is mapped to point. When you look in the 'Check tool' (linked above), you can see that the raw data is not in invalid WKT format and so the issue is being raised correctly.

The issue appears to be that the following is happening:

• The LPA are submitting both geometry and point columns
• In the raw data, the point is invalid WKT
• An issue is being raised for this invalid WKT point. The issue in datasette has the value that is causing the problem (i.e. not in WKT).
• As it has geometry and an invalid point, the pipeline transformation is automatically creating a point based on the centroid of the geometry multipolygon. This is expected behaviour for all geometry facts to create a point if one doesn't already exist.
• The LPA Dashboard task list appears to be pulling through the issue however it is showing the transformed point value (i.e. the one that the pipeline created rather than the original one provided by the LPA).

This may require a conversation with design/providers to try and work out what we want to be happening in the edge-case where we have a correct geometry and an invalid point being provided. Do we want to flag this as an issue at all? If so, we should probably figure out how to display the offending data as provided by the LPA.

[stevenjmesser]

As it has geometry and an invalid point, the pipeline transformation is automatically creating a point based on the centroid of the geometry multipolygon. This is expected behaviour for all geometry facts to create a point if one doesn't already exist.

Are there other scenarios in which a point is created based on the centroid of the geometry?

If we create a point on behalf of an LPA because they’ve provided a geometry, do we need them to provide a point? Maybe the invalid point issue is unnecessary in this scenario.

[stevenjmesser]

Owen and I were just chatting. This may not have come up before because we didn’t have Check and Submit to play these sorts of scenarios back before.

We think the conversation might be better framed as: when do we derive data from other fields and provide this downstream? how do we make a data provider aware of derived data so they can correct it?

It’s all about using the feedback loop the Submit service now allows us.

[Ben-Hodgkiss]

Are there other scenarios in which a point is created based on the centroid of the geometry?

As far as I'm aware, we always create a point if a geometry is provided and a valid point is not.

If we create a point on behalf of an LPA because they’ve provided a geometry, do we need them to provide a point? Maybe the invalid point issue is unnecessary in this scenario.

No, we don't need them to provide a point. Most LPAs either provide a point or a geometry. However, we should be able to support LPAs providing both fields - and support if one (or both) of these have issues.

We think the conversation might be better framed as: when do we derive data from other fields and provide this downstream? how do we make a data provider aware of derived data so they can correct it?

Yes, agreed - we should be making derived data clearer, both on the provider-facing tools and probably on the entity/API itself. Regardless, we do need to work out the best design for this particular bug whereby we display valid issues but with corrected data which then causes confusion.

[Swati-Dash]

@Ben-Hodgkiss I have assigned this ticket to you https://mhclgdigital.atlassian.net/browse/DLSD-1436

[CarlosCoelhoSL]

My investigation agrees with the process that Ben outlined earlier.

The point data is invalid, point data should look like POINT(0 0).
In the two resources downloaded the points look like this:

• -2.1373804, 52.542632 from resource 22822b88922f36c1956de3b58363f04f9822c1a3aa3bdb3c4c3d569154c03091
• POINT (-2.1355831, 52.543164) from resource 4261412a19923d0687631f8800a237378c5c95bb5d1621a0281df54fa1639eb0

The first one is clearly invalid, and the second has a comma in it where there should only be a space between the co-ordinates, so the invalid WKT issues are legitimate.
I'm not sure what you want to do about how this is displaying on check/submit, we can discuss tomorrow @Ben-Hodgkiss?

[DilwoarH]

To see the issue, go to:

• https://submit.staging.planning.data.gov.uk/organisations/local-authority%3ASST/tree/invalid%20WKT/point
• Check service: https://submit.staging.planning.data.gov.uk/check/link?dataset=tree&orgName=South%20Staffordshire%20Council&orgId=local-authority%3ASST with this URL: https://www.sstaffs.gov.uk/sites/default/files/2025-01/South%20Staffs%20TPO%20non%20Zone%20Geometry%20Dataset.geojson

Further investigation found the following issues:

• In submit service, the check service does not stop users from continuing if there is an error (invalid WKT).
• In the issue page, we display the transformed value and not the value that the user uploaded (this value can be found in the issues table).

Action needed:

• Speak to @alextea about displaying user uploaded values
• Speak to @alextea about why we don't stop users from progressing when error happens
• Investigate and understand the logic for errors

[alexglasertpx]

Currently investigating what is happening to the invalid points when they get into the pipelines (are they getting into the parquet files and what form do they take, what is outputted?).