Web UI sometimes says "Old version" is "Available in IATI Datastore" when it's not.

[jarofgreen]

Brief Description
Web UI sometimes says "Old version" is "Available in IATI Datastore" when it's not.

Severity
Medium

Issue Location
https://validator.iatistandard.org/report/dipr-998
This document is not actually available.

Details
In this case, it has happened because yesterday we rebuild the Solr cores from scratch, using the "solrize_reindex" option.
This document was not included in that rebuilding process.
However, as the only check to show that "Old version" label is if "last_solrize_end" is set, it falsely shows it.

validator-web/src/utils/document.js

Line 112 in 00c6ea3

if (last_solrize_end) {

[jarofgreen]

(And maybe document anything we do here in IATI/iati-datastore-docs#14 )

[jarofgreen]

Fix current situation

unifiedplatform=> select count(*) from document where last_solrize_end < '2025-04-15 08:00:00' and solrize_end is null;
 count 
-------
   276
(1 row)
unifiedplatform=> select count(*) from document where last_solrize_end < '2025-04-15 08:00:00';
 count 
-------
   276
(1 row)

And that 276 includes the example above.

We could run

update document set last_solrize_end = Null where last_solrize_end < '2025-04-15 08:00:00';

And that should clear "Old version" from pages like that.

Process fix

In the future, when rebuilding a core from scratch part of the process could be to run first:

update document set last_solrize_end = Null;

However,

1. This can be done in an emergency situation, and it would be easy to overlook in an emergency
2. I suspect ....

Overhaul criteria for showing "Old version"

I suspect, looking at the if (last_solrize_end) { and comparing it to the selection criteria for resolizeing in https://github.com/IATI/refresher/blob/develop/src/library/db.py#L345 that if (last_solrize_end) { is actually to simplistic and should be overhauled. However I'm not immediately sure with what - need to review the existing pipeline process first.

[jarofgreen]

I ran:

update document set last_solrize_end=null where last_solrize_end < '2025-04-15 08:00:00' and solrize_end is null;

And that cleared the example above.

Leaving issues open to look into this more.

[jarofgreen]

So I think there are only a few ways to solve this:

Suggestion 1

We change our process, and when we rebuild a solr core from scratch a new step must be done first:

update document set last_solrize_end = Null;

Suggestion 2

The postgres database can't know the state of the solr datastore as a result of an operation done directly on solr, so the only way for validator-web to know is not to use the last_solrize_end column but instead do a query on solr to see if any data is there.

This adds extra overhead and a performance hit - for a publishers page we'd want to find a way to only query solr once, not once per dataset.

Suggestion 3

I suppose a regular cron job can query solr and update the postgres database if something is missing, but that's getting a bit much.

Proposal

I think we go for way 1, given the work and complexity in other ways and the lower impact if way 1 is forgotten.