UI not performing as expected - jobs not displaying in ingested (or staged) list after attempted ingest

[atcooper1]

Ingested Job #675 on Friday at approx. 12:00pm, at 5:00pm server error msg was visible and the job was still ingesting. Monday morning and the UI is still displaying the ingesting screen and server busy error. Refreshed screen and re-logged in, the job has disappeared from the staged list but is also not present in the Ingested jobs list. A quick check reveals the observations seem to exist in the DB, however with no feedback from the UI, it's hard to check that all observations were ingested as expected.

Other recent UI issues include:

• Progressively longer ingest times causing significant workflow issues.
• "Save and Validate" when the server is busy (due to backend processes?) causing UI to save blank jobs (i.e. Job 645 and Job 619)
• WoRMS/NRMN searches in New Observations not always returning results as expected. I.e. user has to refresh page 2-3 times before results display (server timeout?)

[utas-raymondng]

Hi Toni,

I have a quick check, for job 675 it is called "RLS QLD Cairns to TI combined 230908.xlsx", which appears in INGESTED.

regards
Raymond

[utas-raymondng]

Meta data also shows correct record

[atcooper1]

Thanks Raymond.
Before closing this, it would be great to resolve the progressively longer ingest times (and server error messages) as this is significantly affecting workflow? Thanks

[vfisaac]

Hi Toni,
We will create an internal backlog issue to look at possible improvements to the long ingest times. This work will most likely involve significant effort and major changes and as such we would be more comfortable undertaking this work when Bene is returned from leave to review and test any proposed changes to the ingest process. As such I would like to schedule this work for the following iteration ie. starting the 13th August.

[utas-raymondng]

Hi Raymond,

I’ve done some testing on some larger sheets and in summary:

Save and validate is taking 40-60 seconds regardless of job size, which seems to be slower than the prod UI
Ingest is taking approx. 6 minutes for larger sheets, which is faster than prod UI

Therefore, ingest speed is improved in Systest, but the save and validate process (which is used very regularly in the UI) is slower – can this be improved at all to match Prod?

Cheers,
Toni

Followup ticket https://github.com/aodn/backlog/issues/6118, after initial improvement.