MWE annotations disappear

[savary]

After having successfully worked with FLAT for almost 10 years, we have a new major issue.

We are currently working with FLAT in a large group of people (about 50) at the UniDive Training School in Chisinau. We are using FLAT for the PARSEME community of multiword expression annotations.

Several people signaled an issue with disappearing annotations. They have been working on a file (in the online mode, not using the queue), submitted some annotations, and when they came back to the same file, the annotations have disappeared.

Our guess is that this might be related to the fact that when we close the annotation window in the browser and then open the same file in another window, the number of connections for the file (seen on the top right-hand corner) increases, although there is only one person in one window working on the file.

It is the first time that we have this issue. Some months ago we made some changes to our environment. Namely, we have:

• upgraded FLAT
• moved it to another server
• added a new configuration called UNIDIVE PILOT 2024

Any idea of what is the problem and how to fix it? We would be very grateful for a hint.

[savary]

Another strange behavior is that one person is annotating her file since the beginning of the day, she is the only one to annotate it, and has this file open only in one window. Still, during the day, she has been observing a varying number of users editing the document (between 1 and 6) although no one else was connected to the same file.

[proycon]

On Thu Jul 11, 2024 at 1:36 PM CEST, Agata Savary wrote: After having successfully worked with FLAT for almost 10 years, we have a new major issue. We are currently working with FLAT in a large group of people (about 50) at the [UniDive Training School in Chisinau](https://unidive.lisn.upsaclay.fr/doku.php?id=meetings:other-events:1st_unidive_training_school). We are using FLAT for the PARSEME community of multiword expression annotations.

Good to hear you're still using it!

Several people signaled an issue with disappearing annotations. They have been working on a file (in the online mode, not using the queue), submitted some annotations, and when they came back to the same file, the annotations have disappeared.

Hmm.. that sounds serious. Data dissappearing is something we never want of course, and there are quite some safeguards against this. If I understand correctly, this is a problem that does not always happen? I want to rule out that all annotations are lost, which would indicate a problem in writing to disk.

Our guess is that this might be related to the fact that when we close the annotation window in the browser and then open the same file in another window, the number of connections for the file (seen on the top right-hand corner) increases, although there is only one person in one window working on the file.

Hmm, I can reproduce this number going up by too much indeed, that seems a new bug I hadn't noticed before. I'll look into it. But I'm not sure whether it's related to annotations actually getting lost though. I'd expect that's either due to the annotation never being registered correctly in the first place for some reason, or as I said earlier, an issue in writing to disk. What would help a lot is having some logs (via `docker logs`, assuming you use docker), ideally start the FLAT container anew with `docker run --env FLAT_DEBUG=1` for verbose logging and mark the times/files when an annotator notices something is wrong. Then we can see if there are some anomalies there that could explain lost annotations. I'll try some other tests in the meantime to see if I can reproduce this.

It is the first time that we have this issue. Some months ago we made some changes to our environment. Namely, we have: - upgraded FLAT

Do you know what version you were on before the upgrade, I assume you're on the latest v0.11.5 now?

- moved it to another server

If writing to disk is the in fact issue, it could be a file permission thing possibly or git setup (if you use the git backend). But this is just a guess at this point.

- added a new configuration called [UNIDIVE PILOT 2024](eme-support/blob/master/parseme-mwe-pilot-unidive-2024.foliaset.xml)

(That link doesn't work, but I doubt a set definition is the cause)

[savary]

Hi Maarten,
Sorry it took me so long to answer due to the holiday period.

Data dissappearing is something we never want
of course, and there are quite some safeguards against this. If I
understand correctly, this is a problem that does not always happen?

No. I have just checked: I made an annotation to a file, I closed the window, then I came back to FLAT and the annotation was still there. Also, when I I logged out and logged in again, the annotation was still there (but each time I come to the same text, the number of people working on the file (in the upper right-hand corner) increases.

What would help a lot is having some logs (via docker logs, assuming you
use docker),

I have just logged in to the server.
First I see a message: => There is 1 zombie process. (unsure if that can be an issue)

Yes, the server run in a docker image.
I got the docker logs from the period when the problem occurred (3 days) but this 45 MB.
Grepping on (err |can|prob|disk|write|abl|crit) gives one line:
{"log":"[09/Jul/2024:15:25:41] ENGINE socket.error 'cannot read from timed out object'\n","stream":"stdout","time":"2024-07-09T13:25:41.383571853Z"}

I also filtered the logs concerning one of the persons who had the issue with disappearing annotations (1.7 MB) but I see nothing special.

ideally start the FLAT container anew with docker run --env FLAT_DEBUG=1 for verbose logging and mark the times/files when an
annotator notices something is wrong. Then we can see if there are some
anomalies there that could explain lost annotations. I'll try some other
tests in the meantime to see if I can reproduce this.

I will try this when the problem starts occurring again.

Do you know what version you were on before the upgrade,
I think it was 0.9.1

I assume you're on the latest v0.11.5 now?

How do I know my current FLAT version.

• added a new configuration called UNIDIVE PILOT 2024
  (That link doesn't work, but I doubt a set definition is the cause)

Sorry, this one

Another thing I remembered is that I was defining new groups and moving accounts among groups when people we annotating. Could this be a problem?

Otherwise, I think I will try to restart the server and wait until the problem occurs again.

[proycon]

Hi Agata,

Sorry it took me so long to answer due to the holiday period.

Sorry, same here!

No. I have just checked: I made an annotation to a file, I closed the window, then I came back to FLAT and the annotation was still there. Also, when I I logged out and logged in again, the annotation was still there (but each time I come to the same text, the number of people working on the file (in the upper right-hand corner) increases.

Ok, good that it's working normally. Still odd and scary that it fails sometimes, though. I still have to look into what broke that counter.

I have just logged in to the server. First I see a message: _=> There is 1 zombie process._ (unsure if that can be an issue)

Probably not, that could be anything on the server.

Yes, the server run in a docker image. I got the docker logs from the period when the problem occurred (3 days) but this 45 MB. Grepping on (err |can|prob|disk|write|abl|crit) gives one line: {"log":"[09/Jul/2024:15:25:41] ENGINE socket.error 'cannot read from timed out object'\n","stream":"stdout","time":"2024-07-09T13:25:41.383571853Z"}

Not too suspicious, timeouts may happen when connections are cut.

I also filtered the logs concerning one of the persons who had the issue with disappearing annotations (1.7 MB) but I see nothing special.

If you see nothing in the logs, I wonder if perhaps the entire server had an unclean shutdown (power cut even?) and the processes never got to write to disk. But that is not the most likely scenario either.

How do I know my current FLAT version.

It should say in the footer in the interface.

Another thing I remembered is that I was defining new groups and moving accounts among groups when people we annotating. Could this be a problem?

I don't think so, the underlying foliadocserve that writes the files to disk does not depend on that user authentication layer.

Otherwise, I think I will try to restart the server and wait until the problem occurs again.

Yes, restarting the container with that debug option I mentioned is the best bet at getting enough data to debug the problem when it occurs again.

[proycon]

I wonder, has this issue re-occurred lately?

[savary]

We continue tracking FLAR issues. No vanishing annotations were reported to us since last time but there is some server instability. I will talk about it is a new issues.