Can I get timestamp alignment error information

[ozancaglayan]

Hello!

I have two sets of subtitles and I just want to compute how well the second one (hypothesis) covers the region boundaries in the first one (reference). I'm not interested in the actual text pieces within the regions. Do you think SubER will be able to do this if I change the format of the SRT files with some fake content?

Thanks

[patrick-wilken]

Hi,

I assume by "cover the region boundaries" you mean for what portion of the reference subtitle time intervals there is a hypothesis at all.
No, not really. Because SubER only checks whether or not whole subtitles overlap in time. (This is because we found heuristics like interpolating word-level times as done in t-BLEU to be too unreliable.) Let's say you replaced the text of each subtitle with one "DUMMY" token, then SubER would be a measure of how many reference subtitles are "missed completely", how many hypothesized subtitles are "not at all" in the reference, and also how well you match the number of subtitles where there is overlap. But it wouldn't account for how large the overlap is. E.g. hypothesizing a subtitle that (in terms of timing) covers only half of the reference subtitle wouldn't be penalized.

For that you would want something more fine-grained. Basically the total duration where hypothesis and reference disagree whether any subtitle is "on", divided by the total reference duration, right? (Or precision/recall variants of that.) That would actually be a useful additional metric to have in the tool. But it's not directly related to SubER or the other text-based metrics.

[ozancaglayan]

Hi,

Many thanks! Yes, my objective is to measure the time alignment and/or coverage of two sets of subtitled regions basically. I found another tool which seems to be computing some metrics relevant to that, will explore that one as well
https://github.com/fyvo/EvalSubtitle

[patrick-wilken]

Good tool, but I think also there you will find only token-based metrics, nothing that explicitly looks at the time stamps.

"Time alignment" could also mean how far (in seconds) your time stamps are off the reference. I thought about measuring that, but it's not well defined if there is no one-to-one correspondence between hypothesis and reference subtitles. I guess one could get creative here to find a meaningful score.
But coverage/overlap really makes sense to be calculated as I described above.

[patrick-wilken]

So, I did a bit of research. 🙂 Timing overlap recall/precision is indeed widely used for voice activity detection, which is very related. Subtitles intentionally do not always correspond to the exact speech start and end times, and some subtitles may contain non-speech, but the evaluation of timings could be done in the exact same way. One popular implementation is pyannote: https://www.isca-archive.org/interspeech_2017/bredin17_interspeech.pdf
I have a first version of adding it to the SubER tool here: #10 Feel free to try. I didn't update the dependencies yet, so requires a manual pip install pyannote.metrics.

[patrick-wilken]

There are also other metrics which could be interesting. pyannote has segmentation recall / precision which evaluate exact timing matches of boundaries (within a certain tolerance interval) rather than overlap: https://pyannote.github.io/pyannote-metrics/reference.html#segmentation
There is also "coverage" / "purity", I don't have an intuition yet whether they are useful for subtitling.

Also, people use more fine-grained distinctions of false positives / false negatives depending on whether they occur at the start, end or in the middle of speech / silence segments, e.g. section 5 of: https://www.eurasip.org/Proceedings/Eusipco/2002/articles/paper512.pdf
This should actually be relevant for subtitling, because placing of subtitles is done very precisely by professionals depending on start/end of speech, but also reading speed, scene changes, and surrounding consecutive subtitle blocks.
But that's probably more detailed than what you were asking for. 😄

[ozancaglayan]

Hi there,

Sorry for the late reply, I got sidetracked a bit on this but I'll definitely come back to it at some point. Feel free to convert this issue to a discussion if you like.

Many thanks for the inputs and even the implementation 🙏