Document why Montenegrin (sr-ME) is mapped to "srp" and not to "cnr"

[sebastian-nagel]

@pjox, @laurieburchell

CLD2's detected language Montenegrin (sr-ME) is mapped to Serbian ("srp") and to to "cnr" because several code comments and the 2014 CLD2 release notes state that there was little training data for it and that it is recognized as Serbian by CLD2.

However, a text recognized as Serbian with a URL in the .me top-level domain, may be classified as sr-ME. The annotations in the WARC metadata record than include sr-ME.

This was discussed here.

[laurieburchell]

I'm reading back through the links you've presented, but is it as simple as that the cnr code was only introduced in 2017 (https://iso639-3.sil.org/code/hbs)?

[laurieburchell]

As a more general point, I think that the only way a language classifier would be able to distinguish between Bosnian/Serbian/Croatian/Montenegrin would be by TLD or possibly script (I believe Serbian is the only one to commonly use the Cyrillic script these days). @sebastian-nagel likely has more informed opinions on this though.

[sebastian-nagel]

the cnr code was only introduced in 2017

Yes, but that's only the label. The Montenegrin language was definitely a standard language when CLD2 was trained, it's the official language of Montenegro since 2007. So, the authors of CLD2 knew about it, but used "sr-ME" as code, although the detector has issues detecting Montenegrin.

Here, it's about the mapping from CLD2 to iso-639-3 codes.

[laurieburchell]

After some further research on Montenegrin and Montenegro, I discovered that a plurality of Montenegrins refer to the language they speak as "Serbian" (https://en.wikipedia.org/wiki/Languages_of_Montenegro with a link to the census). Therefore, I suspect that the lack of ISO recognition of Montenegrin in 2014 plus the closeness of Serbian/Montenegrin (and difficulty in distinguishing them) is why the CLD2 authors decided to use "sr-ME" as the code.

Reading more of the comments in CLD2, it looks like they use a weighting system to distinguish close pairs, but that there was some kind of implementation problem with distinguishing between four close languages rather than two (e.g. this comment). I would need to spend more time understanding CLD2 to understand why though.