Remove language limits

[cutefluffyfox]

Hi!

I've noticed your project and goals you try to achieve with it, and I found it curious why there is such strict limit on languages (support only on en/ru). I see that you used spacy to tokenize text, and this idea seemed a bit off to me. I'd agree that spacy tokenization can be beneficial for some use-cases, it seems a bit out-of-place for a project that aims to simplify NER and language understanding pipelines as you are heavily limiting opportunity for others to use your solution. I am not sure whether Spacy is a crucial step in your pipeline, but I want to highlight that it only supports 24 languages out of the box.

Are there any plans to expand supported language set or remove such limitation entirely?

[gregarshinov]

@cutefluffyfox Hey! Thank you for this suggestion! Current release works best with Russian and English data due to limitations around handling NER format conversions correctly and using LLM-features (that are currently tied to language specific prompts, that were thoroughly tested on real data).
Right now you can choose between any BERT and Deberta base checkpoints by redefining model_name_or_path in config_overrides. However, unfortunately, LLM-features and AncSetFIt are not guaranteed to work in best way possible if you feed them data in languages other than Russian or English. To make language independence happen we need to make prompt redefinition possible through configuration. We plan to enhance the UX by showing warnings in case a user tries using language dependent features on non-supported languages. You are welcome to contribute!