feat: comprehensive multilingual word error correction system #519
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KristjanESPERANTO
force-pushed
the
feat/gen_word_error_correction
branch
from
10f5a31 to
ffa3e14
Compare
HolgerJeromin
left a comment
HolgerJeromin
left a comment
There was a problem hiding this comment.
Nice!
I was not aware the Intl API is available in node.
Collaborator
Author
Yes, that's cool. So we don't need any external dependencies for that. It just feels a bit wrong to use brute force to find out which languages are supported by Intl. |
Yeah, but js engine local only and at build time. So 🫣 |
HolgerJeromin
approved these changes
Aug 8, 2025
KristjanESPERANTO
force-pushed
the
feat/gen_word_error_correction
branch
from
fe9b4c9 to
95b6166
Compare
…r_correction.js``
KristjanESPERANTO
force-pushed
the
feat/gen_word_error_correction
branch
from
95b6166 to
b1fd4ae
Compare
…endencies - Replacement for removed non-functional concept script gen_word_error_correction.js with production-ready implementation - Without external dependencies - Use native Intl.DateTimeFormat for date/time formatting - Use native Intl.DisplayNames for dynamic language name resolution - Add dynamic locale discovery covering 140+ languages - Implement ambiguous word detection with warning system
These entries are now automatically generated.
…ion system details
… and ambiguous words
- Increase supported languages from 146 to 244 - Maintain low conflict rate with only on additional ambiguous word detected
KristjanESPERANTO
force-pushed
the
feat/gen_word_error_correction
branch
from
b1fd4ae to
d573aa3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is replacing the non-functioning concept script (
gen_word_error_correction.js) with a working one (gen_word_error_correction.mjs) that is integrated into the build process 🥳It retrieves all month and week names from the Intl API and packs them together with the manual definitions (in
word_error_correction_manual.yaml) intoword_error_correction.yaml.This means that we support over 240 languages with that, compared to only a handful previously 🤯 And we don't even have to worry about maintaining the strings, as they are always queried dynamically during the build 😁
Example
Before, the string
月曜日-金曜日 09:00-17:00was not usable.Before
Prettified: not possible
Warnings:
After
Prettified:
Mo-Fr 09:00-17:00Warnings:
Short names
With my last commit, I also added short names. However, I had to filter out ambiguous names because there were too many of them. Without filtering them out, we would receive a large number of such warnings:
I'm really glad to be creating this PR now. It took me a lot of time 😴. Since the tests look good, I'd actually like to merge it right away. But since it's quite a significant change, I'll wait a little for your feedback @ypid 🙂