algorithm-implementation #4
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
text normalization algorithm with TextNormalization, WhitespaceNormalization, RepetitionNormalization, InvisibleCharacterNormalization functions. Implemented leetspeak normalization with LeetspeakNormalization and IsPrimarlyCyrillic functions. Implemented word checker with Tokenization and Hash-Trie building with further checking with Wordchecking and Tokenize functions. Currently they are NOT adapted to the moderation-service environment and require further adjustments and combination with proto and with the database
added united namespace TextProcessor for optimal usage of all new functions. Implemented the text processing in the service. The attempt was successful
Closed
added model folder with constants used by the algorithm, cleaned up functions
kataevandrey
force-pushed
the
algorithm-implementation
branch
from
36082c0 to
25cc18c
Compare
kataevandrey
approved these changes
Oct 30, 2025
…t/moderation-service into algorithm-implementation
now grpc stubs are being generated directly by cmake to avoid any versioning issues
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.