Skip to content

Add support for phoneme literals in the tokenizer#36

Merged
Lyrcaxis merged 2 commits intoLyrcaxis:mainfrom
PhilippNaused:phonemes-literal
Jul 22, 2025
Merged

Add support for phoneme literals in the tokenizer#36
Lyrcaxis merged 2 commits intoLyrcaxis:mainfrom
PhilippNaused:phonemes-literal

Conversation

@PhilippNaused
Copy link
Contributor

Fixes #35

Tells the tokenizer to not use espeak on parts look like this: [Kokoro](/kˈOkəɹO/).
It should translate everything before and after that using espeak, but insert this part as kˈOkəɹO just before tokenizing.

I'm using the format from misaki because it's easy to detect using regex, but I'm open to suggestion for a better pattern.

@PhilippNaused PhilippNaused marked this pull request as ready for review June 12, 2025 10:24
Copy link
Owner

@Lyrcaxis Lyrcaxis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice feature! But let's make it only work when users explicitly enable it.

@Lyrcaxis Lyrcaxis self-requested a review July 18, 2025 12:57
Copy link
Owner

@Lyrcaxis Lyrcaxis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, sorry about the slow review time, I've been busy.

Looks good! Awesome feature :) Will have to update the README to display this functionality.

Some future changes that would be nice are:

  1. Currently, SpeechGuesser's LowEffort mode will take the raw text + pronounciation in account for its calculations -- without any transformations. Would be good to have the higher-effort modes account for the pronounciation part or map them better (to more characters).

  2. Add some way to 'validate' that the user indeed wants the transformation to take place (although the current syntax is quite safe -- hard to happen by accident).

Will include in v0.6.2 along with the japanese + mandarin improvements👍

@Lyrcaxis Lyrcaxis merged commit c2ce4ba into Lyrcaxis:main Jul 22, 2025
1 check passed
@PhilippNaused PhilippNaused deleted the phonemes-literal branch July 22, 2025 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adding phonemes to input text.

2 participants