Improved Speech2IPA Streaming

[Reuben1987AI]

Hey guys, I tested the /stream endpoint with websockets and got pretty bad results compared to regular transcription, my guess is that the endpoint is breaking the audio in arbitrary pieces which breaks the context of the model for that word or phrase.

It seems to me that one way to make this works could be to detect silence either on the client or the server and slice the audio there, so each piece of audio is determined by the user silences.

Just wanted to know what you think, have a good day :)

[Reuben1987AI]

Another thing that could be done to speed up things is to do transcoding on the client to raw PCM on the javascript worklet to remove load from the server, I did some tests and got pretty good results on speed since I can feed the audio directly to the model on the server, I don't even need to stream for short audios anymore. For long texts it still takes a noticeable time running on local CPU.

[SanderGi]

The streaming logic definitely needs improving. We are working on a version that utilizes the parallelizable nature of Wav2Vec2 to accomplish this without having to split at silence (splitting at silence can become problematic with background noise, and doesn't work if the user speaks continuously but we still want gradual results)

[SanderGi]

Another thing that could be done to speed up things is to do transcoding on the client to raw PCM on the javascript worklet to remove load from the server, I did some tests and got pretty good results on speed since I can feed the audio directly to the model on the server, I don't even need to stream for short audios anymore. For long texts it still takes a noticeable time running on local CPU.

The transcoding to PCM already happens on the client and audio is also fed directly to the model. It does need some improvement to increase browser support, especially for the sample rate in Firefox. Do note that our use case requires streaming results in near-real-time as the user is speaking so we can color in syllables as they are being said (see it in action here). This means we don't want to remove streaming, even for short audio. If you want to submit a PR that improves the audio worklet, especially in regards to browser support, that would be very welcome. The word detection model that highlights which word is currently being said, could also use improvement

[Reuben1987AI]

I see, thank you for the in-depth explanation. I'm a fullstack dev so anything that is not training AIs I can do. I would really like to use your service in the near future, watching funny videos or clips of a series and actually learn the language sounds great. @arunasrivastava if its ok with you can I (at least try) to do some of those? Or maybe you can point me to a good first issue?

[SanderGi]

It's probably easier to have one person tinkering with the streaming endpoint at a time. I have added some good first issues that you can take a look at. Feel free to follow up with questions on any of them. I'll also add other good issues that don't touch the same part of the codebase as I think of them