feat: Add Toggleable Enhanced Noise Suppression Using RNNoise#783
Open
Dadadah wants to merge 8 commits intostoatchat:mainfrom
Open
feat: Add Toggleable Enhanced Noise Suppression Using RNNoise#783Dadadah wants to merge 8 commits intostoatchat:mainfrom
Dadadah wants to merge 8 commits intostoatchat:mainfrom
Conversation
Imports livekit-rnnoise-process, this package requires an externally hosted rnnoise model. This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
Buttons should now only turn off when the other turns on only. This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
Member
Author
|
The livekit-rnnoise-processor package should not have webpack as a dependency, I'll fix that this evening. |
This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
New version does not use webpack as a dependency and allows stubbing in a cdn url for the worklet This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
This was accidentally commited. This commit was made without the use of generative AI. Signed-off-by: Jacob Schlecht <dadadah@echoha.us>
Member
Author
|
Removed webpack dependency, removed the translations as I didn't know that process was automatic (new to a lot of this tooling), added the option to pass in a CDN URL to the processor to allow a safe worklet host via Stoat's maintainer team. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds enhanced noise suppression using RNNoise. It adds a package that I have refactored and put up on npm. Here's the link to the package:
https://www.npmjs.com/package/livekit-rnnoise-processor
The RNNoise model + worklet is a little under 2MB, it only downloads when a user connects to voice with Enhanced Noise Suppression enabled.
What is RNNoise and how does this PR use it
RNNoise is a noise suppression library based on a recurrent neural network. It is open source under the BSD-3 license and the source can be found here. This PR uses RNNoise by downloading via CDN a WASM compilled lite model of the library. The library is compiled with Emscripten, added to a worklet, and hosted by CDN. The worklet is then sent to the AudioWorklet API whenever the processor is used in a livekit track. The worklet takes packets from the input, multiplies the bitrate to 48khz as required by RNNoise, sends the packets through RNNoise, then redivides the bitrate down to it's original format and sends it back to the worklet api.
Notes
This package requires importing the following RNNoise model from cdn: dist/DenoiserWorklet.js. This points to my github repo using jsdelivr. The package allows passing a different CDN URL on processor creation. The worklet can be compiled by a maintainer, uploaded to a Stoat CDN, and passed to the processor so it will load that version instead.
Added a toggle switch that disallows both types of noise suppression at once.
Added translations for the switch as well.
One of the changes here is a change of when the microphone channel is enabled. Previously, the microphone channel was enabled immediately upon connecting to livekit. This caused the audioContext on the microphone stream to be overwritten with an undefined context. Processors require an audio context. In this pr, I have made it wait until the client is connected before enabling the microphone stream. This makes a muted icon appear on the user icon when joining a voice channel until they are fully connected.