Very choppy playback when building MumbleGuiClient

[trevorirwin]

Hi,

When building MumbleGuiClient with Visual Studio 2019, I'm getting very choppy playback. I would estimate it at a fraction of a second (so maybe a handful of packets at most) followed by silence, ad nauseum. Recording seems to work fine, as other clients are receiving transmissions from MumbleGuiClient loud and clear.

Could this be a severe case of jitter, as in #46, or is this something more extreme? Perhaps an issue with the build?

[martindevans]

Perhaps try putting some logging into the Audio Decoding Buffer to see if it's running out of samples to play back.

[trevorirwin]

@martindevans I’m trying to understand the structure of the Audio Decoding Buffer. Should I be logging in Read()? Am I correct in my understanding that that function moves data out of the “decoded buffer”, and effectively “hands it off” to NAudio?

[martindevans]

Yeah basically encoded audio gets dumped into this buffer when it arrives from the network and then the audio devices pulls from it (in Read) when it needs something to play back.

[trevorirwin]

Is anyone else experiencing this behavior? It would be good to know if the bug is widespread, or if it's just in my build. (For what it's worth, all unit tests passed.)

Some logging reveals that even with constant transmission of a stream of noise (blowing into the mic) on another client, the readCount value is usually zero, and FillBuffer() is usually returning false. readCount is non-zero for one "frame" almost exactly every second. If I stop and start transmitting, it will get a couple more "frames" and then continue with the once-per-second behavior.

edit: Curiously, it looks like the buffer is sized to be about 1 second long... coincidence? I think not.

[martindevans]

That's definitely the problem then, every time FillBuffer() returns false is a failure that results in audio artifacts.

Fixing it ideally requires a jitter buffer (as I described here). A simpler workaround would be to delay the playback of all speech by a fixed amount (say 100ms), to allow the buffer to fill up a bit.

[trevorirwin]

Do you have any insight as to what an implementation of that delay might look like? I'd be happy to do it.

[martindevans]

I'm afraid I can't remember the details of the playback pipeline (it's been a long time since I wrote this). I'd suggest finding all uses of AudioDecodingBuffer and tracing it out from there.

I think that it's implemented as something just reads from that buffer and plays what it gets. In this case what you'd want to do is make the buffer a little more intelligent so you count up writes, and only start returning audio when writes > threshold. As long as reads (happening in realtime) and writes (also happening in realtime) stay in sync you're ok. The tricky bit will be handling when writes stop (due to the remote end stopping transmission), you'll need to detect this case and reset your counter so the buffer has chance to fill up a bit again when they start talking again.

[trevorirwin]

Thanks for the pointers. After some additional logging and experimentation, it looks like the rate at which the _decodedBuffer fills with network data is considerably slower than the rate at which it's consumed, regardless of whether there is any delay or jitter buffer.

If we look at the line where the array _decodedBuffer is defined, we see this:
_decodedBuffer = new byte[sampleRate * (sampleBits / 8) * sampleChannels];
Which when built and run on my machine, comes out to 96000, or twice the sample rate of my default hardware (48000). So this tells me the buffer is intended to hold one second of audio for two channels.

However, if I have the Read method simply call FillBuffer every time it's called and output the number of bytes in the buffer, I find that the buffer only fills at a rate of ~5760 bytes/sec. Which is 1/8th of 48000 46080. Again, not a coincidence I'm sure.

[martindevans]

Ah, that's a bigger issue than a simple jitter/timing problem!

Which application are you using to play back audio (the core MumbleSharp library itself doesn't handle playback iirc)?

[trevorirwin]

I'm building the MumbleGuiClient example so... NAudio? If I'm understanding your question correctly.

[trevorirwin]

Also, math error: 5760 is close to 1/8 of 48000 but not quite. It's actually 46080. Maybe not wise to assume a correlation between those two numbers yet.

[trevorirwin]

So, some notes:
The second client I was using to transmit voice (the existing Mumble for iOS app) was set to use the "Low" quality preset (16 kbit/s, 60 ms audio per packet.)

I tried implementing a delay by counting writes, but maybe I wasn't doing it very well. Somewhere in this process, I set the quality back to "Normal" (40 kbit/s, 20 ms audio per packet.)

I then tried implementing the workaround described in #46. At the "Normal" quality setting, I'm getting good, clear audio! Jitter-free, I think. However, at the "Low" quality setting, I'm getting the behavior I originally described.

So, we've now teased out that the source of my woes are actually two separate issues!

• Without a jitter buffer, audio is unintelligible in the sample MumbleGuiClient
• Audio packet lengths other than 20 ms are not currently supported

I'll open a separate issue for the packet length problem.

edit: After more testing, the "High" quality setting works too. Might be the bitrate and not the packet audio length?

[trevorirwin]

I've implemented a version of the "dynamic" jitter buffer described by kestasjk in #46. My work is in this branch of my fork. This implementation of the "jitter buffer" is contained entirely within AudioDecodingBuffer.cs, rather than relying on Model/User.cs and the EncodedVoiceDelegate, so it shouldn't require any work to set up from end-users of the library.

It needs more testing, and also some cleanup. I'd love some feedback on the following:

1. Should I be using System.Diagnostics.Stopwatch instead of DateTime for measuring the elapsed time? Does precision really matter here? Doesn't seem to.
2. How should the time delay be made configurable? Where should it be "broken out" for access by library users? Should it be static, or configurable per-buffer (and therefore per-Mumble user)?
3. Speaking of time delay, looking at it in more detail, I've noticed that I turn on isJitterDetected when readCount == 0. However, readCount will always be zero when a transmission starts because FillBuffer() will not yet have been called. So, while in theory this could catch readCount dropping to zero mid-transmission and act as a "dynamic jitter buffer", for the most part it is just going to create a constant delay. Huh!