MelSpectrogram() and unspecified sampling rate

[dsplog]

in the meldataset.py, could see that all wav files are resampled to 24000sps. however, as the MelSpectrogram() transform is called without sample_rate argument defaults to 16000sps.

to_mel = torchaudio.transforms.MelSpectrogram(
    n_mels=80, n_fft=2048, win_length=1200, hop_length=300)
mean, std = -4, 4

def preprocess(wave):
    wave_tensor = torch.from_numpy(wave).float()
    mel_tensor = to_mel(wave_tensor)
    mel_tensor = (torch.log(1e-5 + mel_tensor.unsqueeze(0)) - mean) / std
    return mel_tensor

questions :

1. believe 2400sps (vs 16000sps needed) was an oversight ?
2. also, how was the mean/std of -4, 4 arrived.

[gnitoah]

yl4579/StarGANv2-VC#10 and #57, should be helpful.

[magicse]

Hi @dsplog .
HiFi Gan was trained as 16Khz mels (24Khz audio with incorrect sr 16Khz) -> 24 Khz wav and it work something like as fake upsampler.
Therefore, now we have to give 24Khz audio for calculation of the mel spectrogram as 16Khz audio without real resampling.

I think it was a mistake when training the HiFi Gan model (torchaudio.transforms.MelSpectrogram set by default sr = 16Khz). And now it's baked into the pipeline. The correct way is of course that the mel spectrograms should be calculated from 24Khz and after passing through HiFi Gan give 24Khz wave.
To avoid this situation you need retrain all models ASR, JDC and HiFi-Gan with correct mels. Force set sample_rate=24000

to_mel = torchaudio.transforms.MelSpectrogram(
    sample_rate=24000,
    n_fft=2048,
    win_length=1200,
    hop_length=300,
    n_mels=80,
)

I've encountered this problem now as I'm trying to port StyleTTS to C++