Skip to content

Conversation

@acadarmeria
Copy link
Contributor

Previously, speech_edit.py worked in wav domain (inserting zeros into the waveform before computing mel spectrogram), which caused boundary artifacts when mel spectrogram windows straddled zeros and real audio.

This commit refactors the approach to work in mel domain:

  • Compute mel spectrogram on the clean original audio first
  • Insert zero frames in mel domain instead of zero samples in wav domain
  • Use frame-level granularity throughout for consistency

Benefits:

  • Eliminates boundary artifacts
  • More consistent behavior regardless of small float variations in input times
  • Cleaner edit boundaries

Changes to speech_edit.py (lines 148-220):

  • Convert audio to mel using model.mel_spec() before editing
  • Build mel_cond by concatenating original mel frames + zero frames
  • Calculate all time-based values at frame level first, then convert to samples
  • Pass mel_cond directly to model.sample() instead of raw audio

Previously, speech_edit.py worked in wav domain (inserting zeros into the waveform before computing mel spectrogram), which caused boundary artifacts when mel spectrogram windows straddled zeros and real audio.

This commit refactors the approach to work in mel domain:
- Compute mel spectrogram on the clean original audio first
- Insert zero frames in mel domain instead of zero samples in wav domain
- Use frame-level granularity throughout for consistency

Benefits:
- Eliminates boundary artifacts
- More consistent behavior regardless of small float variations in input times
- Cleaner edit boundaries

Changes to speech_edit.py (lines 148-220):
- Convert audio to mel using model.mel_spec() before editing
- Build mel_cond by concatenating original mel frames + zero frames
- Calculate all time-based values at frame level first, then convert to samples
- Pass mel_cond directly to model.sample() instead of raw audio
@SWivid SWivid merged commit 27e20fc into SWivid:main Dec 26, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants