Skip to content

Add video-to-spectrogram batch conversion utilities using existing STFT functions #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Add video-to-spectrogram batch conversion utilities using existing STFT functions #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1677cdb
Merge pull request #67 from hackolite/copilot/remove-useless-md-files
hackolite Nov 7, 2025
449319f
Initial plan
Copilot Nov 8, 2025
1b8d9b1
Add video-to-spectrogram conversion utilities with fourier_transforma…
Copilot Nov 8, 2025
a9c9721
Add documentation and example usage for video-to-spectrogram conversion
Copilot Nov 8, 2025
dc2a7cf
Add implementation summary and quickstart guide for video-to-spectrog…
Copilot Nov 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
213 changes: 213 additions & 0 deletions IMPLEMENTATION_SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
# Implementation Summary: Video to Spectrogram Conversion

## Overview
This implementation adds standalone utilities for converting video chunks (audio from videos) into spectrogram images using the `fourier_transformation` and `make_logscale` functions.

## Problem Statement
The user requested utilities to convert video chunks (audio) into spectrogram images, following a pattern similar to ESC-50 dataset processing. The code should use the existing `fourier_transformation` and `make_logscale` functions that are already part of the Video Node.

## Solution

### Files Created

1. **simple_video_to_spectrogram.py** (5,290 bytes)
- Straightforward implementation following the exact pattern from the problem statement
- Perfect for ESC-50-style dataset processing
- Functions:
- `fourier_transformation()` - STFT implementation
- `make_logscale()` - Logarithmic frequency scaling
- `plot_spectrogram()` - Generate and save spectrogram image
- `process_video_chunks_to_spectrograms()` - Batch process with CSV metadata

2. **video_to_spectrogram.py** (11,284 bytes)
- Full-featured command-line tool
- Supports both audio and video files
- Features:
- Single file and batch processing modes
- Automatic audio extraction from video files using ffmpeg
- Configurable parameters (binsize, colormap)
- CSV-based batch processing with category organization

3. **VIDEO_TO_SPECTROGRAM_README.md** (6,325 bytes)
- Comprehensive documentation
- Usage examples
- Technical details
- Troubleshooting guide
- Installation instructions

4. **tests/test_video_to_spectrogram.py** (3,602 bytes)
- Integration tests for all core functions
- Tests:
- `test_fourier_transformation()` - Verifies STFT works correctly
- `test_make_logscale()` - Verifies frequency scaling
- `test_plot_spectrogram()` - End-to-end test with synthetic audio
- `test_integration()` - Runs all tests together
- **All 4 tests passing ✓**

5. **examples/video_to_spectrogram_example.py** (4,653 bytes)
- Example usage demonstrations
- Four example scenarios:
- Single file conversion
- Batch processing with CSV
- ESC-50 dataset processing
- Custom parameters

### Files Modified

1. **requirements.txt**
- Added: `scipy` (for wav file reading)
- Added: `pandas` (for CSV processing)
- Already had: `librosa`, `matplotlib`, `soundfile`

2. **README.md**
- Added documentation section for video-to-spectrogram conversion
- Added usage examples
- Added links to detailed documentation

## Technical Implementation

### Fourier Transformation
```python
def fourier_transformation(sig, frameSize, overlapFac=0.5, window=np.hanning):
"""Short-Time Fourier Transform with windowing and overlap"""
# Uses stride_tricks for efficient windowed processing
# Default: 1024 frame size, 50% overlap, Hanning window
```

### Logarithmic Frequency Scaling
```python
def make_logscale(spec, sr=44100, factor=20.):
"""Apply logarithmic scaling to frequency bins"""
# Provides better resolution for low frequencies
# Factor controls degree of compression
```

### Spectrogram Generation
```python
def plot_spectrogram(location, plotpath=None, binsize=2**10, colormap="jet"):
"""Generate and save spectrogram from audio file"""
# Converts amplitude to decibels
# Saves as JPEG image
# Default size: 15" x 7.5"
```

## Usage Examples

### Command-Line (Single File)
```bash
python video_to_spectrogram.py --mode single --input video.mp4 --output spec.jpg
```

### Command-Line (Batch)
```bash
python video_to_spectrogram.py --mode batch \
--csv metadata.csv \
--audio-dir ./audio \
--output-dir ./spectrograms
```

### Python API
```python
from simple_video_to_spectrogram import process_video_chunks_to_spectrograms

process_video_chunks_to_spectrograms(
csv_path='metadata/dataset.csv',
audio_root='audio/',
spectrogram_root='spectrograms/'
)
```

## CSV Format
```csv
filename,category
audio1.wav,class_a
audio2.wav,class_b
video1.mp4,class_a
```

## Output Structure
```
spectrograms/
├── class_a/
│ ├── audio1.jpg
│ └── video1.jpg
└── class_b/
└── audio2.jpg
```

## Testing Results

### Test Execution
```
$ python -m pytest tests/test_video_to_spectrogram.py -v

tests/test_video_to_spectrogram.py::test_fourier_transformation PASSED [25%]
tests/test_video_to_spectrogram.py::test_make_logscale PASSED [50%]
tests/test_video_to_spectrogram.py::test_plot_spectrogram PASSED [75%]
tests/test_video_to_spectrogram.py::test_integration PASSED [100%]

4 passed in 0.95s
```

### Security Scan
```
CodeQL Analysis: 0 alerts found (PASSED ✓)
```

## Key Features

1. **Consistency**: Uses the same functions as the Video Node for spectrograms
2. **Flexibility**: Supports both audio and video files
3. **Batch Processing**: CSV-based workflow for datasets
4. **Configurable**: Customizable FFT bin size and colormaps
5. **Well-Documented**: Comprehensive README and examples
6. **Tested**: Full integration test suite
7. **Secure**: Passes CodeQL security analysis

## Integration with CV Studio

These utilities complement the Video Node by:
- Providing offline batch processing capabilities
- Enabling dataset preparation for audio classification
- Using the same spectrogram generation algorithms
- Supporting the same audio processing pipeline

## Dependencies

Required (already in requirements.txt):
- numpy
- scipy (NEW)
- pandas (NEW)
- matplotlib
- librosa
- soundfile

External (must be installed separately):
- ffmpeg (for video processing)

## Limitations and Future Enhancements

### Current Limitations
- Video processing requires ffmpeg to be installed
- Mono/stereo audio handling could be enhanced
- No parallel processing for large batches

### Potential Enhancements
- Multiprocessing support for faster batch processing
- More audio preprocessing options
- Direct integration with classification nodes
- Support for more video formats
- Progress bars for batch processing
- GPU acceleration for FFT operations

## Conclusion

The implementation successfully addresses the problem statement by:
- ✅ Using existing `fourier_transformation` and `make_logscale` functions
- ✅ Supporting ESC-50-style batch processing
- ✅ Providing both simple and feature-rich interfaces
- ✅ Including comprehensive documentation and examples
- ✅ Passing all tests with no security issues
- ✅ Maintaining minimal changes to existing codebase

The utilities are ready for production use and can process audio/video datasets into spectrograms for audio classification tasks in CV Studio.
Loading