Skip to content

Conversation

@danielsreichenbach
Copy link
Member

Summary

This PR completes the implementation of three critical TACT/NGDP features:

  • 40-bit integer support for handling large file offsets and sizes (up to 1TB)
  • ESpec parser for BLTE encoding specifications
  • TVFS compliance for TACT Virtual File System parsing

Changes

Core Features

  • Implemented 40-bit big-endian integer reading/writing in tact-parser/src/ioutils.rs
  • Added complete ESpec parser with support for all encoding types (none, zlib, bcpack, gdeflate, block tables)
  • TVFS manifest parser with proper 40-bit offset/size handling for path, VFS, and container file tables

Performance & Safety Improvements

  • Fixed unsafe string slicing in ngdp-bpsv parser to prevent runtime panics
  • Optimized LRU cache from O(n) to O(1) operations for common cases
  • Replaced Arc<Mutex<u64>> with AtomicU64 for better performance
  • Eliminated duplicate logic between BpsvRow types using traits
  • Consolidated HTTP retry logic in tact-client

Code Quality

  • Added comprehensive test utilities for real WoW data testing
  • Cleaned up exploratory code and temporary test files
  • All tests passing (520+ tests across workspace)
  • Documentation updated

Test Plan

  • All unit tests pass
  • All integration tests pass
  • TVFS tests validate against real WoW 11.0.7 build 58238 data
  • ESpec parser tests cover all encoding types
  • 40-bit integer tests verify max 1TB file support
  • ngdp CLI commands tested and working

## New Features

### Test Utilities (`test-utils` crate)
- Add comprehensive WoW data discovery using environment variables
- Support for WOW_CLASSIC_ERA_DATA, WOW_CLASSIC_DATA, WOW_RETAIL_DATA
- Automatic fallback to common installation paths (Linux, macOS, Windows)
- Convenience macros: `require_wow_data\!()`, `skip_test_if_no_wow_data\!()`
- CI-friendly tests that skip gracefully when no WoW data is available

### Documentation Improvements
- Add comprehensive testing section to main README
- Create detailed test-utils documentation with setup instructions
- Add examples showing real WoW data integration
- Update ROADMAP.md with completed Phase 2.5 (Architecture Improvements)

## Code Cleanup

### Removed Exploratory/Development Files (33 files)
- **Root level**: test-real-data.rs, test-archive-parsing.rs, test_install_manifest.rs, examples/ directory
- **BLTE examples**: test_*.rs, investigate_*.rs, analyze_*.rs files (11 files)
- **Ribbit examples**: debug_*.rs, check_*.rs, analyze_*.rs, explore_*.rs (12 files)
- **Other examples**: exploratory tact-client and tact-parser examples (5 files)
- **Tests**: hardcoded path tests in casc-storage (3 files)

### Consolidated Documentation
- Streamlined TODO.md by moving completed items to ROADMAP.md
- Reduced TODO.md size by ~70% while maintaining important information
- Added Phase 2.5 completion to ROADMAP.md (quality checks, serial tests, examples)

## Integration Examples

### New Files
- `casc-storage/tests/real_data_integration.rs` - Shows environment variable usage in tests
- `casc-storage/examples/list_casc_files.rs` - Example with discovered WoW data
- `test-utils/examples/discovery_demo.rs` - Demonstrates data discovery functionality

## Breaking Changes
- None - all changes are additive or remove development-only files

## Migration Guide
- Tests now use environment variables instead of hardcoded paths
- Set WOW_*_DATA environment variables to enable real data testing
- All tests skip gracefully if no WoW data is available

Closes: Documentation and test data management improvements
- Fix unsafe string slicing in ngdp-bpsv parser to prevent panics
- Optimize LRU cache from O(n) to O(1) for common operations
- Replace Arc<Mutex<u64>> with AtomicU64 for better performance
- Eliminate duplicate logic between BpsvRow types using traits
- Refactor repetitive HTTP retry patterns in tact-client
- Consolidate mutex-protected data structures to reduce lock contention

All tests passing, code ready for release
@danielsreichenbach danielsreichenbach added the enhancement New feature or request label Aug 11, 2025
…mpatibility

- Fixed 17 let-chain patterns across casc-storage crate
- Fixed clippy lifetime elision warning in ngdp-bpsv
- All changes ensure compatibility with Rust 1.86 (MSRV)
- Code now compiles cleanly with both 1.86 and 1.89
- All 520+ tests passing on both Rust versions
@danielsreichenbach danielsreichenbach merged commit 5a97bdc into main Aug 11, 2025
20 of 21 checks passed
@danielsreichenbach danielsreichenbach deleted the feat/40bit-espec-tvfs branch August 11, 2025 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Development

Successfully merging this pull request may close these issues.

2 participants