Epic: libmagic-rs Phase 1 MVP

[unclesp1d3r]

Summary

Complete Phase 1 MVP of libmagic-rs, a pure Rust implementation of libmagic for file type detection. The core infrastructure (AST, parser components, evaluator, I/O, output formatters, CLI skeleton, error handling) is complete. This Epic focuses on implementing the critical missing pieces: text magic file parsing (binary .mgc format is explicitly deferred to Phase 2 following the proven OpenBSD approach), end-to-end integration, comprehensive testing, and comprehensive documentation. The goal is to deliver a production-ready, memory-safe alternative to C-based libmagic that achieves high compatibility with GNU file command, works with system text magic databases, and is ready for early adopters with >85% test coverage and complete API documentation (rustdoc + mdbook).

Context & Problem

Who's Affected

Primary Users:

• Rust Application Developers who need file type detection in their applications and want memory-safe alternatives to C libraries
• Security-Conscious Organizations requiring file type identification without the memory safety vulnerabilities present in C-based libmagic
• CLI Users seeking a drop-in replacement for the GNU file command with better safety guarantees

Secondary Users:

• Library Maintainers who depend on libmagic and want to migrate to safer alternatives
• DevOps Engineers integrating file type detection into build pipelines and automation workflows

Current Pain

Incomplete Implementation:
The libmagic-rs project has solid foundations (tasks 1-13 complete) but lacks the critical functionality needed for real-world usage:

• Cannot parse magic files (the DSL that defines file type detection rules)
• Cannot load system magic databases (text .magic format - binary .mgc is deferred to Phase 2)
• Missing end-to-end integration between parsing and evaluation
• No test infrastructure to validate compatibility with GNU file
• Lacks documentation for library consumers

Memory Safety Concerns:
The C-based libmagic has a history of memory safety vulnerabilities (buffer overflows, out-of-bounds reads) that pose security risks. Developers need a memory-safe alternative that provides the same functionality without these risks.

Ecosystem Gap:
While Rust's ecosystem is growing, there's no production-ready, fully-featured file type detection library that:

• Works with existing magic databases (compatibility)
• Provides both CLI and library interfaces
• Achieves performance parity with libmagic
• Offers comprehensive documentation and migration guides

Where in the Product

This Epic addresses the core functionality gap in libmagic-rs:

1. Magic File Parsing Layer - Currently missing, needs implementation for text format (binary .mgc deferred to Phase 2 per OpenBSD approach)
2. Integration Layer - MagicDatabase loading and file evaluation pipeline needs completion
3. Validation Layer - Test infrastructure required to ensure compatibility and correctness
4. Documentation Layer - API documentation and examples needed for library adoption

Scope Appetite

Phase 1 MVP Boundaries:

• In Scope: Absolute offsets, basic types (byte/short/long/string), core operators (=, !=, &), nested rules, text magic file support (Magdir directory loading), stdin input support, high GNU file compatibility, comprehensive testing, comprehensive documentation (rustdoc + mdbook)
• Out of Scope (Stretch Goals): Indirect offsets, regex support, Aho-Corasick optimization, PE/Mach-O format detection, advanced operators, performance optimization
• Timeline: Quality and completeness over speed - no hard deadlines
• Success Criteria:
  • CLI works end-to-end with file and stdin input
  • 100% compatibility with GNU file for common types (ELF, PE, ZIP, JPEG, PNG, PDF)
  • 95%+ compatibility for full test corpus (using text magic files)

  • 85% test coverage

  • Complete API documentation: rustdoc for all public APIs + mdbook documentation site with guides, examples, and migration information

Value Proposition

Completing Phase 1 MVP delivers:

• For Developers: A production-ready, memory-safe file type detection library with clean Rust APIs
• For Security Teams: Elimination of memory safety vulnerabilities present in C libmagic
• For CLI Users: A drop-in replacement for GNU file with better safety guarantees
• For the Ecosystem: A foundation for future enhancements (regex, advanced formats, optimizations)

Key Assumptions

1. Existing third-party test corpus (file:third_party/tests/) is sufficient for compatibility validation
2. Binary .mgc format is explicitly deferred to Phase 2 following the proven OpenBSD approach (see file:docs/research/magic-mgc-format-analysis.md)

• Rationale: All successful reimplementations (OpenBSD, PolyFile, arcana) parse text format only. Binary .mgc has version lock-in, platform-specific issues, and high complexity (~3,000 lines). Text format is the stable interface.

3. Built-in fallback rules for common file types will enable out-of-the-box functionality
4. Vertical slice approach (text format first, then binary) minimizes risk and enables early validation
5. Existing design documentation (file:.kiro/specs/rust-libmagic-implementation/) accurately represents the technical approach
6. No performance targets for MVP - focus on correctness and compatibility, optimize in Phase 3

[traycerai]

Plan

Implementation Plan: libmagic-rs Phase 1 MVP

Observations

The libmagic-rs project has a solid foundation with tasks 1-13 complete, including comprehensive AST structures, parser components, evaluator engine, I/O layer, output formatters, CLI framework, and error handling. The codebase demonstrates excellent code quality with 98 unit tests, zero unsafe code, and strict linting. Task 14.1 (complete magic rule parsing) is implemented in file:src/parser/mod.rs with the parse_text_magic_file function ready. The critical missing pieces are: (1) text magic file loading integration into MagicDatabase, (2) end-to-end pipeline connection, (3) comprehensive testing infrastructure with >85% coverage, and (4) complete documentation (rustdoc + mdbook). Binary .mgc format is explicitly deferred to Phase 2 per the proven OpenBSD approach.

Approach

The implementation follows a vertical slice strategy prioritizing text magic file support first, then comprehensive testing, and finally documentation. This approach enables early validation and minimizes risk. The plan focuses on four core areas: (1) Magic File Integration - connect existing parser to MagicDatabase with text format support and Magdir directory loading, (2) End-to-End Pipeline - integrate loaded rules with evaluation engine and add built-in fallback rules, (3) Comprehensive Testing - expand test infrastructure to achieve >85% coverage with integration tests, compatibility tests, and performance benchmarks, (4) Complete Documentation - enhance rustdoc coverage and build mdbook site with guides, examples, and migration information. This delivers a production-ready MVP that works with system text magic databases and achieves high GNU file compatibility.

Implementation Steps

Phase 1: Text Magic File Integration (Tasks 14.2-14.3, 16.1)

1. Implement Text Magic File Loading in MagicDatabase

File: file:src/lib.rs

Update MagicDatabase::load_from_file to integrate the existing parse_text_magic_file function from file:src/parser/mod.rs:

• Replace the placeholder implementation that returns empty rules
• Call parser::parse_text_magic_file to parse the magic file content
• Handle file reading with proper error propagation using std::fs::read_to_string
• Convert ParseError to LibmagicError for consistent error handling
• Add validation of loaded rules (check for empty rule sets, malformed rules)
• Store parsed rules in the MagicDatabase.rules field

Dependencies: Requires file:src/parser/mod.rs (already complete with parse_text_magic_file)

2. Add Magic File Format Detection

File: file:src/lib.rs (new helper function)

Create detect_magic_file_format function to distinguish text vs binary formats:

• Check file extension (.magic for text, .mgc for binary)
• Read first 4 bytes to detect binary magic signature (0x0d0a1a0a or similar)
• Return enum MagicFileFormat { Text, Binary, Unknown }
• Add error handling for file access failures
• Document that binary .mgc format returns error in Phase 1 (deferred to Phase 2)

Note: Binary format detection returns LibmagicError::InvalidFormat with message directing users to text format in Phase 1.

3. Implement Magdir Directory Loading

File: file:src/lib.rs (new function load_from_directory)

Add support for loading multiple magic files from a directory (Magdir pattern):

• Create MagicDatabase::load_from_directory function
• Use std::fs::read_dir to iterate directory entries
• Filter for .magic files (skip binary .mgc in Phase 1)
• Parse each file using parse_text_magic_file
• Merge rules from all files into single Vec<MagicRule>
• Preserve rule order (files processed alphabetically)
• Add comprehensive error handling with file-specific error context

Dependencies: Requires step 1 (text file loading)

4. Add System Magic File Discovery

File: file:src/lib.rs (new function load_system_magic)

Implement platform-specific magic file discovery:

• Create MagicDatabase::load_system_magic function
• Unix paths: /usr/share/file/magic, /usr/share/misc/magic, /etc/magic
• Windows paths: %APPDATA%\Magic, %PROGRAMDATA%\Magic
• Try each path in order, use first that exists
• Support both single files and Magdir directories
• Add fallback to built-in rules if no system magic found
• Document search order and fallback behavior in rustdoc

Dependencies: Requires steps 1-3

Phase 2: End-to-End Pipeline Integration (Task 16.2-16.3)

5. Fix File Evaluation Pipeline

File: file:src/lib.rs

Update MagicDatabase::evaluate_file to properly integrate loaded rules:

• Remove placeholder "data" return when rules are empty
• Ensure evaluate_rules_with_config from file:src/evaluator/mod.rs is called correctly
• Add proper error handling for evaluation failures
• Implement confidence scoring based on match depth and specificity
• Add MIME type mapping when enable_mime_types is true (placeholder for now)
• Ensure result formatting matches GNU file output style

Dependencies: Requires Phase 1 completion

6. Implement Built-in Fallback Magic Rules

File: file:src/lib.rs (new module builtin_rules)

Create comprehensive built-in rules for common file types:

• Define static rules for: ELF (0x7fELF), PE (MZ header), ZIP (PK\x03\x04), JPEG (JFIF), PNG (PNG signature), PDF (%PDF), GIF (GIF87a/GIF89a)
• Structure as const BUILTIN_RULES: &[MagicRule] using AST types from file:src/parser/ast.rs
• Add MagicDatabase::with_builtin_rules constructor
• Implement fallback logic: use built-in rules when external magic file loading fails
• Add configuration option to disable built-in rules (use_builtin_fallback: bool)
• Document built-in rule coverage and limitations

Example structure:

const BUILTIN_RULES: &[MagicRule] = &[
    MagicRule {
        offset: OffsetSpec::Absolute(0),
        typ: TypeKind::Byte,
        op: Operator::Equal,
        value: Value::Bytes(vec![0x7f, 0x45, 0x4c, 0x46]),
        message: "ELF".to_string(),
        children: vec![],
        level: 0,
    },
    // ... more rules
];

7. Add Stdin Input Support

File: file:src/main.rs

Extend CLI to support reading from stdin:

• Add --stdin flag to file:src/main.rs CLI arguments
• Implement stdin reading using std::io::stdin().read_to_end()
• Create in-memory buffer for evaluation (no FileBuffer needed)
• Handle stdin with same evaluation pipeline as file input
• Add proper error handling for stdin read failures
• Update help text and examples in rustdoc

Dependencies: Requires Phase 2 steps 5-6

Phase 3: Comprehensive Testing Infrastructure (Task 17)

8. Expand Integration Test Coverage

File: file:tests/integration_tests.rs (new file)

Create comprehensive integration tests for end-to-end workflows:

• Test magic file loading from single file
• Test Magdir directory loading with multiple files
• Test system magic file discovery on current platform
• Test file evaluation with loaded rules
• Test stdin input evaluation
• Test built-in fallback rules when no magic file available
• Test error handling for missing files, invalid magic files, corrupted input
• Use insta crate (already in dev-dependencies) for snapshot testing
• Target: 30+ integration test cases

Test structure:

• Create tests/fixtures/magic/ with sample magic files
• Create tests/fixtures/samples/ with test binary files (ELF, PE, ZIP, etc.)
• Use tempfile crate (already in dev-dependencies) for temporary file testing

9. Enhance Compatibility Test Framework

File: file:tests/compatibility_tests.rs

Expand existing compatibility tests to achieve 95%+ compatibility:

• Add test cases for common file types: ELF, PE, ZIP, JPEG, PNG, PDF, GIF, TAR, GZIP
• Implement comparison with GNU file command output
• Add normalization for platform-specific differences (paths, line endings)
• Create test corpus in file:third_party/tests/ with expected results
• Add compatibility percentage calculation and reporting
• Document compatibility gaps and known differences
• Target: 100% compatibility for common types, 95%+ for full corpus

Note: Use existing CompatibilityTestRunner structure, expand test file coverage

10. Implement Performance Benchmarks

File: file:benches/performance.rs (new file)

Create performance benchmarks using criterion crate (already in dev-dependencies):

• Benchmark magic file parsing (text format)
• Benchmark file evaluation for different file sizes (1KB, 1MB, 10MB)
• Benchmark rule matching for different rule set sizes
• Benchmark memory-mapped I/O vs standard file reading
• Compare with GNU file command performance (informational, not blocking)
• Add regression detection for future changes
• Target: Establish baseline metrics, no specific performance targets for MVP

Benchmark categories:

• parse_magic_file: Measure parsing speed
• evaluate_small_file: 1KB file evaluation
• evaluate_large_file: 10MB file evaluation
• load_system_magic: System magic database loading

11. Achieve >85% Test Coverage

Files: All source files in file:src/

Expand unit tests to achieve >85% code coverage:

• Add unit tests for MagicDatabase loading functions
• Add unit tests for magic file format detection
• Add unit tests for Magdir directory loading
• Add unit tests for built-in fallback rules
• Add unit tests for stdin input handling
• Add edge case tests for error conditions
• Use cargo llvm-cov to measure coverage
• Document coverage gaps and justification for uncovered code

Coverage targets:

• file:src/lib.rs: >90% coverage
• file:src/parser/mod.rs: >95% coverage (already high)
• file:src/evaluator/mod.rs: >90% coverage
• file:src/io/mod.rs: >95% coverage (already high)
• file:src/output/: >90% coverage

Phase 4: Complete Documentation (Tasks 20, 23)

12. Enhance Rustdoc Coverage for All Public APIs

Files: All public modules in file:src/

Complete rustdoc documentation for all public APIs:

• Add comprehensive doc comments to all public functions in file:src/lib.rs
• Document MagicDatabase methods with usage examples
• Document EvaluationConfig options with security implications
• Document error types and recovery strategies
• Add code examples for common use cases (loading, evaluating, output formatting)
• Document magic file format requirements and limitations
• Add "Examples" sections to all public functions
• Add "Errors" sections documenting failure modes
• Add "Security" sections for security-sensitive functions

Target: 100% rustdoc coverage for public APIs with examples

13. Build mdbook Documentation Site

File: file:docs/src/ (expand existing structure)

Complete the mdbook documentation site with comprehensive guides:

User Guide (Part I):

• Expand file:docs/src/getting-started.md with installation, quick start, first steps
• Expand file:docs/src/cli-usage.md with all CLI flags, examples, common workflows
• Expand file:docs/src/library-api.md with API overview, basic usage, advanced patterns
• Expand file:docs/src/configuration.md with all config options, security considerations

Architecture (Part II):

• Expand file:docs/src/architecture.md with parser-evaluator design, data flow diagrams
• Expand file:docs/src/parser.md with magic file syntax, parsing pipeline, AST structure
• Expand file:docs/src/evaluator.md with rule evaluation, offset resolution, type interpretation
• Expand file:docs/src/output.md with text/JSON formatters, MIME type mapping

Advanced Topics (Part III):

• Expand file:docs/src/magic-format.md with text format specification, examples, best practices
• Expand file:docs/src/testing.md with testing strategy, running tests, writing tests
• Expand file:docs/src/performance.md with optimization techniques, benchmarking, profiling

Migration (Part IV):

• Expand file:docs/src/migration.md with C API comparison, code examples, compatibility matrix
• Expand file:docs/src/troubleshooting.md with common issues, debugging, FAQ

Development (Part V):

• Expand file:docs/src/development.md with setup, building, contributing
• Expand file:docs/src/testing-guidelines.md with test writing, coverage, best practices

Appendices:

• Expand file:docs/src/api-reference.md with complete API listing
• Expand file:docs/src/cli-reference.md with all CLI options
• Expand file:docs/src/magic-examples.md with magic file examples
• Expand file:docs/src/compatibility.md with compatibility matrix, known differences

Target: Complete documentation site with 20+ pages, diagrams, examples

14. Create Library Usage Examples

File: file:examples/ (new directory)

Create comprehensive examples demonstrating library usage:

• basic_usage.rs: Load magic file, evaluate single file, print result
• batch_processing.rs: Evaluate multiple files with progress reporting
• custom_config.rs: Use custom EvaluationConfig with security limits
• json_output.rs: Generate JSON output for integration
• error_handling.rs: Demonstrate error handling and recovery
• stdin_processing.rs: Process files from stdin
• builtin_rules.rs: Use built-in fallback rules

Target: 7+ runnable examples with documentation

15. Add Migration Guide with Code Comparisons

File: file:docs/src/migration.md

Create comprehensive migration guide from C libmagic:

• Side-by-side code comparisons (C vs Rust)
• API mapping table (magic_open → MagicDatabase::load_from_file, etc.)
• Memory management differences (RAII vs manual cleanup)
• Error handling differences (Result vs errno)
• Configuration differences (flags vs EvaluationConfig)
• Performance considerations and optimization tips
• Common pitfalls and solutions
• Migration checklist for existing projects

Include diagrams:

sequenceDiagram
    participant App
    participant MagicDB as MagicDatabase
    participant Parser
    participant Evaluator
    participant FileBuffer
    
    App->>MagicDB: load_from_file("magic.db")
    MagicDB->>Parser: parse_text_magic_file
    Parser-->>MagicDB: Vec<MagicRule>
    App->>MagicDB: evaluate_file("sample.bin")
    MagicDB->>FileBuffer: new("sample.bin")
    FileBuffer-->>MagicDB: &[u8]
    MagicDB->>Evaluator: evaluate_rules
    Evaluator-->>MagicDB: Vec<MatchResult>
    MagicDB-->>App: EvaluationResult

Loading

Phase 5: Final Integration and Validation

16. End-to-End Validation

Files: All test files

Perform comprehensive end-to-end validation:

• Run full test suite: cargo test --all-features
• Run compatibility tests: cargo test --test compatibility_tests -- --ignored
• Run benchmarks: cargo bench
• Measure test coverage: cargo llvm-cov --html
• Validate >85% coverage threshold
• Test on multiple platforms (Linux, macOS, Windows via CI)
• Test with system magic databases on each platform
• Validate GNU file compatibility (100% for common types, 95%+ for full corpus)

17. Documentation Build and Review

Files: Documentation files

Build and validate documentation:

• Build rustdoc: cargo doc --no-deps --open
• Build mdbook: cd docs && mdbook build
• Validate all links: mdbook test
• Review all examples for correctness
• Ensure all public APIs have documentation
• Validate code examples compile and run
• Check for broken links and missing sections

18. CLI Validation and User Testing

File: file:src/main.rs

Validate CLI functionality:

• Test all CLI flags and options
• Test file input, stdin input, multiple files
• Test output formats (text, JSON)
• Test custom magic file loading
• Test system magic file discovery
• Test error messages and help text
• Validate exit codes for different scenarios
• Test on real-world files (ELF, PE, ZIP, images, documents)

Success Criteria

Functionality:

• ✅ CLI works end-to-end with file and stdin input
• ✅ Text magic file loading from single file and Magdir directories
• ✅ System magic database discovery on Unix and Windows
• ✅ Built-in fallback rules for common file types
• ✅ 100% compatibility with GNU file for common types (ELF, PE, ZIP, JPEG, PNG, PDF)
• ✅ 95%+ compatibility for full test corpus

Testing:

• ✅ >85% test coverage across all modules
• ✅ 30+ integration tests covering end-to-end workflows
• ✅ Comprehensive compatibility test suite
• ✅ Performance benchmarks established

Documentation:

• ✅ 100% rustdoc coverage for all public APIs with examples
• ✅ Complete mdbook documentation site (20+ pages)
• ✅ Migration guide with C API comparisons
• ✅ 7+ runnable examples

Quality:

• ✅ Zero unsafe code (except vetted dependencies)
• ✅ Zero clippy warnings
• ✅ All tests passing on Linux, macOS, Windows

Import In IDE

🤖 Prompt for AI Agents

# Implementation Plan: libmagic-rs Phase 1 MVP

## Observations

The libmagic-rs project has a solid foundation with tasks 1-13 complete, including comprehensive AST structures, parser components, evaluator engine, I/O layer, output formatters, CLI framework, and error handling. The codebase demonstrates excellent code quality with 98 unit tests, zero unsafe code, and strict linting. Task 14.1 (complete magic rule parsing) is implemented in `file:src/parser/mod.rs` with the `parse_text_magic_file` function ready. The critical missing pieces are: (1) text magic file loading integration into `MagicDatabase`, (2) end-to-end pipeline connection, (3) comprehensive testing infrastructure with >85% coverage, and (4) complete documentation (rustdoc + mdbook). Binary .mgc format is explicitly deferred to Phase 2 per the proven OpenBSD approach.

## Approach

The implementation follows a vertical slice strategy prioritizing text magic file support first, then comprehensive testing, and finally documentation. This approach enables early validation and minimizes risk. The plan focuses on four core areas: (1) **Magic File Integration** - connect existing parser to `MagicDatabase` with text format support and Magdir directory loading, (2) **End-to-End Pipeline** - integrate loaded rules with evaluation engine and add built-in fallback rules, (3) **Comprehensive Testing** - expand test infrastructure to achieve >85% coverage with integration tests, compatibility tests, and performance benchmarks, (4) **Complete Documentation** - enhance rustdoc coverage and build mdbook site with guides, examples, and migration information. This delivers a production-ready MVP that works with system text magic databases and achieves high GNU file compatibility.

## Implementation Steps

### Phase 1: Text Magic File Integration (Tasks 14.2-14.3, 16.1)

#### 1. Implement Text Magic File Loading in MagicDatabase

**File:** `file:src/lib.rs`

Update `MagicDatabase::load_from_file` to integrate the existing `parse_text_magic_file` function from `file:src/parser/mod.rs`:

- Replace the placeholder implementation that returns empty rules
- Call `parser::parse_text_magic_file` to parse the magic file content
- Handle file reading with proper error propagation using `std::fs::read_to_string`
- Convert `ParseError` to `LibmagicError` for consistent error handling
- Add validation of loaded rules (check for empty rule sets, malformed rules)
- Store parsed rules in the `MagicDatabase.rules` field

**Dependencies:** Requires `file:src/parser/mod.rs` (already complete with `parse_text_magic_file`)

#### 2. Add Magic File Format Detection

**File:** `file:src/lib.rs` (new helper function)

Create `detect_magic_file_format` function to distinguish text vs binary formats:

- Check file extension (.magic for text, .mgc for binary)
- Read first 4 bytes to detect binary magic signature (0x0d0a1a0a or similar)
- Return enum `MagicFileFormat { Text, Binary, Unknown }`
- Add error handling for file access failures
- Document that binary .mgc format returns error in Phase 1 (deferred to Phase 2)

**Note:** Binary format detection returns `LibmagicError::InvalidFormat` with message directing users to text format in Phase 1.

#### 3. Implement Magdir Directory Loading

**File:** `file:src/lib.rs` (new function `load_from_directory`)

Add support for loading multiple magic files from a directory (Magdir pattern):

- Create `MagicDatabase::load_from_directory` function
- Use `std::fs::read_dir` to iterate directory entries
- Filter for `.magic` files (skip binary .mgc in Phase 1)
- Parse each file using `parse_text_magic_file`
- Merge rules from all files into single `Vec<MagicRule>`
- Preserve rule order (files processed alphabetically)
- Add comprehensive error handling with file-specific error context

**Dependencies:** Requires step 1 (text file loading)

#### 4. Add System Magic File Discovery

**File:** `file:src/lib.rs` (new function `load_system_magic`)

Implement platform-specific magic file discovery:

- Create `MagicDatabase::load_system_magic` function
- Unix paths: `/usr/share/file/magic`, `/usr/share/misc/magic`, `/etc/magic`
- Windows paths: `%APPDATA%\Magic`, `%PROGRAMDATA%\Magic`
- Try each path in order, use first that exists
- Support both single files and Magdir directories
- Add fallback to built-in rules if no system magic found
- Document search order and fallback behavior in rustdoc

**Dependencies:** Requires steps 1-3

### Phase 2: End-to-End Pipeline Integration (Task 16.2-16.3)

#### 5. Fix File Evaluation Pipeline

**File:** `file:src/lib.rs`

Update `MagicDatabase::evaluate_file` to properly integrate loaded rules:

- Remove placeholder "data" return when rules are empty
- Ensure `evaluate_rules_with_config` from `file:src/evaluator/mod.rs` is called correctly
- Add proper error handling for evaluation failures
- Implement confidence scoring based on match depth and specificity
- Add MIME type mapping when `enable_mime_types` is true (placeholder for now)
- Ensure result formatting matches GNU file output style

**Dependencies:** Requires Phase 1 completion

#### 6. Implement Built-in Fallback Magic Rules

**File:** `file:src/lib.rs` (new module `builtin_rules`)

Create comprehensive built-in rules for common file types:

- Define static rules for: ELF (0x7fELF), PE (MZ header), ZIP (PK\x03\x04), JPEG (JFIF), PNG (PNG signature), PDF (%PDF), GIF (GIF87a/GIF89a)
- Structure as `const BUILTIN_RULES: &[MagicRule]` using AST types from `file:src/parser/ast.rs`
- Add `MagicDatabase::with_builtin_rules` constructor
- Implement fallback logic: use built-in rules when external magic file loading fails
- Add configuration option to disable built-in rules (`use_builtin_fallback: bool`)
- Document built-in rule coverage and limitations

**Example structure:**
```rust
const BUILTIN_RULES: &[MagicRule] = &[
    MagicRule {
        offset: OffsetSpec::Absolute(0),
        typ: TypeKind::Byte,
        op: Operator::Equal,
        value: Value::Bytes(vec![0x7f, 0x45, 0x4c, 0x46]),
        message: "ELF".to_string(),
        children: vec![],
        level: 0,
    },
    // ... more rules
];
```

#### 7. Add Stdin Input Support

**File:** `file:src/main.rs`

Extend CLI to support reading from stdin:

- Add `--stdin` flag to `file:src/main.rs` CLI arguments
- Implement stdin reading using `std::io::stdin().read_to_end()`
- Create in-memory buffer for evaluation (no FileBuffer needed)
- Handle stdin with same evaluation pipeline as file input
- Add proper error handling for stdin read failures
- Update help text and examples in rustdoc

**Dependencies:** Requires Phase 2 steps 5-6

### Phase 3: Comprehensive Testing Infrastructure (Task 17)

#### 8. Expand Integration Test Coverage

**File:** `file:tests/integration_tests.rs` (new file)

Create comprehensive integration tests for end-to-end workflows:

- Test magic file loading from single file
- Test Magdir directory loading with multiple files
- Test system magic file discovery on current platform
- Test file evaluation with loaded rules
- Test stdin input evaluation
- Test built-in fallback rules when no magic file available
- Test error handling for missing files, invalid magic files, corrupted input
- Use `insta` crate (already in dev-dependencies) for snapshot testing
- Target: 30+ integration test cases

**Test structure:**
- Create `tests/fixtures/magic/` with sample magic files
- Create `tests/fixtures/samples/` with test binary files (ELF, PE, ZIP, etc.)
- Use `tempfile` crate (already in dev-dependencies) for temporary file testing

#### 9. Enhance Compatibility Test Framework

**File:** `file:tests/compatibility_tests.rs`

Expand existing compatibility tests to achieve 95%+ compatibility:

- Add test cases for common file types: ELF, PE, ZIP, JPEG, PNG, PDF, GIF, TAR, GZIP
- Implement comparison with GNU file command output
- Add normalization for platform-specific differences (paths, line endings)
- Create test corpus in `file:third_party/tests/` with expected results
- Add compatibility percentage calculation and reporting
- Document compatibility gaps and known differences
- Target: 100% compatibility for common types, 95%+ for full corpus

**Note:** Use existing `CompatibilityTestRunner` structure, expand test file coverage

#### 10. Implement Performance Benchmarks

**File:** `file:benches/performance.rs` (new file)

Create performance benchmarks using `criterion` crate (already in dev-dependencies):

- Benchmark magic file parsing (text format)
- Benchmark file evaluation for different file sizes (1KB, 1MB, 10MB)
- Benchmark rule matching for different rule set sizes
- Benchmark memory-mapped I/O vs standard file reading
- Compare with GNU file command performance (informational, not blocking)
- Add regression detection for future changes
- Target: Establish baseline metrics, no specific performance targets for MVP

**Benchmark categories:**
- `parse_magic_file`: Measure parsing speed
- `evaluate_small_file`: 1KB file evaluation
- `evaluate_large_file`: 10MB file evaluation
- `load_system_magic`: System magic database loading

#### 11. Achieve >85% Test Coverage

**Files:** All source files in `file:src/`

Expand unit tests to achieve >85% code coverage:

- Add unit tests for `MagicDatabase` loading functions
- Add unit tests for magic file format detection
- Add unit tests for Magdir directory loading
- Add unit tests for built-in fallback rules
- Add unit tests for stdin input handling
- Add edge case tests for error conditions
- Use `cargo llvm-cov` to measure coverage
- Document coverage gaps and justification for uncovered code

**Coverage targets:**
- `file:src/lib.rs`: >90% coverage
- `file:src/parser/mod.rs`: >95% coverage (already high)
- `file:src/evaluator/mod.rs`: >90% coverage
- `file:src/io/mod.rs`: >95% coverage (already high)
- `file:src/output/`: >90% coverage

### Phase 4: Complete Documentation (Tasks 20, 23)

#### 12. Enhance Rustdoc Coverage for All Public APIs

**Files:** All public modules in `file:src/`

Complete rustdoc documentation for all public APIs:

- Add comprehensive doc comments to all public functions in `file:src/lib.rs`
- Document `MagicDatabase` methods with usage examples
- Document `EvaluationConfig` options with security implications
- Document error types and recovery strategies
- Add code examples for common use cases (loading, evaluating, output formatting)
- Document magic file format requirements and limitations
- Add "Examples" sections to all public functions
- Add "Errors" sections documenting failure modes
- Add "Security" sections for security-sensitive functions

**Target:** 100% rustdoc coverage for public APIs with examples

#### 13. Build mdbook Documentation Site

**File:** `file:docs/src/` (expand existing structure)

Complete the mdbook documentation site with comprehensive guides:

**User Guide (Part I):**
- Expand `file:docs/src/getting-started.md` with installation, quick start, first steps
- Expand `file:docs/src/cli-usage.md` with all CLI flags, examples, common workflows
- Expand `file:docs/src/library-api.md` with API overview, basic usage, advanced patterns
- Expand `file:docs/src/configuration.md` with all config options, security considerations

**Architecture (Part II):**
- Expand `file:docs/src/architecture.md` with parser-evaluator design, data flow diagrams
- Expand `file:docs/src/parser.md` with magic file syntax, parsing pipeline, AST structure
- Expand `file:docs/src/evaluator.md` with rule evaluation, offset resolution, type interpretation
- Expand `file:docs/src/output.md` with text/JSON formatters, MIME type mapping

**Advanced Topics (Part III):**
- Expand `file:docs/src/magic-format.md` with text format specification, examples, best practices
- Expand `file:docs/src/testing.md` with testing strategy, running tests, writing tests
- Expand `file:docs/src/performance.md` with optimization techniques, benchmarking, profiling

**Migration (Part IV):**
- Expand `file:docs/src/migration.md` with C API comparison, code examples, compatibility matrix
- Expand `file:docs/src/troubleshooting.md` with common issues, debugging, FAQ

**Development (Part V):**
- Expand `file:docs/src/development.md` with setup, building, contributing
- Expand `file:docs/src/testing-guidelines.md` with test writing, coverage, best practices

**Appendices:**
- Expand `file:docs/src/api-reference.md` with complete API listing
- Expand `file:docs/src/cli-reference.md` with all CLI options
- Expand `file:docs/src/magic-examples.md` with magic file examples
- Expand `file:docs/src/compatibility.md` with compatibility matrix, known differences

**Target:** Complete documentation site with 20+ pages, diagrams, examples

#### 14. Create Library Usage Examples

**File:** `file:examples/` (new directory)

Create comprehensive examples demonstrating library usage:

- `basic_usage.rs`: Load magic file, evaluate single file, print result
- `batch_processing.rs`: Evaluate multiple files with progress reporting
- `custom_config.rs`: Use custom `EvaluationConfig` with security limits
- `json_output.rs`: Generate JSON output for integration
- `error_handling.rs`: Demonstrate error handling and recovery
- `stdin_processing.rs`: Process files from stdin
- `builtin_rules.rs`: Use built-in fallback rules

**Target:** 7+ runnable examples with documentation

#### 15. Add Migration Guide with Code Comparisons

**File:** `file:docs/src/migration.md`

Create comprehensive migration guide from C libmagic:

- Side-by-side code comparisons (C vs Rust)
- API mapping table (magic_open → MagicDatabase::load_from_file, etc.)
- Memory management differences (RAII vs manual cleanup)
- Error handling differences (Result vs errno)
- Configuration differences (flags vs EvaluationConfig)
- Performance considerations and optimization tips
- Common pitfalls and solutions
- Migration checklist for existing projects

**Include diagrams:**
```mermaid
sequenceDiagram
    participant App
    participant MagicDB as MagicDatabase
    participant Parser
    participant Evaluator
    participant FileBuffer
    
    App->>MagicDB: load_from_file("magic.db")
    MagicDB->>Parser: parse_text_magic_file
    Parser-->>MagicDB: Vec<MagicRule>
    App->>MagicDB: evaluate_file("sample.bin")
    MagicDB->>FileBuffer: new("sample.bin")
    FileBuffer-->>MagicDB: &[u8]
    MagicDB->>Evaluator: evaluate_rules
    Evaluator-->>MagicDB: Vec<MatchResult>
    MagicDB-->>App: EvaluationResult
```

### Phase 5: Final Integration and Validation

#### 16. End-to-End Validation

**Files:** All test files

Perform comprehensive end-to-end validation:

- Run full test suite: `cargo test --all-features`
- Run compatibility tests: `cargo test --test compatibility_tests -- --ignored`
- Run benchmarks: `cargo bench`
- Measure test coverage: `cargo llvm-cov --html`
- Validate >85% coverage threshold
- Test on multiple platforms (Linux, macOS, Windows via CI)
- Test with system magic databases on each platform
- Validate GNU file compatibility (100% for common types, 95%+ for full corpus)

#### 17. Documentation Build and Review

**Files:** Documentation files

Build and validate documentation:

- Build rustdoc: `cargo doc --no-deps --open`
- Build mdbook: `cd docs && mdbook build`
- Validate all links: `mdbook test`
- Review all examples for correctness
- Ensure all public APIs have documentation
- Validate code examples compile and run
- Check for broken links and missing sections

#### 18. CLI Validation and User Testing

**File:** `file:src/main.rs`

Validate CLI functionality:

- Test all CLI flags and options
- Test file input, stdin input, multiple files
- Test output formats (text, JSON)
- Test custom magic file loading
- Test system magic file discovery
- Test error messages and help text
- Validate exit codes for different scenarios
- Test on real-world files (ELF, PE, ZIP, images, documents)

## Success Criteria

**Functionality:**
- ✅ CLI works end-to-end with file and stdin input
- ✅ Text magic file loading from single file and Magdir directories
- ✅ System magic database discovery on Unix and Windows
- ✅ Built-in fallback rules for common file types
- ✅ 100% compatibility with GNU file for common types (ELF, PE, ZIP, JPEG, PNG, PDF)
- ✅ 95%+ compatibility for full test corpus

**Testing:**
- ✅ >85% test coverage across all modules
- ✅ 30+ integration tests covering end-to-end workflows
- ✅ Comprehensive compatibility test suite
- ✅ Performance benchmarks established

**Documentation:**
- ✅ 100% rustdoc coverage for all public APIs with examples
- ✅ Complete mdbook documentation site (20+ pages)
- ✅ Migration guide with C API comparisons
- ✅ 7+ runnable examples

**Quality:**
- ✅ Zero unsafe code (except vetted dependencies)
- ✅ Zero clippy warnings
- ✅ All tests passing on Linux, macOS, Windows

---

Execution Information

Branch: main
Commit: 50b370c

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[unclesp1d3r]

@coderabbitai Assign the proper labels and other metadata to this Epic. It is an overarching issue that encapsulates the sub-issues that make up the Phase 1 MVP.