Implement Semantic Pattern Matching: GUID, Email, Base64, Printf Format, and User Agent Detection

[unclesp1d3r]

Overview

Implement semantic pattern matching for the remaining high-value string classifications: GUID, email addresses, Base64 data, printf-style format strings, and user agent strings. These patterns are critical for identifying security indicators and code artifacts in binary analysis.

Current State

• Tag Enum: Already defined in src/types.rs with Guid, Email, Base64, FormatString, and UserAgent variants
• Classification Module: Empty stub at src/classification/mod.rs (only contains comment)
• Documentation: Comprehensive patterns and implementation examples exist in docs/src/classification.md
• Dependency Gap: Missing regex crate for pattern matching
• Blocker: Depends on Semantic Classification Framework implementation

Technical Requirements

Dependencies to Add

Add to Cargo.toml:

[dependencies]
regex = "1.10"
lazy_static = "1.4"  # For regex compilation caching

Implementation Details

Create SemanticClassifier struct in src/classification/mod.rs:

use regex::Regex;
use lazy_static::lazy_static;
use crate::types::{Tag, FoundString, StringContext};

pub struct SemanticClassifier {
    guid_regex: Regex,
    email_regex: Regex,
    base64_regex: Regex,
    format_regex: Regex,
    user_agent_regex: Regex,
}

impl SemanticClassifier {
    pub fn new() -> Self {
        // Initialize with compiled regex patterns
    }
    
    pub fn classify(&self, text: &str, context: &StringContext) -> Vec<Tag> {
        // Pattern matching logic
    }
}

Regex Patterns

Based on docs/src/classification.md:

1. GUID/UUID

   • Pattern: \{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\}
   • Example: {12345678-1234-1234-1234-123456789abc}
   • Validation: Format compliance, version field checking

2. Email Address

   • Pattern: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
   • Example: admin@malware.com
   • Validation: RFC compliance, domain validation

3. Base64

   • Pattern: [A-Za-z0-9+/]{20,}={0,2}
   • Example: SGVsbG8gV29ybGQ=
   • Validation: Length divisibility by 4, padding correctness, minimum length threshold

4. Printf Format Strings

   • Pattern: %[sdxofcpn]|%\d+[sdxofcpn]|\{\d+\}
   • Examples: Error: %s at line %d, User {0} logged in
   • Context: Proximity to other format strings, common in .rodata

5. User Agent

   • Pattern: Mozilla/[0-9.]+|Chrome/[0-9.]+|Safari/[0-9.]+|AppleWebKit/[0-9.]+
   • Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
   • Validation: Known browser identifiers, version format

Acceptance Criteria

• SemanticClassifier struct implemented with all five pattern types
• Each pattern has dedicated detection method with confidence scoring
• Context-aware classification considers section type and binary format
• False positive reduction through validation (length, entropy, format)
• Regex patterns compiled once and cached using lazy_static
• Integration with FoundString type to populate tags field
• Comprehensive unit tests for each pattern type:
  • Positive matches (valid GUIDs, emails, Base64, format strings, user agents)
  • Negative matches (invalid formats, edge cases)
  • Context-dependent behavior
  • Multi-tag scenarios (string matching multiple patterns)
• Integration tests with real binary samples
• Documentation with examples and pattern explanations
• Benchmark tests for performance validation

Test Coverage Requirements

Unit Tests (tests/classification_tests.rs)

#[test]
fn test_guid_detection() {
    // Valid GUID formats
    // Invalid GUID formats
    // Case sensitivity
}

#[test]
fn test_email_detection() {
    // Valid emails
    // Invalid emails (missing @, invalid TLD)
}

#[test]
fn test_base64_detection() {
    // Valid Base64 (with/without padding)
    // Invalid Base64 (wrong length, invalid characters)
    // Minimum length threshold
}

#[test]
fn test_format_string_detection() {
    // Printf-style: %s, %d, %x
    // Python-style: {0}, {1}
    // Mixed format strings
}

#[test]
fn test_user_agent_detection() {
    // Common browsers
    // Mobile user agents
    // Bot user agents
}

#[test]
fn test_false_positive_reduction() {
    // High-entropy binary data
    // Very short matches
    // Invalid context
}

Integration Tests

Test with real binaries containing these patterns extracted from:

• ELF binaries with GUIDs in .rodata
• PE files with user agents in .rdata
• Mach-O binaries with format strings in __TEXT,__cstring

Performance Considerations

• Use lazy_static for one-time regex compilation
• Implement short-circuit evaluation (check simpler patterns first)
• Consider minimum string length before applying expensive regex
• Profile regex performance on large binaries

Related Issues

• Blocks: Implement Main String Extraction Orchestrator and Pipeline Integration #36 (Main String Extraction Orchestrator)
• Blocks: Complete End-to-End Pipeline Integration with Error Recovery and Testing #37 (Complete Pipeline Integration)
• Related: Performance: Implement Regex Compilation Caching for Semantic Classifier #33 (Regex Caching)

References

• Detailed patterns: docs/src/classification.md
• Type definitions: src/types.rs
• Tag enum: Lines 20-40 in src/types.rs

Implementation Notes

1. Start with SemanticClassifier struct definition
2. Implement each pattern matcher as a separate method
3. Add validation logic for each pattern type
4. Integrate with existing FoundString and Tag types
5. Write comprehensive unit tests for each pattern
6. Add integration tests with real binaries
7. Optimize with benchmarking
8. Update documentation with usage examples

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# Fetch the current issue details
echo "=== Current Issue Details ==="
gh issue view 18 --json title,body,labels,assignees,state --repo EvilBit-Labs/StringyMcStringFace

echo -e "\n=== Repository Structure ==="
# Find pattern-related files
fd -e py -e go -e rs | grep -i "pattern\|semantic" | head -20

echo -e "\n=== Look for existing semantic pattern implementations ==="
# Search for semantic pattern related code
rg -n "semantic" --type py -C 2 | head -50

echo -e "\n=== Look for pattern matching implementations ==="
# Search for pattern matching code
rg -n "class.*Pattern|def.*pattern" --type py -C 1 | head -50

echo -e "\n=== Check for test files ==="
# Find test files
fd test -e py | head -20

Length of output: 1749

---

🏁 Script executed:

#!/bin/bash

echo "=== List all files in repository ==="
fd -t f | head -30

echo -e "\n=== Rust source files ==="
fd -e rs

echo -e "\n=== Look for pattern implementations in Rust ==="
rg -n "pattern|semantic" --type rust -C 2 | head -80

echo -e "\n=== Look for struct/enum definitions ==="
rg -n "^(pub )?(struct|enum|impl).*Pattern" --type rust -C 2 | head -50

echo -e "\n=== Check README and docs ==="
cat README.md 2>/dev/null || echo "No README.md found"

echo -e "\n=== Check Cargo.toml ==="
cat Cargo.toml 2>/dev/null || echo "No Cargo.toml found"

Length of output: 9296

---

🏁 Script executed:

#!/bin/bash

echo "=== Classification Module ==="
cat src/classification/mod.rs

echo -e "\n=== Types Definition ==="
cat src/types.rs

echo -e "\n=== Check for existing pattern/regex usage ==="
rg -n "regex|Regex|pattern" --type rust -C 3

echo -e "\n=== Check docs for classification details ==="
cat docs/src/classification.md

echo -e "\n=== Check Cargo.toml dependencies ==="
grep -A 20 "dependencies" Cargo.toml

Length of output: 15494

---

🏁 Script executed:

#!/bin/bash

# Check available labels in the repository
echo "=== Available Labels ==="
gh label list --repo EvilBit-Labs/StringyMcStringFace --limit 50

# Check if there are any related PRs or issues
echo -e "\n=== Related Issues ==="
gh issue list --repo EvilBit-Labs/StringyMcStringFace --label "area:analyzer" --limit 10

# Check the project task file if it exists
echo -e "\n=== Check for task/spec files ==="
fd -t f "task" | head -10
cat .kiro/specs/stringy-binary-analyzer/tasks.md 2>/dev/null | grep -A 20 "Semantic Classification" || echo "Task file not accessible"

Length of output: 3091

---

🏁 Script executed:

#!/bin/bash

# Create enhanced issue body
cat > /tmp/enhanced_issue.md <<'EOF'
## Overview

Implement semantic pattern matching for the remaining high-value string classifications: GUID, email addresses, Base64 data, printf-style format strings, and user agent strings. These patterns are critical for identifying security indicators and code artifacts in binary analysis.

## Current State

- **Tag Enum**: Already defined in `src/types.rs` with `Guid`, `Email`, `Base64`, `FormatString`, and `UserAgent` variants
- **Classification Module**: Empty stub at `src/classification/mod.rs` (only contains comment)
- **Documentation**: Comprehensive patterns and implementation examples exist in `docs/src/classification.md`
- **Dependency Gap**: Missing `regex` crate for pattern matching
- **Blocker**: Depends on Semantic Classification Framework implementation

## Technical Requirements

### Dependencies to Add

Add to `Cargo.toml`:
```toml
[dependencies]
regex = "1.10"
lazy_static = "1.4"  # For regex compilation caching
```

### Implementation Details

Create `SemanticClassifier` struct in `src/classification/mod.rs`:

```rust
use regex::Regex;
use lazy_static::lazy_static;
use crate::types::{Tag, FoundString, StringContext};

pub struct SemanticClassifier {
    guid_regex: Regex,
    email_regex: Regex,
    base64_regex: Regex,
    format_regex: Regex,
    user_agent_regex: Regex,
}

impl SemanticClassifier {
    pub fn new() -> Self {
        // Initialize with compiled regex patterns
    }
    
    pub fn classify(&self, text: &str, context: &StringContext) -> Vec<Tag> {
        // Pattern matching logic
    }
}
```

### Regex Patterns

Based on `docs/src/classification.md`:

1. **GUID/UUID**
   - Pattern: `\{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\}`
   - Example: `{12345678-1234-1234-1234-123456789abc}`
   - Validation: Format compliance, version field checking

2. **Email Address**
   - Pattern: `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`
   - Example: `admin@malware.com`
   - Validation: RFC compliance, domain validation

3. **Base64**
   - Pattern: `[A-Za-z0-9+/]{20,}={0,2}`
   - Example: `SGVsbG8gV29ybGQ=`
   - Validation: Length divisibility by 4, padding correctness, minimum length threshold

4. **Printf Format Strings**
   - Pattern: `%[sdxofcpn]|%\d+[sdxofcpn]|\{\d+\}`
   - Examples: `Error: %s at line %d`, `User {0} logged in`
   - Context: Proximity to other format strings, common in `.rodata`

5. **User Agent**
   - Pattern: `Mozilla/[0-9.]+|Chrome/[0-9.]+|Safari/[0-9.]+|AppleWebKit/[0-9.]+`
   - Example: `Mozilla/5.0 (Windows NT 10.0; Win64; x64)`
   - Validation: Known browser identifiers, version format

## Acceptance Criteria

- [ ] `SemanticClassifier` struct implemented with all five pattern types
- [ ] Each pattern has dedicated detection method with confidence scoring
- [ ] Context-aware classification considers section type and binary format
- [ ] False positive reduction through validation (length, entropy, format)
- [ ] Regex patterns compiled once and cached using `lazy_static`
- [ ] Integration with `FoundString` type to populate `tags` field
- [ ] Comprehensive unit tests for each pattern type:
  - [ ] Positive matches (valid GUIDs, emails, Base64, format strings, user agents)
  - [ ] Negative matches (invalid formats, edge cases)
  - [ ] Context-dependent behavior
  - [ ] Multi-tag scenarios (string matching multiple patterns)
- [ ] Integration tests with real binary samples
- [ ] Documentation with examples and pattern explanations
- [ ] Benchmark tests for performance validation

## Test Coverage Requirements

### Unit Tests (`tests/classification_tests.rs`)

```rust
#[test]
fn test_guid_detection() {
    // Valid GUID formats
    // Invalid GUID formats
    // Case sensitivity
}

#[test]
fn test_email_detection() {
    // Valid emails
    // Invalid emails (missing @, invalid TLD)
}

#[test]
fn test_base64_detection() {
    // Valid Base64 (with/without padding)
    // Invalid Base64 (wrong length, invalid characters)
    // Minimum length threshold
}

#[test]
fn test_format_string_detection() {
    // Printf-style: %s, %d, %x
    // Python-style: {0}, {1}
    // Mixed format strings
}

#[test]
fn test_user_agent_detection() {
    // Common browsers
    // Mobile user agents
    // Bot user agents
}

#[test]
fn test_false_positive_reduction() {
    // High-entropy binary data
    // Very short matches
    // Invalid context
}
```

### Integration Tests

Test with real binaries containing these patterns extracted from:
- ELF binaries with GUIDs in `.rodata`
- PE files with user agents in `.rdata`
- Mach-O binaries with format strings in `__TEXT,__cstring`

## Performance Considerations

- Use `lazy_static` for one-time regex compilation
- Implement short-circuit evaluation (check simpler patterns first)
- Consider minimum string length before applying expensive regex
- Profile regex performance on large binaries

## Related Issues

- Blocks: #36 (Main String Extraction Orchestrator)
- Blocks: #37 (Complete Pipeline Integration)
- Related: #33 (Regex Caching)

## References

- Detailed patterns: `docs/src/classification.md`
- Type definitions: `src/types.rs`
- Tag enum: Lines 20-40 in `src/types.rs`

## Implementation Notes

1. Start with `SemanticClassifier` struct definition
2. Implement each pattern matcher as a separate method
3. Add validation logic for each pattern type
4. Integrate with existing `FoundString` and `Tag` types
5. Write comprehensive unit tests for each pattern
6. Add integration tests with real binaries
7. Optimize with benchmarking
8. Update documentation with usage examples
EOF

# Update the issue
gh issue edit 18 \
  --repo EvilBit-Labs/StringyMcStringFace \
  --title "Implement Semantic Pattern Matching: GUID, Email, Base64, Printf Format, and User Agent Detection" \
  --body-file /tmp/enhanced_issue.md

# Add priority:medium label
gh issue edit 18 \
  --repo EvilBit-Labs/StringyMcStringFace \
  --add-label "priority:medium"

echo -e "\n=== Updated Issue ==="
gh issue view 18 --repo EvilBit-Labs/StringyMcStringFace

Length of output: 6043

---

@unclesp1d3r I've successfully enhanced issue #18 with comprehensive technical details and improved metadata. Here's what was updated:

🎯 Title Enhancement

Before: "Remaining Semantic Patterns"
After: "Implement Semantic Pattern Matching: GUID, Email, Base64, Printf Format, and User Agent Detection"

📋 Enhanced Issue Body

The issue now includes:

Current State Analysis

• Tag enum already defined in src/types.rs
• Classification module is an empty stub
• Documentation exists with comprehensive patterns
• Missing regex crate dependency

Technical Requirements

• Specific dependencies to add (regex 1.10, lazy_static 1.4)
• SemanticClassifier struct architecture
• Integration approach with existing types

Regex Patterns & Validation

All five pattern types with detailed specifications:

1. GUID/UUID: \{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}...\} with version field checking
2. Email: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} with RFC compliance
3. Base64: [A-Za-z0-9+/]{20,}={0,2} with padding validation
4. Printf Format: %[sdxofcpn]|%\d+[sdxofcpn]|\{\d+\} with context awareness
5. User Agent: Mozilla/[0-9.]+|Chrome/[0-9.]+... with browser identification

Detailed Acceptance Criteria

• Implementation checklist for SemanticClassifier
• Confidence scoring requirements
• False positive reduction strategies
• Comprehensive test coverage expectations

Test Requirements

• Unit test structure for each pattern type
• Integration tests with real ELF, PE, and Mach-O binaries
• Performance benchmarking requirements

Performance Considerations

• Regex compilation caching strategy
• Short-circuit evaluation approach
• Minimum length thresholds

🏷️ Metadata Updates

• ✅ Label Added: priority:medium (appropriate for non-MVP but important feature)
• ✅ Existing Labels Retained: area:analyzer, type:enhancement, status:backlog, lang:rust, needs:tests
• ✅ Assignee: unclesp1d3r (unchanged)
• ✅ Milestone: v0.1 (unchanged)

🔗 Related Issues

The issue now references:

• Blocks: Implement Main String Extraction Orchestrator and Pipeline Integration #36 (Main String Extraction Orchestrator), Complete End-to-End Pipeline Integration with Error Recovery and Testing #37 (Complete Pipeline Integration)
• Related: Performance: Implement Regex Compilation Caching for Semantic Classifier #33 (Regex Caching)

📚 References Added

• Links to docs/src/classification.md for detailed patterns
• References to src/types.rs for type definitions
• Specific line numbers for Tag enum location

The issue is now ready for implementation with all the context, technical specifications, and acceptance criteria needed to complete this feature successfully! 🚀

[traycerai]

Plan

Observations

The codebase has a solid foundation with:

• Complete container parsing infrastructure (ELF, PE, Mach-O)
• Well-defined type system with all Tag variants already in place
• Empty classification module stub ready for implementation
• Comprehensive documentation with detailed patterns and examples
• Test infrastructure using insta for snapshot testing
• Clear architecture following a pipeline approach

The issue requires implementing semantic pattern matching for 5 specific patterns: GUID, Email, Base64, Printf Format, and User Agent. The StringContext struct mentioned in documentation doesn't exist yet and needs to be created.

Approach

Implement the SemanticClassifier in src/classification/mod.rs with regex-based pattern matching for the five required semantic categories. Add necessary dependencies (regex and lazy_static) to Cargo.toml. Create the StringContext struct in src/types.rs to support context-aware classification. Implement comprehensive unit tests and integration tests following the existing test patterns. The implementation will use lazy_static for regex compilation caching and include validation logic for each pattern type to reduce false positives.

Reasoning

I explored the repository structure and identified that the container parsing module is complete while the classification module is an empty stub. I examined the type definitions to understand the Tag enum and FoundString structure. I reviewed the comprehensive documentation in docs/src/classification.md which provides detailed regex patterns and validation requirements. I analyzed the existing test structure to understand the testing patterns using insta for snapshots. I checked the architecture documentation to understand how the classification module fits into the overall pipeline and reviewed the ranking system to understand how scores and tags are used.

Proposed File Changes

📄 Cargo.toml ^(MODIFY) 🔗

References

• 📄 benches/classification.rs ^(NEW)

Add two new dependencies to the [dependencies] section:

• regex = "1.10" for pattern matching
• lazy_static = "1.4" for regex compilation caching

These dependencies are required for implementing the semantic pattern matching system with efficient regex compilation.
Add a new benchmark entry to the [[bench]] section (after the existing elf benchmark around line 40):

[[bench]]
name = "classification"
harness = false

This registers the new classification benchmark with Cargo so it can be run with cargo bench --bench classification.

📄 src/types.rs ^(MODIFY) 🔗

References

• 📄 docs/src/classification.md ^(MODIFY)

Add the StringContext struct after the FoundString struct definition (around line 164). The struct should include:

• section_type: SectionType - the type of section where the string was found
• section_name: Option<String> - the name of the section
• binary_format: BinaryFormat - the format of the binary (ELF, PE, Mach-O)
• encoding: Encoding - the encoding of the string
• source: StringSource - where the string came from (section data, import, etc.)

This struct provides context information needed for context-aware classification decisions, as referenced in docs/src/classification.md.

📄 src/classification/mod.rs ^(MODIFY) 🔗

References

• 📄 docs/src/classification.md ^(MODIFY)
• 📄 .kiro/specs/stringy-binary-analyzer/design.md
• 📄 src/types.rs ^(MODIFY)

Implement the complete semantic classification system:

1. Imports: Add necessary imports for regex, lazy_static, and types from crate::types (Tag, StringContext, etc.)

2. Regex Patterns: Use lazy_static! macro to define and compile regex patterns once:

   • GUID_REGEX: Pattern \{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\} for GUID detection
   • EMAIL_REGEX: Pattern [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} for email addresses
   • BASE64_REGEX: Pattern [A-Za-z0-9+/]{20,}={0,2} for Base64 data (minimum 20 chars)
   • FORMAT_REGEX: Pattern %[sdxofcpn]|%\d+[sdxofcpn]|\{\d+\} for printf-style and Python-style format strings
   • USER_AGENT_REGEX: Pattern Mozilla/[0-9.]+|Chrome/[0-9.]+|Safari/[0-9.]+|AppleWebKit/[0-9.]+ for user agent strings

3. SemanticClassifier struct: Create the main classifier with a new() constructor

4. classify method: Implement pub fn classify(&self, text: &str, context: &StringContext) -> Vec<Tag> that:

   • Checks each pattern against the input text
   • Calls validation methods for each match
   • Returns a vector of applicable tags
   • Avoids duplicate tags

5. Validation methods (private helper functions):

   • is_valid_guid: Validates GUID format compliance
   • is_valid_email: Validates email format (checks for valid TLD, proper @ placement)
   • is_valid_base64: Validates Base64 (checks length divisibility by 4, padding correctness, minimum length threshold)
   • is_valid_format_string: Validates format strings (checks for reasonable format specifier count)
   • is_valid_user_agent: Validates user agent strings (checks for known browser identifiers)

6. Context-aware helpers:

   • should_boost_confidence: Checks if context (section type, source) should increase confidence
   • calculate_min_length: Returns minimum length thresholds based on pattern type

7. False positive reduction:

   • Check minimum length before applying expensive regex
   • Validate entropy for Base64 to avoid matching random hex strings
   • Check context appropriateness (e.g., format strings more likely in StringData sections)

Follow the implementation patterns from docs/src/classification.md and the design document at .kiro/specs/stringy-binary-analyzer/design.md. The classifier should be stateless and thread-safe for potential parallel processing.

📄 src/lib.rs ^(MODIFY) 🔗

References

• 📄 src/types.rs ^(MODIFY)

Update the re-exports section (around line 51) to include StringContext in the public API exports. Add it to the existing pub use types:: block alongside the other exported types like FoundString, Tag, etc.

This makes StringContext available to users of the library without requiring them to import from types directly.

📄 tests/classification_tests.rs ^(NEW) 🔗

References

• 📄 tests/integration_elf.rs
• 📄 src/classification/mod.rs ^(MODIFY)
• 📄 docs/src/classification.md ^(MODIFY)

Create comprehensive unit tests for the classification system:

1. GUID Detection Tests (test_guid_detection):

   • Valid GUID formats with braces: {12345678-1234-1234-1234-123456789abc}
   • Valid GUIDs with different case combinations
   • Invalid GUIDs: missing braces, wrong length, invalid characters
   • Edge cases: partial GUIDs, GUIDs without hyphens

2. Email Detection Tests (test_email_detection):

   • Valid emails: admin@malware.com, user.name+tag@example.co.uk
   • Invalid emails: missing @, invalid TLD, multiple @, no domain
   • Edge cases: very long emails, emails with special characters

3. Base64 Detection Tests (test_base64_detection):

   • Valid Base64 with padding: SGVsbG8gV29ybGQ=
   • Valid Base64 without padding
   • Invalid Base64: wrong length, invalid characters, incorrect padding
   • Minimum length threshold testing (should reject strings < 20 chars)
   • High-entropy binary data that looks like Base64

4. Format String Detection Tests (test_format_string_detection):

   • Printf-style: Error: %s at line %d, Value: %x
   • Python-style: User {0} logged in, Result: {1}
   • Mixed format strings
   • Invalid format strings: malformed specifiers

5. User Agent Detection Tests (test_user_agent_detection):

   • Common browsers: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
   • Chrome user agents
   • Safari user agents
   • Mobile user agents
   • Bot user agents

6. False Positive Reduction Tests (test_false_positive_reduction):

   • High-entropy binary data that shouldn't match
   • Very short strings that shouldn't match
   • Invalid context scenarios

7. Multi-Tag Tests (test_multi_tag_scenarios):

   • Strings that match multiple patterns
   • Ensure no duplicate tags

8. Context-Aware Tests (test_context_aware_classification):

   • Test how section type affects classification
   • Test how source type affects classification
   • Test format-specific behavior

Use the test helper function pattern from tests/integration_elf.rs for creating test contexts. Import necessary types from stringy::classification and stringy::types.

📄 tests/classification_integration_tests.rs ^(NEW) 🔗

References

• 📄 tests/integration_elf.rs
• 📄 tests/integration_pe.rs
• 📄 tests/integration_macho.rs
• 📄 src/classification/mod.rs ^(MODIFY)

Create integration tests that use real binary fixtures:

1. Setup: Create helper function create_test_context that builds a StringContext with appropriate values for testing

2. ELF Binary Classification Test (test_elf_string_classification):

   • Read the ELF fixture from tests/fixtures/test_binary_elf
   • Parse it using ElfParser (similar to tests/integration_elf.rs)
   • Create sample strings that would be found in the binary
   • Apply classification to these strings
   • Verify expected tags are applied

3. PE Binary Classification Test (test_pe_string_classification):

   • Read the PE fixture from tests/fixtures/test_binary_pe.exe
   • Parse it using PeParser
   • Test classification of strings from different PE sections (.rdata, .rsrc)
   • Verify context-aware behavior for PE-specific patterns

4. Mach-O Binary Classification Test (test_macho_string_classification):

   • Read the Mach-O fixture from tests/fixtures/test_binary_macho
   • Parse it using MachoParser
   • Test classification of strings from Mach-O sections

5. Real-World Pattern Test (test_real_world_patterns):

   • Test with strings commonly found in malware/binaries:
     • C2 URLs: https://evil.com/payload
     • Registry persistence: HKEY_LOCAL_MACHINE\SOFTWARE\...
     • GUIDs from COM objects
     • User agents from network code
     • Format strings from logging code

6. Performance Test (test_classification_performance):

   • Classify a large batch of strings (1000+)
   • Ensure performance is acceptable
   • Verify regex compilation is cached (not recompiled per call)

Follow the pattern from existing integration tests in tests/integration_elf.rs, tests/integration_pe.rs, and tests/integration_macho.rs. Use the get_fixture_path helper and proper error handling.

📄 benches/classification.rs ^(NEW) 🔗

References

• 📄 benches/elf.rs
• 📄 src/classification/mod.rs ^(MODIFY)

Create benchmark tests using the criterion crate (already in dev-dependencies):

1. Setup: Import criterion::{black_box, criterion_group, criterion_main, Criterion} and the classification module

2. Regex Compilation Benchmark (bench_regex_compilation):

   • Measure the time to create a new SemanticClassifier instance
   • Verify that lazy_static caching works (subsequent calls should be instant)

3. Single Pattern Benchmarks:

   • bench_guid_classification: Classify strings with GUID patterns
   • bench_email_classification: Classify strings with email patterns
   • bench_base64_classification: Classify strings with Base64 patterns
   • bench_format_string_classification: Classify format strings
   • bench_user_agent_classification: Classify user agent strings

4. Batch Classification Benchmark (bench_batch_classification):

   • Classify 1000 strings with mixed patterns
   • Measure throughput (strings per second)

5. Worst Case Benchmark (bench_worst_case):

   • Test with strings that don't match any pattern (all regex checks fail)
   • Verify short-circuit optimization works

6. Context Creation Benchmark (bench_context_creation):

   • Measure overhead of creating StringContext instances

Follow the benchmark pattern from benches/elf.rs. Use criterion_group! and criterion_main! macros to register benchmarks. Set appropriate sample sizes and measurement times.

📄 docs/src/classification.md ^(MODIFY) 🔗

References

• 📄 src/classification/mod.rs ^(MODIFY)
• 📄 tests/classification_tests.rs ^(NEW)

Update the documentation to reflect the actual implementation:

1. Implementation Status: Update the section around line 92-134 to indicate that the SemanticClassifier is now implemented (change from example/pseudocode to actual implementation reference)

2. Usage Examples: Add a new section showing how to use the classifier:

   • Creating a SemanticClassifier instance
   • Creating a StringContext
   • Calling the classify method
   • Interpreting the returned tags

3. Pattern Details: Ensure the documented regex patterns match the actual implementation in src/classification/mod.rs

4. Validation Rules: Document the validation logic for each pattern type:

   • GUID: Format compliance requirements
   • Email: TLD validation, RFC compliance notes
   • Base64: Length requirements, padding rules, entropy checks
   • Format String: Specifier validation
   • User Agent: Known browser identifier requirements

5. Performance Notes: Add a section documenting:

   • Regex compilation caching via lazy_static
   • Short-circuit evaluation for minimum length checks
   • Thread-safety guarantees

6. Testing: Add a section referencing the test files and how to run them

Diagram

sequenceDiagram
    participant User
    participant Classifier as SemanticClassifier
    participant Regex as Lazy Static Regexes
    participant Validator as Validation Logic
    participant Context as StringContext

    User->>Classifier: classify(text, context)
    Classifier->>Regex: Check GUID pattern
    alt GUID matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_guid(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::Guid
        end
    end
    
    Classifier->>Regex: Check Email pattern
    alt Email matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_email(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::Email
        end
    end
    
    Classifier->>Regex: Check Base64 pattern
    alt Base64 matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_base64(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::Base64
        end
    end
    
    Classifier->>Regex: Check Format String pattern
    alt Format matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_format_string(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::FormatString
        end
    end
    
    Classifier->>Regex: Check User Agent pattern
    alt User Agent matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_user_agent(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::UserAgent
        end
    end
    
    Classifier->>Context: Check context for boosts
    Context-->>Classifier: Section type, source info
    Classifier->>Classifier: Apply context-aware adjustments
    
    Classifier-->>User: Vec<Tag>

Loading

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a solid foundation with:
- Complete container parsing infrastructure (ELF, PE, Mach-O)
- Well-defined type system with all Tag variants already in place
- Empty classification module stub ready for implementation
- Comprehensive documentation with detailed patterns and examples
- Test infrastructure using `insta` for snapshot testing
- Clear architecture following a pipeline approach

The issue requires implementing semantic pattern matching for 5 specific patterns: GUID, Email, Base64, Printf Format, and User Agent. The `StringContext` struct mentioned in documentation doesn't exist yet and needs to be created.

### Approach

Implement the `SemanticClassifier` in `src/classification/mod.rs` with regex-based pattern matching for the five required semantic categories. Add necessary dependencies (`regex` and `lazy_static`) to `Cargo.toml`. Create the `StringContext` struct in `src/types.rs` to support context-aware classification. Implement comprehensive unit tests and integration tests following the existing test patterns. The implementation will use lazy_static for regex compilation caching and include validation logic for each pattern type to reduce false positives.

### Reasoning

I explored the repository structure and identified that the container parsing module is complete while the classification module is an empty stub. I examined the type definitions to understand the Tag enum and FoundString structure. I reviewed the comprehensive documentation in `docs/src/classification.md` which provides detailed regex patterns and validation requirements. I analyzed the existing test structure to understand the testing patterns using `insta` for snapshots. I checked the architecture documentation to understand how the classification module fits into the overall pipeline and reviewed the ranking system to understand how scores and tags are used.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant Classifier as SemanticClassifier
    participant Regex as Lazy Static Regexes
    participant Validator as Validation Logic
    participant Context as StringContext

    User->>Classifier: classify(text, context)
    Classifier->>Regex: Check GUID pattern
    alt GUID matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_guid(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::Guid
        end
    end
    
    Classifier->>Regex: Check Email pattern
    alt Email matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_email(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::Email
        end
    end
    
    Classifier->>Regex: Check Base64 pattern
    alt Base64 matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_base64(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::Base64
        end
    end
    
    Classifier->>Regex: Check Format String pattern
    alt Format matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_format_string(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::FormatString
        end
    end
    
    Classifier->>Regex: Check User Agent pattern
    alt User Agent matches
        Regex-->>Classifier: Match found
        Classifier->>Validator: is_valid_user_agent(text)
        Validator-->>Classifier: Valid/Invalid
        alt Valid
            Classifier->>Classifier: Add Tag::UserAgent
        end
    end
    
    Classifier->>Context: Check context for boosts
    Context-->>Classifier: Section type, source info
    Classifier->>Classifier: Apply context-aware adjustments
    
    Classifier-->>User: Vec<Tag>

## Proposed File Changes

### Cargo.toml(MODIFY)

References: 

- benches/classification.rs(NEW)

Add two new dependencies to the `[dependencies]` section:
- `regex = "1.10"` for pattern matching
- `lazy_static = "1.4"` for regex compilation caching

These dependencies are required for implementing the semantic pattern matching system with efficient regex compilation.
Add a new benchmark entry to the `[[bench]]` section (after the existing `elf` benchmark around line 40):

```toml
[[bench]]
name = "classification"
harness = false
```

This registers the new classification benchmark with Cargo so it can be run with `cargo bench --bench classification`.

### src/types.rs(MODIFY)

References: 

- docs/src/classification.md(MODIFY)

Add the `StringContext` struct after the `FoundString` struct definition (around line 164). The struct should include:
- `section_type: SectionType` - the type of section where the string was found
- `section_name: Option<String>` - the name of the section
- `binary_format: BinaryFormat` - the format of the binary (ELF, PE, Mach-O)
- `encoding: Encoding` - the encoding of the string
- `source: StringSource` - where the string came from (section data, import, etc.)

This struct provides context information needed for context-aware classification decisions, as referenced in `docs/src/classification.md`.

### src/classification/mod.rs(MODIFY)

References: 

- docs/src/classification.md(MODIFY)
- .kiro/specs/stringy-binary-analyzer/design.md
- src/types.rs(MODIFY)

Implement the complete semantic classification system:

1. **Imports**: Add necessary imports for `regex`, `lazy_static`, and types from `crate::types` (Tag, StringContext, etc.)

2. **Regex Patterns**: Use `lazy_static!` macro to define and compile regex patterns once:
   - `GUID_REGEX`: Pattern `\{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\}` for GUID detection
   - `EMAIL_REGEX`: Pattern `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` for email addresses
   - `BASE64_REGEX`: Pattern `[A-Za-z0-9+/]{20,}={0,2}` for Base64 data (minimum 20 chars)
   - `FORMAT_REGEX`: Pattern `%[sdxofcpn]|%\d+[sdxofcpn]|\{\d+\}` for printf-style and Python-style format strings
   - `USER_AGENT_REGEX`: Pattern `Mozilla/[0-9.]+|Chrome/[0-9.]+|Safari/[0-9.]+|AppleWebKit/[0-9.]+` for user agent strings

3. **SemanticClassifier struct**: Create the main classifier with a `new()` constructor

4. **classify method**: Implement `pub fn classify(&self, text: &str, context: &StringContext) -> Vec<Tag>` that:
   - Checks each pattern against the input text
   - Calls validation methods for each match
   - Returns a vector of applicable tags
   - Avoids duplicate tags

5. **Validation methods** (private helper functions):
   - `is_valid_guid`: Validates GUID format compliance
   - `is_valid_email`: Validates email format (checks for valid TLD, proper @ placement)
   - `is_valid_base64`: Validates Base64 (checks length divisibility by 4, padding correctness, minimum length threshold)
   - `is_valid_format_string`: Validates format strings (checks for reasonable format specifier count)
   - `is_valid_user_agent`: Validates user agent strings (checks for known browser identifiers)

6. **Context-aware helpers**:
   - `should_boost_confidence`: Checks if context (section type, source) should increase confidence
   - `calculate_min_length`: Returns minimum length thresholds based on pattern type

7. **False positive reduction**:
   - Check minimum length before applying expensive regex
   - Validate entropy for Base64 to avoid matching random hex strings
   - Check context appropriateness (e.g., format strings more likely in StringData sections)

Follow the implementation patterns from `docs/src/classification.md` and the design document at `.kiro/specs/stringy-binary-analyzer/design.md`. The classifier should be stateless and thread-safe for potential parallel processing.

### src/lib.rs(MODIFY)

References: 

- src/types.rs(MODIFY)

Update the re-exports section (around line 51) to include `StringContext` in the public API exports. Add it to the existing `pub use types::` block alongside the other exported types like `FoundString`, `Tag`, etc.

This makes `StringContext` available to users of the library without requiring them to import from `types` directly.

### tests/classification_tests.rs(NEW)

References: 

- tests/integration_elf.rs
- src/classification/mod.rs(MODIFY)
- docs/src/classification.md(MODIFY)

Create comprehensive unit tests for the classification system:

1. **GUID Detection Tests** (`test_guid_detection`):
   - Valid GUID formats with braces: `{12345678-1234-1234-1234-123456789abc}`
   - Valid GUIDs with different case combinations
   - Invalid GUIDs: missing braces, wrong length, invalid characters
   - Edge cases: partial GUIDs, GUIDs without hyphens

2. **Email Detection Tests** (`test_email_detection`):
   - Valid emails: `admin@malware.com`, `user.name+tag@example.co.uk`
   - Invalid emails: missing @, invalid TLD, multiple @, no domain
   - Edge cases: very long emails, emails with special characters

3. **Base64 Detection Tests** (`test_base64_detection`):
   - Valid Base64 with padding: `SGVsbG8gV29ybGQ=`
   - Valid Base64 without padding
   - Invalid Base64: wrong length, invalid characters, incorrect padding
   - Minimum length threshold testing (should reject strings < 20 chars)
   - High-entropy binary data that looks like Base64

4. **Format String Detection Tests** (`test_format_string_detection`):
   - Printf-style: `Error: %s at line %d`, `Value: %x`
   - Python-style: `User {0} logged in`, `Result: {1}`
   - Mixed format strings
   - Invalid format strings: malformed specifiers

5. **User Agent Detection Tests** (`test_user_agent_detection`):
   - Common browsers: `Mozilla/5.0 (Windows NT 10.0; Win64; x64)`
   - Chrome user agents
   - Safari user agents
   - Mobile user agents
   - Bot user agents

6. **False Positive Reduction Tests** (`test_false_positive_reduction`):
   - High-entropy binary data that shouldn't match
   - Very short strings that shouldn't match
   - Invalid context scenarios

7. **Multi-Tag Tests** (`test_multi_tag_scenarios`):
   - Strings that match multiple patterns
   - Ensure no duplicate tags

8. **Context-Aware Tests** (`test_context_aware_classification`):
   - Test how section type affects classification
   - Test how source type affects classification
   - Test format-specific behavior

Use the test helper function pattern from `tests/integration_elf.rs` for creating test contexts. Import necessary types from `stringy::classification` and `stringy::types`.

### tests/classification_integration_tests.rs(NEW)

References: 

- tests/integration_elf.rs
- tests/integration_pe.rs
- tests/integration_macho.rs
- src/classification/mod.rs(MODIFY)

Create integration tests that use real binary fixtures:

1. **Setup**: Create helper function `create_test_context` that builds a `StringContext` with appropriate values for testing

2. **ELF Binary Classification Test** (`test_elf_string_classification`):
   - Read the ELF fixture from `tests/fixtures/test_binary_elf`
   - Parse it using `ElfParser` (similar to `tests/integration_elf.rs`)
   - Create sample strings that would be found in the binary
   - Apply classification to these strings
   - Verify expected tags are applied

3. **PE Binary Classification Test** (`test_pe_string_classification`):
   - Read the PE fixture from `tests/fixtures/test_binary_pe.exe`
   - Parse it using `PeParser`
   - Test classification of strings from different PE sections (.rdata, .rsrc)
   - Verify context-aware behavior for PE-specific patterns

4. **Mach-O Binary Classification Test** (`test_macho_string_classification`):
   - Read the Mach-O fixture from `tests/fixtures/test_binary_macho`
   - Parse it using `MachoParser`
   - Test classification of strings from Mach-O sections

5. **Real-World Pattern Test** (`test_real_world_patterns`):
   - Test with strings commonly found in malware/binaries:
     - C2 URLs: `https://evil.com/payload`
     - Registry persistence: `HKEY_LOCAL_MACHINE\SOFTWARE\...`
     - GUIDs from COM objects
     - User agents from network code
     - Format strings from logging code

6. **Performance Test** (`test_classification_performance`):
   - Classify a large batch of strings (1000+)
   - Ensure performance is acceptable
   - Verify regex compilation is cached (not recompiled per call)

Follow the pattern from existing integration tests in `tests/integration_elf.rs`, `tests/integration_pe.rs`, and `tests/integration_macho.rs`. Use the `get_fixture_path` helper and proper error handling.

### benches/classification.rs(NEW)

References: 

- benches/elf.rs
- src/classification/mod.rs(MODIFY)

Create benchmark tests using the `criterion` crate (already in dev-dependencies):

1. **Setup**: Import `criterion::{black_box, criterion_group, criterion_main, Criterion}` and the classification module

2. **Regex Compilation Benchmark** (`bench_regex_compilation`):
   - Measure the time to create a new `SemanticClassifier` instance
   - Verify that lazy_static caching works (subsequent calls should be instant)

3. **Single Pattern Benchmarks**:
   - `bench_guid_classification`: Classify strings with GUID patterns
   - `bench_email_classification`: Classify strings with email patterns
   - `bench_base64_classification`: Classify strings with Base64 patterns
   - `bench_format_string_classification`: Classify format strings
   - `bench_user_agent_classification`: Classify user agent strings

4. **Batch Classification Benchmark** (`bench_batch_classification`):
   - Classify 1000 strings with mixed patterns
   - Measure throughput (strings per second)

5. **Worst Case Benchmark** (`bench_worst_case`):
   - Test with strings that don't match any pattern (all regex checks fail)
   - Verify short-circuit optimization works

6. **Context Creation Benchmark** (`bench_context_creation`):
   - Measure overhead of creating `StringContext` instances

Follow the benchmark pattern from `benches/elf.rs`. Use `criterion_group!` and `criterion_main!` macros to register benchmarks. Set appropriate sample sizes and measurement times.

### docs/src/classification.md(MODIFY)

References: 

- src/classification/mod.rs(MODIFY)
- tests/classification_tests.rs(NEW)

Update the documentation to reflect the actual implementation:

1. **Implementation Status**: Update the section around line 92-134 to indicate that the `SemanticClassifier` is now implemented (change from example/pseudocode to actual implementation reference)

2. **Usage Examples**: Add a new section showing how to use the classifier:
   - Creating a `SemanticClassifier` instance
   - Creating a `StringContext`
   - Calling the `classify` method
   - Interpreting the returned tags

3. **Pattern Details**: Ensure the documented regex patterns match the actual implementation in `src/classification/mod.rs`

4. **Validation Rules**: Document the validation logic for each pattern type:
   - GUID: Format compliance requirements
   - Email: TLD validation, RFC compliance notes
   - Base64: Length requirements, padding rules, entropy checks
   - Format String: Specifier validation
   - User Agent: Known browser identifier requirements

5. **Performance Notes**: Add a section documenting:
   - Regex compilation caching via lazy_static
   - Short-circuit evaluation for minimum length checks
   - Thread-safety guarantees

6. **Testing**: Add a section referencing the test files and how to run them

---

Developer Humor

🔍 Five patterns walk into a binary...
GUID says \I'm {unique}!\
Email replies \user@domain.com\
Base64 mumbles \SGVsbG8=\
Printf shouts \Error: %s!\
User-Agent whispers \Mozilla/5.0...\

The classifier tags them all! 🏷️✨

---

Execution Information

Branch: main
Commit: bdd6206

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.