Add output filtering and exclusion support

[sgaunet]

Feature Request

Allow users to filter or exclude certain log lines from output based on patterns or log levels.

Use Cases

1. Exclude debug logs from production
2. Filter out noisy messages
3. Focus on errors/warnings only
4. Remove known false positives

Proposed Configuration

filter:
  enabled: true
  
  # Exclude lines matching patterns
  exclude_patterns:
    - "DEBUG.*"
    - "verbose output"
    - "heartbeat"
  
  # Exclude by log level
  exclude_levels:
    - "DEBUG"
    - "TRACE"
  
  # Only include certain levels (alternative to exclude)
  include_levels:
    - "ERROR"
    - "WARN"
  
  # Include only lines matching patterns
  include_patterns:
    - "important.*"

CLI Usage

# Exclude DEBUG level
logwrap --exclude-level DEBUG -- myapp

# Only show ERROR and WARN
logwrap --include-level ERROR --include-level WARN -- myapp

# Exclude pattern
logwrap --exclude-pattern "heartbeat" -- myapp

Implementation

// pkg/filter/filter.go

type Filter struct {
    ExcludePatterns []*regexp.Regexp
    ExcludeLevels   map[string]bool
    IncludeLevels   map[string]bool
    IncludePatterns []*regexp.Regexp
}

func (f *Filter) ShouldInclude(line, level string) bool {
    // If include levels specified, must match
    if len(f.IncludeLevels) > 0 && !f.IncludeLevels[level] {
        return false
    }
    
    // If exclude levels specified, must not match
    if f.ExcludeLevels[level] {
        return false
    }
    
    // Check include patterns
    if len(f.IncludePatterns) > 0 {
        matched := false
        for _, pattern := range f.IncludePatterns {
            if pattern.MatchString(line) {
                matched = true
                break
            }
        }
        if !matched {
            return false
        }
    }
    
    // Check exclude patterns
    for _, pattern := range f.ExcludePatterns {
        if pattern.MatchString(line) {
            return false
        }
    }
    
    return true
}

Integration

// pkg/processor/processor.go

func (p *Processor) processLine(line, stream string) {
    // Format the line
    formatted := p.formatter.Format(line, stream)
    
    // Apply filter
    if p.filter != nil {
        level := p.formatter.GetLogLevel(line)
        if !p.filter.ShouldInclude(line, level) {
            return  // Skip this line
        }
    }
    
    // Output the line
    fmt.Println(formatted)
}

Examples

Example 1: Production Deployment

# production.yaml
filter:
  enabled: true
  exclude_levels:
    - "DEBUG"
    - "TRACE"

logwrap -config production.yaml -- ./app
# Only ERROR, WARN, INFO shown

Example 2: Focus on Errors

logwrap --include-level ERROR --include-level WARN -- ./app
# Only shows errors and warnings

Example 3: Remove Noise

filter:
  enabled: true
  exclude_patterns:
    - "^Heartbeat:"
    - "Connection pool stats:"
    - "GC stats:"

Metrics Integration

Add filter metrics:

{
  "lines_processed": 125432,
  "lines_filtered": 45123,
  "lines_output": 80309,
  "filter_rate": 0.36
}

Performance Considerations

• Regex matching has overhead (~1-2µs per line)
• Cache compiled regexes
• Consider fast path for simple string matching
• Document performance impact

Implementation Checklist

• Create filter package
• Implement pattern matching
• Implement level filtering
• Add CLI flags
• Add config file support
• Integrate with processor
• Add tests for filter logic
• Add benchmarks (performance impact)
• Document in README

Alternative: Use grep

Note: Users can already filter with Unix tools:

# Exclude DEBUG
logwrap -- myapp | grep -v DEBUG

# Only ERROR/WARN
logwrap -- myapp | grep -E "(ERROR|WARN)"

Why build this in:

• Filtering before formatting is more efficient
• Can filter based on detected level (not just text)
• More user-friendly (no need to learn grep)
• Better integration with metrics

Related Issues

• Implement or remove incomplete buffering feature #16 - Buffering modes (filtering affects buffering)
• Add optional metrics and observability support #28 - Metrics (track filter effectiveness)

References

• Go regexp: https://pkg.go.dev/regexp
• Logging best practices: https://12factor.net/logs