Skip to content

Add generic extended benchmarking harness#260

Open
la3lma wants to merge 27 commits intomasterfrom
codex/add-benchmarking-framework-for-regex-matchers
Open

Add generic extended benchmarking harness#260
la3lma wants to merge 27 commits intomasterfrom
codex/add-benchmarking-framework-for-regex-matchers

Conversation

@la3lma
Copy link
Owner

@la3lma la3lma commented Dec 18, 2025

Summary

  • add an extended-benchmarking directory with a generic Python runner and documentation for comparing regex engines
  • provide sample corpus/pattern files and configuration entries covering python, java.util.regex, and placeholders for rmatch/RE2J/Hyperscan
  • include a minimal java.util.regex runner and sample output to illustrate the harness results

Testing

  • python extended-benchmarking/runner.py --patterns extended-benchmarking/patterns/sample_patterns.txt --corpus extended-benchmarking/corpora/sample_corpus.txt --engines extended-benchmarking/configs/example_engines.json --output-dir extended-benchmarking/results/sample --iterations 1

Codex Task

la3lma and others added 2 commits December 18, 2025 23:21
@la3lma
Merge 269decd into 4fff153
53b2d62
@actions-user
chore: update performance data
25db62f 
    Auto-generated performance timeline charts and README table
    after running performance benchmarks

    [skip ci]
@github-actions
Copy link

github-actions bot commented Dec 18, 2025

✅ PASS Performance Comparison

Result: ⚠️ Cross-architecture comparison: Baseline (x86_64_Azure_GitHub_Actions_Runner) vs Current (amd64_AMD_EPYC_7763_64_Core_Processor_4). Normalization applied. Performance improvement: time 100.0%, memory 83.2% Note: For most accurate comparison, run on same architecture as baseline.

💻 Test Environment

Attribute Value
Architecture amd64_AMD_EPYC_7763_64_Core_Processor_4
Normalization Score 458607 ops/ms
Java Version 25.0.1
OS Linux 6.11.0-1018-azure
Test Runs 5 iterations

⚠️ Architecture Mismatch: Baseline was run on x86_64_Azure_GitHub_Actions_Runner. Performance normalization applied for fair comparison.

📊 rmatch Performance Metrics

Metric Current Baseline Δ
Execution Time 164 ms Baseline 100.0%
Memory Usage 166 MB Baseline 83.2%
Statistical Significance ⚠️ Low - -

🔴 Native Java Matcher Performance

Metric Current Value
Execution Time 2145 ms
Memory Usage 161 MB
Test Runs 5 iterations

⚖️ Relative Performance (rmatch vs Java)

Comparison Ratio Interpretation
Time Ratio (rmatch/java) 0.08x 🟢 rmatch significantly faster
Memory Ratio (rmatch/java) 1.03x 🟡 comparable memory usage
Overall Assessment - ⚖️ Mixed performance: rmatch better in time

Performance check powered by rmatch automated testing

la3lma and others added 4 commits December 19, 2025 12:20
@la3lma
Merge branch 'master' into codex/add-benchmarking-framework-for-regex…
b1941d0 
…-matchers
@la3lma
Working on tests
d02d073
@la3lma
Merge d02d073 into b5ca21e
7a66c52
@actions-user
chore: update performance data
19316c6 
    Auto-generated performance timeline charts and README table
    after running performance benchmarks

    [skip ci]
@github-actions
Copy link

github-actions bot commented Dec 19, 2025

✅ PASS Performance Comparison

Result: ⚠️ Cross-architecture comparison: Baseline (x86_64_Azure_GitHub_Actions_Runner) vs Current (amd64_AMD_EPYC_7763_64_Core_Processor_4). Normalization applied. Performance improvement: time 100.0%, memory 84.9% Note: For most accurate comparison, run on same architecture as baseline.

💻 Test Environment

Attribute Value
Architecture amd64_AMD_EPYC_7763_64_Core_Processor_4
Normalization Score 517928 ops/ms
Java Version 25.0.1
OS Linux 6.11.0-1018-azure
Test Runs 5 iterations

⚠️ Architecture Mismatch: Baseline was run on x86_64_Azure_GitHub_Actions_Runner. Performance normalization applied for fair comparison.

📊 rmatch Performance Metrics

Metric Current Baseline Δ
Execution Time 164 ms Baseline 100.0%
Memory Usage 150 MB Baseline 84.9%
Statistical Significance ⚠️ Low - -

🔴 Native Java Matcher Performance

Metric Current Value
Execution Time 2175 ms
Memory Usage 156 MB
Test Runs 5 iterations

⚖️ Relative Performance (rmatch vs Java)

Comparison Ratio Interpretation
Time Ratio (rmatch/java) 0.08x 🟢 rmatch significantly faster
Memory Ratio (rmatch/java) 0.96x 🟡 comparable memory usage
Overall Assessment - 🏆 rmatch outperforms Java regex in both time and memory

Performance check powered by rmatch automated testing

la3lma and others added 4 commits December 19, 2025 16:00
@la3lma
Maybe a runnable thing
4da9376
@la3lma
Merge branch 'codex/add-benchmarking-framework-for-regex-matchers' of…
e7ae631 
… ssh://github.com/la3lma/rmatch into codex/add-benchmarking-framework-for-regex-matchers
@la3lma
Merge e7ae631 into b5ca21e
430d49c
@actions-user
chore: update performance data
51f1124 
    Auto-generated performance timeline charts and README table
    after running performance benchmarks

    [skip ci]
@github-actions
Copy link

github-actions bot commented Dec 19, 2025

✅ PASS Performance Comparison

Result: ⚠️ Cross-architecture comparison: Baseline (x86_64_Azure_GitHub_Actions_Runner) vs Current (amd64_AMD_EPYC_7763_64_Core_Processor_4). Normalization applied. Performance improvement: time 100.0%, memory 85.8% Note: For most accurate comparison, run on same architecture as baseline.

💻 Test Environment

Attribute Value
Architecture amd64_AMD_EPYC_7763_64_Core_Processor_4
Normalization Score 481861 ops/ms
Java Version 25.0.1
OS Linux 6.11.0-1018-azure
Test Runs 5 iterations

⚠️ Architecture Mismatch: Baseline was run on x86_64_Azure_GitHub_Actions_Runner. Performance normalization applied for fair comparison.

📊 rmatch Performance Metrics

Metric Current Baseline Δ
Execution Time 193 ms Baseline 100.0%
Memory Usage 140 MB Baseline 85.8%
Statistical Significance ⚠️ Low - -

🔴 Native Java Matcher Performance

Metric Current Value
Execution Time 2171 ms
Memory Usage 151 MB
Test Runs 5 iterations

⚖️ Relative Performance (rmatch vs Java)

Comparison Ratio Interpretation
Time Ratio (rmatch/java) 0.09x 🟢 rmatch significantly faster
Memory Ratio (rmatch/java) 0.93x 🟡 comparable memory usage
Overall Assessment - 🏆 rmatch outperforms Java regex in both time and memory

Performance check powered by rmatch automated testing

la3lma and others added 4 commits December 19, 2025 16:23
@la3lma
Updating .gitignore
d21c63d
@la3lma
Merge branch 'codex/add-benchmarking-framework-for-regex-matchers' of…
ee1b21b 
… ssh://github.com/la3lma/rmatch into codex/add-benchmarking-framework-for-regex-matchers
@la3lma
Merge ee1b21b into b5ca21e
d3c62c3
@actions-user
chore: update performance data
e9468d9 
    Auto-generated performance timeline charts and README table
    after running performance benchmarks

    [skip ci]
@github-actions
Copy link

github-actions bot commented Dec 19, 2025

✅ PASS Performance Comparison

Result: ⚠️ Cross-architecture comparison: Baseline (x86_64_Azure_GitHub_Actions_Runner) vs Current (amd64_AMD_EPYC_7763_64_Core_Processor_4). Normalization applied. Performance improvement: time 100.0%, memory 80.8% Note: For most accurate comparison, run on same architecture as baseline.

💻 Test Environment

Attribute Value
Architecture amd64_AMD_EPYC_7763_64_Core_Processor_4
Normalization Score 479523 ops/ms
Java Version 25.0.1
OS Linux 6.11.0-1018-azure
Test Runs 5 iterations

⚠️ Architecture Mismatch: Baseline was run on x86_64_Azure_GitHub_Actions_Runner. Performance normalization applied for fair comparison.

📊 rmatch Performance Metrics

Metric Current Baseline Δ
Execution Time 163 ms Baseline 100.0%
Memory Usage 190 MB Baseline 80.8%
Statistical Significance ⚠️ Low - -

🔴 Native Java Matcher Performance

Metric Current Value
Execution Time 2240 ms
Memory Usage 202 MB
Test Runs 5 iterations

⚖️ Relative Performance (rmatch vs Java)

Comparison Ratio Interpretation
Time Ratio (rmatch/java) 0.07x 🟢 rmatch significantly faster
Memory Ratio (rmatch/java) 0.94x 🟡 comparable memory usage
Overall Assessment - 🏆 rmatch outperforms Java regex in both time and memory

Performance check powered by rmatch automated testing

la3lma and others added 7 commits December 19, 2025 21:00
@la3lma
Slugging oru way towards progress
21f2808
@la3lma
Merge branch 'codex/add-benchmarking-framework-for-regex-matchers' of…
824b2c4 
… ssh://github.com/la3lma/rmatch into codex/add-benchmarking-framework-for-regex-matchers
@la3lma @claude
Add stable git-stored 10K regex patterns for consistent benchmarking
7238c2e 
- 10,000 validated regex patterns compatible with all engines
- Proper metadata file with seed=12345 matching phase1.json
- Eliminates pattern regeneration for consistent benchmark reuse
- All patterns validated against rmatch, re2j, java-native-optimized, java-native-unfair

🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
@la3lma
In the middle of something, that crashed
91367a1
@la3lma
More stuff to run benchmarks, my it's a big task
dc1d9f2
@la3lma
Merge dc1d9f2 into b5ca21e
0545a0c
@actions-user
chore: update performance data
ccb4db7 
    Auto-generated performance timeline charts and README table
    after running performance benchmarks

    [skip ci]
@github-actions
Copy link

github-actions bot commented Dec 23, 2025

✅ PASS Performance Comparison

Result: ⚠️ Cross-architecture comparison: Baseline (x86_64_Azure_GitHub_Actions_Runner) vs Current (amd64_AMD_EPYC_7763_64_Core_Processor_4). Normalization applied. Performance improvement: time 100.0%, memory 85.0% Note: For most accurate comparison, run on same architecture as baseline.

💻 Test Environment

Attribute Value
Architecture amd64_AMD_EPYC_7763_64_Core_Processor_4
Normalization Score 498356 ops/ms
Java Version 25.0.1
OS Linux 6.11.0-1018-azure
Test Runs 5 iterations

⚠️ Architecture Mismatch: Baseline was run on x86_64_Azure_GitHub_Actions_Runner. Performance normalization applied for fair comparison.

📊 rmatch Performance Metrics

Metric Current Baseline Δ
Execution Time 165 ms Baseline 100.0%
Memory Usage 148 MB Baseline 85.0%
Statistical Significance ⚠️ Low - -

🔴 Native Java Matcher Performance

Metric Current Value
Execution Time 2225 ms
Memory Usage 144 MB
Test Runs 5 iterations

⚖️ Relative Performance (rmatch vs Java)

Comparison Ratio Interpretation
Time Ratio (rmatch/java) 0.07x 🟢 rmatch significantly faster
Memory Ratio (rmatch/java) 1.03x 🟡 comparable memory usage
Overall Assessment - ⚖️ Mixed performance: rmatch better in time

Performance check powered by rmatch automated testing

la3lma and others added 3 commits December 31, 2025 01:47
@la3lma
Merge branch 'codex/add-benchmarking-framework-for-regex-matchers' of…
525799d 
… ssh://github.com/la3lma/rmatch into codex/add-benchmarking-framework-for-regex-matchers
@la3lma
Merge 525799d into b5ca21e
8f7434d
@actions-user
chore: update performance data
e7cdf5f 
    Auto-generated performance timeline charts and README table
    after running performance benchmarks

    [skip ci]
@github-actions
Copy link

github-actions bot commented Dec 31, 2025

✅ PASS Performance Comparison

Result: ⚠️ Cross-architecture comparison: Baseline (x86_64_Azure_GitHub_Actions_Runner) vs Current (amd64_AMD_EPYC_7763_64_Core_Processor_4). Normalization applied. Performance improvement: time 100.0%, memory 81.2% Note: For most accurate comparison, run on same architecture as baseline.

💻 Test Environment

Attribute Value
Architecture amd64_AMD_EPYC_7763_64_Core_Processor_4
Normalization Score 489124 ops/ms
Java Version 25.0.1
OS Linux 6.11.0-1018-azure
Test Runs 5 iterations

⚠️ Architecture Mismatch: Baseline was run on x86_64_Azure_GitHub_Actions_Runner. Performance normalization applied for fair comparison.

📊 rmatch Performance Metrics

Metric Current Baseline Δ
Execution Time 161 ms Baseline 100.0%
Memory Usage 186 MB Baseline 81.2%
Statistical Significance ⚠️ Low - -

🔴 Native Java Matcher Performance

Metric Current Value
Execution Time 2223 ms
Memory Usage 167 MB
Test Runs 5 iterations

⚖️ Relative Performance (rmatch vs Java)

Comparison Ratio Interpretation
Time Ratio (rmatch/java) 0.07x 🟢 rmatch significantly faster
Memory Ratio (rmatch/java) 1.11x 🟡 comparable memory usage
Overall Assessment - ⚖️ Mixed performance: rmatch better in time

Performance check powered by rmatch automated testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

codex

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@la3lma @actions-user