Skip to content

feat: add task_24_access_log_anomaly-physical access control log anomaly detection#90

Open
905timur wants to merge 1 commit intopinchbench:mainfrom
905timur:add-task-24-access-log-anomaly
Open

feat: add task_24_access_log_anomaly-physical access control log anomaly detection#90
905timur wants to merge 1 commit intopinchbench:mainfrom
905timur:add-task-24-access-log-anomaly

Conversation

@905timur
Copy link
Copy Markdown

@905timur 905timur commented Apr 1, 2026

New Task: Access Control Log Anomaly Detection

Domain specific analysis task drawn from real physical security operations.
The agent is given a CSV access control event log covering two physically
separate facilities and must identify three classes of security anomaly:
impossible travel (same badge at two buildings within 15 minutes), after-hours
access to restricted server rooms, and repeated denial bursts exceeding a
rolling 10 minute threshold.

The fixture includes deliberate near misses to test rule precision. Badge 6601
has exactly 3 denials (below the threshold of 4) and badge 4105 makes a
legitimate after hours visit to a non restricted door. Models that flag these
are penalised.

Grading is fully automated with no LLM judge dependency.

Baseline Results (5 runs each)

Model Score
anthropic/claude-opus-4-6 100%
anthropic/claude-sonnet-4-6 80%
openrouter/minimax/minimax-m2.5 80%
openrouter/minimax/minimax-m2.7 36%

The score spread confirms the task discriminates well across capability tiers
without being trivially easy or impossible. The 36% floor on M2.7 reflects
failure on the threshold edge case and rolling window logic rather than
complete task misunderstanding.

Files

  • tasks/task_24_access_log_anomaly.md task spec with inline fixture

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant