Skip to content

Conversation

@mike-ferguson
Copy link
Member

PR: Excluded Benchmarks Feature

Summary

Adds the ability to configure benchmarks that are unchecked by default in the leaderboard. These benchmarks remain visible but are excluded from the default score calculation without triggering the blue "modified" column styling.

Changes

  • benchmarks/benchmark_config.py (new): Configuration file where admins can add benchmark identifiers to exclude
  • benchmarks/views/leaderboard.py: Passes excluded benchmarks list to frontend
  • static/benchmarks/js/leaderboard/core/template-initialization.js: Parses excluded list and expands to include all descendants of excluded parent benchmarks
  • static/benchmarks/js/leaderboard/filters/benchmark-filters.js:
    • Unchecks excluded benchmarks on load
    • Adds data-excluded attribute to prevent blue styling
    • Shows info icon (ⓘ) with tooltip on excluded benchmarks
  • static/benchmarks/js/leaderboard/filters/filter-coordinator.js: Skips excluded benchmarks when determining blue column styling

Usage

Add benchmark identifiers to EXCLUDED_BENCHMARKS in benchmark_config.py:

EXCLUDED_BENCHMARKS = [
    "tong.Coggan2024_fMRI.V1-rdm_v1",
    # ... more benchmarks
]

Behavior

  • Excluded benchmarks appear unchecked by default
  • Users can still check them to include in scoring
  • No blue "filtered" column styling when excluded
  • Info tooltip explains the exclusion to users

@mike-ferguson mike-ferguson requested a review from KartikP December 3, 2025 18:49
@KartikP
Copy link
Contributor

KartikP commented Dec 4, 2025

Tests failing likely as a result of the provided exclusion list changing expected outputs.

@KartikP
Copy link
Contributor

KartikP commented Dec 8, 2025

For some reason, any benchmark that is in the excluded benchmarks list when modified does not change the Global Score

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants