Skip to content

chore: clarify how metric config changes will apply to existing experiments #782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mikewilli merged 1 commit into from
Feb 11, 2026
Merged

chore: clarify how metric config changes will apply to existing experiments #782

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/data-analysis/jetstream/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,12 +114,17 @@ It prevents sudden changes of how results are computed after tooling (such as [m
### How to Use the Latest Tooling and Metric Definitions?

When tooling or metrics are updated, there are a few options with different consequences for how Jetstream treats existing experiments:
1. **do nothing**: only experiments that are launched after the new tooling release will use the most recent version of the tooling
1. **do nothing**: only experiments that are launched after the new tooling release will use the most recent version of the tooling
- NOTE: this does not apply to metric changes, which automatically get rerun or rerun-skip depending on how changes are landed.
1. **use new tooling version without rerunning**: install the [jetstream command line tooling](https://pypi.org/project/mozilla-jetstream/) locally and run `jetstream rerun-skip --experiment_slug=<slug>`. This command pretends to rerun the experiment without actually re-running the queries, which will update the date used to determine which tooling and metric versions to use. Rerunning an experiment will always force new versions to be used for experiments.
- NOTE: this option will apply for metric changes if the metric-hub PR is landed **with** `[ci rerun-skip]` in the title.
1. **rerun experiment**: re-running an experiment will always use the most recent version of the tooling on the rerun and update the last updated timestamps of the result tables
- NOTE: this option will apply for metric changes if the metric-hub PR is landed **without** `[ci rerun-skip]` in the title.

More detailled information around versioning is available in [Jetstream Operations].

Also see the [metric-hub README] for more information on how and when to use `[ci rerun-skip]` when landing configuration changes.

## Datasets

The datasets that back the Experimenter results dashboards
Expand All @@ -135,3 +140,4 @@ is available in the Mozilla data docs.
[outcome]: ./outcomes
[mozanalysis]: https://github.com/mozilla/mozanalysis
[Jetstream Operations]: ./operations#tooling-and-metric-versioning
[metric-hub README]: https://github.com/mozilla/metric-hub?tab=readme-ov-file#experiment-analysis-considerations