Skip to content

fix(tests): Fix flaky AI conversations tests exceeding Snuba retention#112612

Merged
mchen-sentry merged 2 commits intomasterfrom
fix/flaky-ai-conversations-retention
Apr 9, 2026
Merged

fix(tests): Fix flaky AI conversations tests exceeding Snuba retention#112612
mchen-sentry merged 2 commits intomasterfrom
fix/flaky-ai-conversations-retention

Conversation

@mchen-sentry
Copy link
Copy Markdown
Member

@mchen-sentry mchen-sentry commented Apr 9, 2026

Screenshot 2026-04-09 at 12 44 50 PM https://github.com/getsentry/sentry/actions/runs/24205379487/job/70659792593

This flake has been showing frequently and being covered by retries.

The test stores spans into Snuba, then immediately queries them. Sometimes the query returns nothing: assert len(response.data) == 1 fails with 0.

span_to_trace_item() hardcodes retention_days=90 for all test spans. But these tests use before_now(days=107) and similar, so the span timestamps are already past their own retention period at insertion time.

Original tests, days=10..80: ec760c4d8f4
Following tests added days 90..109:

Most likely the reason this flakes is that ClickHouse doesn't reject expired data on insert, and relies on a background process to clean up rows past their TTL. Assuming that the test is able to pass when the query runs before that process is able to clean these rows.

https://clickhouse.com/blog/using-ttl-to-manage-data-lifecycles-in-clickhouse
Screenshot 2026-04-09 at 1 01 53 PM

Fixed all 13 tests in this file that had days values >= 90 to within the retention window. Also fixed a duplicate days=30 between two tests.

…n window

Tests used `before_now(days=90..109)` but `retention_days=90` is hardcoded
in `span_to_trace_item()`. ClickHouse TTL merge can non-deterministically
drop already-expired data, causing intermittent empty results.

Move all timestamps to within the 90-day retention window and fix a
duplicate `days=30` value.
Copy link
Copy Markdown
Contributor

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 206953d. Configure here.

Comment thread tests/sentry/api/endpoints/test_organization_ai_conversations.py Outdated
@mchen-sentry mchen-sentry requested a review from a team April 9, 2026 20:10
@mchen-sentry mchen-sentry enabled auto-merge (squash) April 9, 2026 20:11
@mchen-sentry mchen-sentry merged commit b440f9b into master Apr 9, 2026
51 checks passed
@mchen-sentry mchen-sentry deleted the fix/flaky-ai-conversations-retention branch April 9, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants