Skip to content

fix(seer): Make Seer repo preferences cleanup synchronous to prevent stale dual-writes#112949

Draft
srest2021 wants to merge 4 commits intomasterfrom
srest2021/fix-disabled-repo
Draft

fix(seer): Make Seer repo preferences cleanup synchronous to prevent stale dual-writes#112949
srest2021 wants to merge 4 commits intomasterfrom
srest2021/fix-disabled-repo

Conversation

@srest2021
Copy link
Copy Markdown
Member

@srest2021 srest2021 commented Apr 14, 2026

Extract cleanup logic from Celery tasks into synchronous helpers in autofix/utils.py and call them directly from impl.py and organization_repository_details.py. This closes the race window where async Seer cleanup allowed stale preferences (with disabled/hidden repos) to be read back from Seer and dual-written into SeerProjectRepository after they had already been deleted.

srest2021 and others added 2 commits April 14, 2026 10:23
…dual-writes

Extract cleanup logic from Celery tasks into synchronous helpers in
autofix/utils.py and call them directly from impl.py and
organization_repository_details.py. This closes the race window where
async Seer cleanup allowed stale preferences (with disabled/hidden
repos) to be read back from Seer and dual-written into
SeerProjectRepository before the cleanup task ran.

The tasks are kept as thin wrappers (renamed with _task suffix) for
any future async callers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@srest2021 srest2021 changed the title fix(seer): Make Seer repository cleanup synchronous to prevent stale dual-writes fix(seer): Make Seer repo preferences cleanup synchronous to prevent stale dual-writes Apr 14, 2026
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Apr 14, 2026
@@ -100,12 +100,13 @@ def put(self, request: Request, organization: Organization, repo_id) -> Response
repository_cascade_delete_on_hide.apply_async(kwargs={"repo_id": repo.id})
Copy link
Copy Markdown
Member Author

@srest2021 srest2021 Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this thing which deletes associated SeerProjectRepository is still asynchronous.

If read flag is off: we read and write existing prefs from Seer which are now updated synchronously. ✅

If read flag is on: we read and write existing prefs from Sentry which may be stale, just like all the other Repository child relations. However they will be eventually deleted from Sentry via the above task, and from Seer via its own cleanup cron. ✅

So I think we are ok here.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 14, 2026

Backend Test Failures

Failures on 0c9142f in this run:

tests/sentry/integrations/github/tasks/test_sync_repos_on_install_change.py::SyncReposOnInstallChangeTestCase::test_repos_removedlog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/tasks/base.py:206: in wrapped
    return func(*args, **kwargs)
src/sentry/integrations/github/tasks/sync_repos_on_install_change.py:91: in sync_repos_on_install_change
    _sync_repos_for_org(
src/sentry/integrations/github/tasks/sync_repos_on_install_change.py:157: in _sync_repos_for_org
    repository_service.disable_repositories_by_external_ids(
src/sentry/hybridcloud/rpc/service.py:355: in remote_method
    return dispatch_remote_call(
src/sentry/hybridcloud/rpc/service.py:493: in dispatch_remote_call
    return remote_silo_call.dispatch(use_test_client)
src/sentry/hybridcloud/rpc/service.py:528: in dispatch
    serial_response = self._send_to_remote_silo(use_test_client)
src/sentry/hybridcloud/rpc/service.py:610: in _send_to_remote_silo
    self._raise_from_response_status_error(response)
src/sentry/hybridcloud/rpc/service.py:644: in _raise_from_response_status_error
    raise self._remote_exception(
E   sentry.hybridcloud.rpc.service.RpcRemoteException: repository.disable_repositories_by_external_ids: Error invoking rpc at '/api/0/internal/rpc/repository/disable_repositories_by_external_ids/': check error logs for more details

During handling of the above exception, another exception occurred:
tests/sentry/integrations/github/tasks/test_sync_repos_on_install_change.py:74: in test_repos_removed
    sync_repos_on_install_change(
src/sentry/silo/base.py:157: in override
    return original_method(*args, **kwargs)
.venv/lib/python3.13/site-packages/taskbroker_client/task.py:92: in __call__
    return self._func(*args, **kwargs)
src/sentry/tasks/base.py:238: in wrapped
    retry_task(exc, raise_on_no_retries=raise_on_no_retries)
.venv/lib/python3.13/site-packages/taskbroker_client/retry.py:58: in retry_task
    raise RetryTaskError()
E   taskbroker_client.retry.RetryTaskError
tests/sentry/integrations/source_code_management/test_sync_repos.py::SyncReposForOrgTestCase::test_disables_removed_reposlog
[gw1] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/tasks/base.py:206: in wrapped
    return func(*args, **kwargs)
src/sentry/integrations/source_code_management/sync_repos.py:251: in sync_repos_for_org
    repository_service.disable_repositories_by_external_ids(
src/sentry/hybridcloud/rpc/service.py:355: in remote_method
    return dispatch_remote_call(
src/sentry/hybridcloud/rpc/service.py:493: in dispatch_remote_call
    return remote_silo_call.dispatch(use_test_client)
src/sentry/hybridcloud/rpc/service.py:528: in dispatch
    serial_response = self._send_to_remote_silo(use_test_client)
src/sentry/hybridcloud/rpc/service.py:610: in _send_to_remote_silo
    self._raise_from_response_status_error(response)
src/sentry/hybridcloud/rpc/service.py:644: in _raise_from_response_status_error
    raise self._remote_exception(
E   sentry.hybridcloud.rpc.service.RpcRemoteException: repository.disable_repositories_by_external_ids: Error invoking rpc at '/api/0/internal/rpc/repository/disable_repositories_by_external_ids/': check error logs for more details

During handling of the above exception, another exception occurred:
tests/sentry/integrations/source_code_management/test_sync_repos.py:90: in test_disables_removed_repos
    sync_repos_for_org(self.oi.id)
src/sentry/silo/base.py:157: in override
    return original_method(*args, **kwargs)
.venv/lib/python3.13/site-packages/taskbroker_client/task.py:92: in __call__
    return self._func(*args, **kwargs)
src/sentry/tasks/base.py:238: in wrapped
    retry_task(exc, raise_on_no_retries=raise_on_no_retries)
.venv/lib/python3.13/site-packages/taskbroker_client/retry.py:58: in retry_task
    raise RetryTaskError()
E   taskbroker_client.retry.RetryTaskError
tests/sentry/integrations/github/tasks/test_sync_repos_on_install_change.py::SyncReposOnInstallChangeTestCase::test_mixed_add_and_removelog
[gw1] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/tasks/base.py:206: in wrapped
    return func(*args, **kwargs)
src/sentry/integrations/github/tasks/sync_repos_on_install_change.py:91: in sync_repos_on_install_change
    _sync_repos_for_org(
src/sentry/integrations/github/tasks/sync_repos_on_install_change.py:157: in _sync_repos_for_org
    repository_service.disable_repositories_by_external_ids(
src/sentry/hybridcloud/rpc/service.py:355: in remote_method
    return dispatch_remote_call(
src/sentry/hybridcloud/rpc/service.py:493: in dispatch_remote_call
    return remote_silo_call.dispatch(use_test_client)
src/sentry/hybridcloud/rpc/service.py:528: in dispatch
    serial_response = self._send_to_remote_silo(use_test_client)
src/sentry/hybridcloud/rpc/service.py:610: in _send_to_remote_silo
    self._raise_from_response_status_error(response)
src/sentry/hybridcloud/rpc/service.py:644: in _raise_from_response_status_error
    raise self._remote_exception(
E   sentry.hybridcloud.rpc.service.RpcRemoteException: repository.disable_repositories_by_external_ids: Error invoking rpc at '/api/0/internal/rpc/repository/disable_repositories_by_external_ids/': check error logs for more details

During handling of the above exception, another exception occurred:
tests/sentry/integrations/github/tasks/test_sync_repos_on_install_change.py:104: in test_mixed_add_and_remove
    sync_repos_on_install_change(
src/sentry/silo/base.py:157: in override
    return original_method(*args, **kwargs)
.venv/lib/python3.13/site-packages/taskbroker_client/task.py:92: in __call__
    return self._func(*args, **kwargs)
src/sentry/tasks/base.py:238: in wrapped
    retry_task(exc, raise_on_no_retries=raise_on_no_retries)
.venv/lib/python3.13/site-packages/taskbroker_client/retry.py:58: in retry_task
    raise RetryTaskError()
E   taskbroker_client.retry.RetryTaskError

lambda: cleanup_seer_repository_preferences(
organization_id=repo.organization_id,
repo_external_id=repo.external_id, # type: ignore[arg-type]
repo_provider=repo.provider, # type: ignore[arg-type]
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mypy keeps complaining even though these can't be none

@github-actions
Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on e52d767 in this run:

tests/sentry/taskworker/test_config.py::test_import_pathslog
[gw1] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/taskworker/test_config.py:25: in test_import_paths
    __import__(path)
E   ModuleNotFoundError: No module named 'sentry.tasks.seer.cleanup'

During handling of the above exception, another exception occurred:
tests/sentry/taskworker/test_config.py:27: in test_import_paths
    raise AssertionError(f"Unable to import {path} from TASKWORKER_IMPORTS")
E   AssertionError: Unable to import sentry.tasks.seer.cleanup from TASKWORKER_IMPORTS

@srest2021 srest2021 added the Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling label Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant