test: add RBAC end-to-end test suite under backend/tests/e2e/rbac/#189
Closed
test: add RBAC end-to-end test suite under backend/tests/e2e/rbac/#189
Conversation
Builds a comprehensive RBAC e2e suite that exercises real users, roles,
and DataSourceMembership grants against the live FastAPI app using the
existing test_client + create_user/login_user/whoami fixture style.
New fixtures (backend/tests/e2e/rbac/conftest.py):
- invite_user_to_org: real invite+accept flow.
- set_member_role: PUT /organizations/{id}/members/{mid}.
- create_sqlite_ds: creates a private/public SQLite data source against
the existing chinook fixture.
- grant_ds_membership / revoke_ds_membership.
- rbac_principals: admin + plain member + ds_a_member + ds_b_member +
outsider (in a separate org), plus two private DSs and one public DS.
- Autouse toggle for allow_uninvited_signups + allow_multiple_organizations
so the outsider-in-separate-org scenario is actually possible.
New test files:
- test_rbac_registry.py: AST-walks every @requires_permission and
@requires_data_source_access literal and asserts it exists in the
ROLES_PERMISSIONS source of truth. Also checks admin is a superset
of member.
- test_rbac_data_sources.py: matrix of list/detail/PUT/DSM operations
+ list/detail invariant + cross-org isolation.
- test_rbac_instructions.py: private vs global creation, owner vs admin
updates/deletes, cross-org rejection.
- test_rbac_builds.py: view_builds (member+admin) vs create_builds
(admin-only), list/detail invariant, cross-org build id check.
- test_rbac_entities.py: list/detail invariant (no-ds variant),
admin-only mutations, cross-org rejection. Includes an xfail that
documents a pre-existing MissingGreenlet bug in
EntityService.create_entity when data_source_ids is provided.
- test_rbac_evals.py: every /api/tests/* endpoint is gated by
manage_tests (admin only); member + outsider are 403.
- test_rbac_role_principals.py: role transitions take effect
immediately, DSM does not imply org admin, cross-org token +
X-Organization-Id rejection, revoking org membership revokes
all access.
Backend fixes surfaced by the registry/entity tests:
1. routes/metadata_resource.py — two endpoints were decorated with
@requires_permission('read_data_source'), but there has never been
a permission by that name in MEMBER_PERMISSIONS / ADMIN_PERMISSIONS.
Both endpoints would have 403'd for every caller, member and admin
alike. Renamed to the canonical 'view_data_source' (member-accessible),
matching every other /data_sources read endpoint.
2. services/entity_service.py — routes/entity.py calls
service.create_entity(..., force_global=...) on both POST /entities
and POST /entities/global, but the service signature didn't accept
force_global. Both endpoints 500'd with TypeError before this change.
Added force_global=False to the signature, matching the same
(currently no-op) parameter on InstructionService.create_instruction.
Known issue flagged as xfail:
- EntityService.create_entity with data_source_ids hits
sqlalchemy.exc.MissingGreenlet in the response serializer because
the post-refresh data_sources relationship lazy-loads in the sync
thread. Tracked in test_entity_create_with_data_source_ids_does_not_500.
Results under TESTING=true pytest -s -m e2e --db=sqlite backend/tests/e2e/rbac/:
31 passed, 1 xfailed
Regression check: tests/e2e/test_eval.py, test_membership.py,
test_entity.py, test_instruction.py, test_data_source.py all still pass.
The repo already ignores backend/venv/ and backend/backend_venv/ but the docs/design/sandbox-feedback-loop.md flow creates the venv at backend/.venv/, which was slipping through. Ignoring it so it never gets accidentally committed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds a comprehensive RBAC e2e suite that exercises real users, roles,
and DataSourceMembership grants against the live FastAPI app using the
existing test_client + create_user/login_user/whoami fixture style.
New fixtures (backend/tests/e2e/rbac/conftest.py):
the existing chinook fixture.
outsider (in a separate org), plus two private DSs and one public DS.
so the outsider-in-separate-org scenario is actually possible.
New test files:
@requires_data_source_access literal and asserts it exists in the
ROLES_PERMISSIONS source of truth. Also checks admin is a superset
of member.
updates/deletes, cross-org rejection.
(admin-only), list/detail invariant, cross-org build id check.
admin-only mutations, cross-org rejection. Includes an xfail that
documents a pre-existing MissingGreenlet bug in
EntityService.create_entity when data_source_ids is provided.
manage_tests (admin only); member + outsider are 403.
immediately, DSM does not imply org admin, cross-org token +
X-Organization-Id rejection, revoking org membership revokes
all access.
Backend fixes surfaced by the registry/entity tests:
routes/metadata_resource.py — two endpoints were decorated with
@requires_permission('read_data_source'), but there has never been
a permission by that name in MEMBER_PERMISSIONS / ADMIN_PERMISSIONS.
Both endpoints would have 403'd for every caller, member and admin
alike. Renamed to the canonical 'view_data_source' (member-accessible),
matching every other /data_sources read endpoint.
services/entity_service.py — routes/entity.py calls
service.create_entity(..., force_global=...) on both POST /entities
and POST /entities/global, but the service signature didn't accept
force_global. Both endpoints 500'd with TypeError before this change.
Added force_global=False to the signature, matching the same
(currently no-op) parameter on InstructionService.create_instruction.
Known issue flagged as xfail:
sqlalchemy.exc.MissingGreenlet in the response serializer because
the post-refresh data_sources relationship lazy-loads in the sync
thread. Tracked in test_entity_create_with_data_source_ids_does_not_500.
Results under TESTING=true pytest -s -m e2e --db=sqlite backend/tests/e2e/rbac/:
31 passed, 1 xfailed
Regression check: tests/e2e/test_eval.py, test_membership.py,
test_entity.py, test_instruction.py, test_data_source.py all still pass.