Skip to content

test: add RBAC end-to-end test suite under backend/tests/e2e/rbac/#189

Closed
yochze wants to merge 2 commits into0.0.355from
claude/rbac-e2e-tests-2KaAk
Closed

test: add RBAC end-to-end test suite under backend/tests/e2e/rbac/#189
yochze wants to merge 2 commits into0.0.355from
claude/rbac-e2e-tests-2KaAk

Conversation

@yochze
Copy link
Copy Markdown
Contributor

@yochze yochze commented Apr 8, 2026

Builds a comprehensive RBAC e2e suite that exercises real users, roles,
and DataSourceMembership grants against the live FastAPI app using the
existing test_client + create_user/login_user/whoami fixture style.

New fixtures (backend/tests/e2e/rbac/conftest.py):

  • invite_user_to_org: real invite+accept flow.
  • set_member_role: PUT /organizations/{id}/members/{mid}.
  • create_sqlite_ds: creates a private/public SQLite data source against
    the existing chinook fixture.
  • grant_ds_membership / revoke_ds_membership.
  • rbac_principals: admin + plain member + ds_a_member + ds_b_member +
    outsider (in a separate org), plus two private DSs and one public DS.
  • Autouse toggle for allow_uninvited_signups + allow_multiple_organizations
    so the outsider-in-separate-org scenario is actually possible.

New test files:

  • test_rbac_registry.py: AST-walks every @requires_permission and
    @requires_data_source_access literal and asserts it exists in the
    ROLES_PERMISSIONS source of truth. Also checks admin is a superset
    of member.
  • test_rbac_data_sources.py: matrix of list/detail/PUT/DSM operations
    • list/detail invariant + cross-org isolation.
  • test_rbac_instructions.py: private vs global creation, owner vs admin
    updates/deletes, cross-org rejection.
  • test_rbac_builds.py: view_builds (member+admin) vs create_builds
    (admin-only), list/detail invariant, cross-org build id check.
  • test_rbac_entities.py: list/detail invariant (no-ds variant),
    admin-only mutations, cross-org rejection. Includes an xfail that
    documents a pre-existing MissingGreenlet bug in
    EntityService.create_entity when data_source_ids is provided.
  • test_rbac_evals.py: every /api/tests/* endpoint is gated by
    manage_tests (admin only); member + outsider are 403.
  • test_rbac_role_principals.py: role transitions take effect
    immediately, DSM does not imply org admin, cross-org token +
    X-Organization-Id rejection, revoking org membership revokes
    all access.

Backend fixes surfaced by the registry/entity tests:

  1. routes/metadata_resource.py — two endpoints were decorated with
    @requires_permission('read_data_source'), but there has never been
    a permission by that name in MEMBER_PERMISSIONS / ADMIN_PERMISSIONS.
    Both endpoints would have 403'd for every caller, member and admin
    alike. Renamed to the canonical 'view_data_source' (member-accessible),
    matching every other /data_sources read endpoint.

  2. services/entity_service.py — routes/entity.py calls
    service.create_entity(..., force_global=...) on both POST /entities
    and POST /entities/global, but the service signature didn't accept
    force_global. Both endpoints 500'd with TypeError before this change.
    Added force_global=False to the signature, matching the same
    (currently no-op) parameter on InstructionService.create_instruction.

Known issue flagged as xfail:

  • EntityService.create_entity with data_source_ids hits
    sqlalchemy.exc.MissingGreenlet in the response serializer because
    the post-refresh data_sources relationship lazy-loads in the sync
    thread. Tracked in test_entity_create_with_data_source_ids_does_not_500.

Results under TESTING=true pytest -s -m e2e --db=sqlite backend/tests/e2e/rbac/:
31 passed, 1 xfailed

Regression check: tests/e2e/test_eval.py, test_membership.py,
test_entity.py, test_instruction.py, test_data_source.py all still pass.

claude added 2 commits April 8, 2026 08:47
Builds a comprehensive RBAC e2e suite that exercises real users, roles,
and DataSourceMembership grants against the live FastAPI app using the
existing test_client + create_user/login_user/whoami fixture style.

New fixtures (backend/tests/e2e/rbac/conftest.py):
- invite_user_to_org: real invite+accept flow.
- set_member_role: PUT /organizations/{id}/members/{mid}.
- create_sqlite_ds: creates a private/public SQLite data source against
  the existing chinook fixture.
- grant_ds_membership / revoke_ds_membership.
- rbac_principals: admin + plain member + ds_a_member + ds_b_member +
  outsider (in a separate org), plus two private DSs and one public DS.
- Autouse toggle for allow_uninvited_signups + allow_multiple_organizations
  so the outsider-in-separate-org scenario is actually possible.

New test files:
- test_rbac_registry.py: AST-walks every @requires_permission and
  @requires_data_source_access literal and asserts it exists in the
  ROLES_PERMISSIONS source of truth. Also checks admin is a superset
  of member.
- test_rbac_data_sources.py: matrix of list/detail/PUT/DSM operations
  + list/detail invariant + cross-org isolation.
- test_rbac_instructions.py: private vs global creation, owner vs admin
  updates/deletes, cross-org rejection.
- test_rbac_builds.py: view_builds (member+admin) vs create_builds
  (admin-only), list/detail invariant, cross-org build id check.
- test_rbac_entities.py: list/detail invariant (no-ds variant),
  admin-only mutations, cross-org rejection. Includes an xfail that
  documents a pre-existing MissingGreenlet bug in
  EntityService.create_entity when data_source_ids is provided.
- test_rbac_evals.py: every /api/tests/* endpoint is gated by
  manage_tests (admin only); member + outsider are 403.
- test_rbac_role_principals.py: role transitions take effect
  immediately, DSM does not imply org admin, cross-org token +
  X-Organization-Id rejection, revoking org membership revokes
  all access.

Backend fixes surfaced by the registry/entity tests:

1. routes/metadata_resource.py — two endpoints were decorated with
   @requires_permission('read_data_source'), but there has never been
   a permission by that name in MEMBER_PERMISSIONS / ADMIN_PERMISSIONS.
   Both endpoints would have 403'd for every caller, member and admin
   alike. Renamed to the canonical 'view_data_source' (member-accessible),
   matching every other /data_sources read endpoint.

2. services/entity_service.py — routes/entity.py calls
   service.create_entity(..., force_global=...) on both POST /entities
   and POST /entities/global, but the service signature didn't accept
   force_global. Both endpoints 500'd with TypeError before this change.
   Added force_global=False to the signature, matching the same
   (currently no-op) parameter on InstructionService.create_instruction.

Known issue flagged as xfail:
- EntityService.create_entity with data_source_ids hits
  sqlalchemy.exc.MissingGreenlet in the response serializer because
  the post-refresh data_sources relationship lazy-loads in the sync
  thread. Tracked in test_entity_create_with_data_source_ids_does_not_500.

Results under TESTING=true pytest -s -m e2e --db=sqlite backend/tests/e2e/rbac/:
  31 passed, 1 xfailed

Regression check: tests/e2e/test_eval.py, test_membership.py,
test_entity.py, test_instruction.py, test_data_source.py all still pass.
The repo already ignores backend/venv/ and backend/backend_venv/ but
the docs/design/sandbox-feedback-loop.md flow creates the venv at
backend/.venv/, which was slipping through. Ignoring it so it never
gets accidentally committed.
@yochze yochze changed the base branch from main to 0.0.355 April 10, 2026 14:17
@yochze yochze closed this Apr 10, 2026
@yochze yochze deleted the claude/rbac-e2e-tests-2KaAk branch April 11, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants