feat: scaffold Agentic platform — orchestration, persistence, auth, ML pipeline

[Copilot]

Production-ready scaffold for autonomous agent platform with human oversight capabilities, reliable task orchestration with ack/retry/escalation, and RBAC-protected APIs.

Backend (FastAPI)

• Agent Systems: Communication (Redis pub/sub ack protocol), Decision, Delegation, Learning modules
• APIs: Auth (JWT/OIDC), Decisions (task CRUD, override, escalate), Admin (agents, models, training jobs), WebSocket oversight stream
• Database: SQLAlchemy 2.0 async with Alembic migrations for events, tasks, agents, audits, feedback, models, training_jobs tables
• Orchestration: Configurable ack timeout, exponential backoff retries, automatic escalation to human review queue

Frontend (React + Vite)

• Oversight dashboard with real-time WebSocket updates
• OIDC login flow with secure token handling
• RBAC-aware UI (override/escalate buttons visible only to permitted roles)

Infrastructure

• docker-compose.yml: Postgres, Redis, backend, frontend
• .github/workflows/ci.yml: Test + build pipeline with proper permissions

Task State Machine

queued → assigned → acked → in_progress → completed → verified
                 ↓              ↓
              (timeout)     (failure)
                 ↓              ↓
             retry (N) ────→ escalated

RBAC Roles

Role	Permissions
admin	Full access
reviewer	Override decisions, view escalations
agent_manager	Manage agents, create tasks
viewer	Read-only

Quick Start

docker-compose up --build
# Frontend: http://localhost:3000
# API: http://localhost:8000/docs

Configure OIDC via OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET env vars. Default dev mode accepts any credentials with role prefix (admin_*, reviewer_*, manager_*).

Original prompt

Add a production-ready scaffold and implementation to bootstrap the Agentic Agent platform in repository Stacey77/RAG7. The PR should add a modular FastAPI backend, a minimal React oversight UI (Vite), Docker Compose to run Postgres + Redis + backend + frontend, CI workflow, and implement the following expanded features: acknowledgement (ack) flow with timeouts and retries, task escalation, a persistent event store in Postgres with Alembic migrations, OIDC-based authentication and RBAC in backend and UI, and a basic ML pipeline integration (training job runner + model registry hooks).

Goals/Deliverables

1. Project scaffold (files and layout)

• backend/ (FastAPI app)
  • app/main.py, app/core.py, app/agents/{communication.py,decision.py,delegation.py,learning.py}
  • app/api/{decisions.py,oversight_ws.py,auth.py,admin.py}
  • app/db/{base.py,models.py,crud.py,session.py}
  • alembic/ (env + versions) and migrations to create events, tasks, agents, audits tables
  • Dockerfile, requirements.txt, and tests stubs
• frontend/ (Vite + React oversight UI)
  • src/components/OversightDashboard.jsx, auth flows, and RBAC-aware UI pieces
  • package.json, Dockerfile for production build
• docker-compose.yml to wire postgres, redis, backend, frontend
• .github/workflows/ci.yml to run tests and build images

2. Reliable orchestration & ack flow

• Implement an ack protocol over Redis pub/sub and Redis streams or ephemeral channels so the Agentic Core can publish tasks, wait for acknowledgement within configurable timeout, retry N times, and escalate if no ack or repeated failures.
• Task states: queued -> assigned -> acked -> in_progress -> completed -> verified | failed -> escalated
• Implement exponential backoff for retries and an escalation policy that moves tasks to a human review queue or higher-priority agents after threshold failures.

3. Persistent event store (Postgres + Alembic)

• Add DB models for events, tasks, agents, audits, feedback, and models table for model registry metadata.
• Create Alembic env and initial migration creating these tables.
• Implement a simple event writer API in backend (crud) that persists every task state change and decision event.

4. Authentication (OIDC) and RBAC

• Add OIDC authentication using python-social-auth / Authlib or FastAPI OIDC integration with configuration via env variables (OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET).
• Protect REST APIs and WS endpoints; add middleware to validate JWTs and extract user claims.
• Implement RBAC roles (admin, reviewer, agent_manager, viewer) and a decorator/Depends that enforces role checks in endpoints.
• Frontend: add OIDC login flow (redirect to provider), token storage (secure, in-memory + refresh), and RBAC-aware UI elements (show override/escalation buttons only to roles allowed).

5. ML pipeline & model registry integration

• Add a lightweight training job runner scaffold (Celery or a managed job runner) that can accept training tasks (pointing at persisted events/feedback), run a training step (placeholder script), and record model metadata into the model registry table in Postgres.
• Provide hooks in AgentLearningSystem to enqueue training jobs and to fetch latest model metadata for agent inference.

Acceptance Criteria

• The repo contains new scaffold files and Alembic migration(s) that create required tables.
• docker-compose up --build starts Postgres, Redis, backend, and frontend; backend connects to DB and Redis and runs migrations on startup (or instructions included).
• Posting a task via /api/decisions/task results in a persisted task event, published to Redis, and an assigned attempt with ack/retry behavior exercised by test stubs.
• WebSocket oversight streams decision/state events only to authenticated users and enforces RBAC for actions like override/escalate.
• The ML pipeline can be triggered from learning.collect_feedback and writes model metadata to DB (placeholder training is acceptable for initial PR).

Implementation notes (detailed)

• Use SQLAlchemy Async (1.4) + asyncpg; Alembic async migration template and initial migration file.
• Use aioredis for Redis connections, and implement an ack mechanism using a dedicated ack channel and waiting with asyncio.wait_for; include retry_count tracking and backoff.
• For OIDC, implement token validation using jose (python-jose) or Authlib, with JWKS fetching from OIDC_ISSUER. Add a dependency get_current_user and get_current_active_user that enforces roles from token claims or a local user table.
• Add tests for ack flow and escalation using pytest-asyncio and a local Redis test instance (or mocked aioredis).
• CI should run unit tests for backend and lint; building containers is optional but keep the step.

Files to add/modify (explicit examples)

• Provide the core scaffold code from the previous message (FastAPI app, agents modules, frontend) with additional modules: app/api/auth.py, app/db/models.py, app/db/session.py, alembic/env.py + versions/0001_initial.py, ...

This pull request was created as a result of the following prompt from Copilot chat.

Add a production-ready scaffold and implementation to bootstrap the Agentic Agent platform in repository Stacey77/RAG7. The PR should add a modular FastAPI backend, a minimal React oversight UI (Vite), Docker Compose to run Postgres + Redis + backend + frontend, CI workflow, and implement the following expanded features: acknowledgement (ack) flow with timeouts and retries, task escalation, a persistent event store in Postgres with Alembic migrations, OIDC-based authentication and RBAC in backend and UI, and a basic ML pipeline integration (training job runner + model registry hooks).

Goals/Deliverables

1. Project scaffold (files and layout)

• backend/ (FastAPI app)
  • app/main.py, app/core.py, app/agents/{communication.py,decision.py,delegation.py,learning.py}
  • app/api/{decisions.py,oversight_ws.py,auth.py,admin.py}
  • app/db/{base.py,models.py,crud.py,session.py}
  • alembic/ (env + versions) and migrations to create events, tasks, agents, audits tables
  • Dockerfile, requirements.txt, and tests stubs
• frontend/ (Vite + React oversight UI)
  • src/components/OversightDashboard.jsx, auth flows, and RBAC-aware UI pieces
  • package.json, Dockerfile for production build
• docker-compose.yml to wire postgres, redis, backend, frontend
• .github/workflows/ci.yml to run tests and build images

2. Reliable orchestration & ack flow

• Implement an ack protocol over Redis pub/sub and Redis streams or ephemeral channels so the Agentic Core can publish tasks, wait for acknowledgement within configurable timeout, retry N times, and escalate if no ack or repeated failures.
• Task states: queued -> assigned -> acked -> in_progress -> completed -> verified | failed -> escalated
• Implement exponential backoff for retries and an escalation policy that moves tasks to a human review queue or higher-priority agents after threshold failures.

3. Persistent event store (Postgres + Alembic)

• Add DB models for events, tasks, agents, audits, feedback, and models table for model registry metadata.
• Create Alembic env and initial migration creating these tables.
• Implement a simple event writer API in backend (crud) that persists every task state change and decision event.

4. Authentication (OIDC) and RBAC

• Add OIDC authentication using python-social-auth / Authlib or FastAPI OIDC integration with configuration via env variables (OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET).
• Protect REST APIs and WS endpoints; add middleware to validate JWTs and extract user claims.
• Implement RBAC roles (admin, reviewer, agent_manager, viewer) and a decorator/Depends that enforces role checks in endpoints.
• Frontend: add OIDC login flow (redirect to provider), token storage (secure, in-memory + refresh), and RBAC-aware UI elements (show override/escalation buttons only to roles allowed).

5. ML pipeline & model registry integration

• Add a lightweight training job runner scaffold (Celery or a managed job runner) that can accept training tasks (pointing at persisted events/feedback), run a training step (placeholder script), and record model metadata into the model registry table in Postgres.
• Provide hooks in AgentLearningSystem to enqueue training jobs and to fetch latest model metadata for agent inference.

Acceptance Criteria

• The repo contains new scaffold files and Alembic migration(s) that create required tables.
• docker-compose up --build starts Postgres, Redis, backend, and frontend; backend connects to DB and Redis and runs migrations on startup (or instructions included).
• Posting a task via /api/decisions/task results in a persisted task event, published to Redis, and an assigned attempt with ack/retry behavior exercised by test stubs.
• WebSocket oversight streams decision/state events only to authenticated users and enforces RBAC for actions like override/escalate.
• The ML pipeline can be triggered from learning.collect_feedback and writes model metadata to DB (placeholder training is acceptable for initial PR).

Implementation notes (detailed)

• Use SQLAlchemy Async (1.4) + asyncpg; Alembic async migration template and initial migration file.
• Use aioredis for Redis connections, and implement an ack mechanism using a dedicated ack channel and waiting with asyncio.wait_for; include retry_count tracking and backoff.
• For OIDC, implement token validation using jose (python-jose) or Authlib, with JWKS fetching from OIDC_ISSUER. Add a dependency get_current_user and get_current_active_user that enforces roles from token claims or a local user table.
• Add tests for ack flow and escalation using pytest-asyncio and a local Redis test instance (or mocked aioredis).
• CI should run unit tests for backend and lint; building containers is optional but keep the step.

Files to add/modify (explicit examples)

• Provide the core scaffold code from the previous message (FastAPI app, agents modules, frontend) with additional modules: app/api/auth.py, app/db/models.py, app/db/session.py, alembic/env.py + versions/0001_initial.py, backend Dockerfile updates, frontend OIDC hooks.

PR details

• Branch name: agentic/scaffold-full (create branch)
• Title: "feat: scaffold Agentic platform — orchestration, persistence, auth, ML pipeline"
• Description: Include the Goals/Deliverables and Acceptance Criteria above; explain how to run (docker-compose up), notes about env vars for OIDC and DB, and suggest next steps (harden auth, telemetry, k8s manifests).

No assignees. Please create the PR with these changes and include the set of scaffold files and migrations. After PR is opened, return the PR URL and a checklist of what was added and what remains, plus instructions for local testing.

Repository: Stacey77/RAG7
Branch: agentic/scaffold-full

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.