diff --git a/.claude/commands/create-spec.md b/.claude/commands/create-spec.md index f8cae28e..f8a1b96f 100644 --- a/.claude/commands/create-spec.md +++ b/.claude/commands/create-spec.md @@ -95,6 +95,27 @@ Ask the user about their involvement preference: **For Detailed Mode users**, ask specific tech questions about frontend, backend, database, etc. +### Phase 3b: Database Requirements (MANDATORY) + +**Always ask this question regardless of mode:** + +> "One foundational question about data storage: +> +> **Does this application need to store user data persistently?** +> +> 1. **Yes, needs a database** - Users create, save, and retrieve data (most apps) +> 2. **No, stateless** - Pure frontend, no data storage needed (calculators, static sites) +> 3. **Not sure** - Let me describe what I need and you decide" + +**Branching logic:** + +- **If "Yes" or "Not sure"**: Continue normally. The spec will include database in tech stack and the initializer will create 5 mandatory Infrastructure features (indices 0-4) to verify database connectivity and persistence. + +- **If "No, stateless"**: Note this in the spec. Skip database from tech stack. Infrastructure features will be simplified (no database persistence tests). Mark this clearly: + ```xml + none - stateless application + ``` + ## Phase 4: Features (THE MAIN PHASE) This is where you spend most of your time. Ask questions in plain language that anyone can answer. @@ -207,12 +228,23 @@ After gathering all features, **you** (the agent) should tally up the testable f **Typical ranges for reference:** -- **Simple apps** (todo list, calculator, notes): ~20-50 features -- **Medium apps** (blog, task manager with auth): ~100 features -- **Advanced apps** (e-commerce, CRM, full SaaS): ~150-200 features +- **Simple apps** (todo list, calculator, notes): ~25-55 features (includes 5 infrastructure) +- **Medium apps** (blog, task manager with auth): ~105 features (includes 5 infrastructure) +- **Advanced apps** (e-commerce, CRM, full SaaS): ~155-205 features (includes 5 infrastructure) These are just reference points - your actual count should come from the requirements discussed. +**MANDATORY: Infrastructure Features** + +If the app requires a database (Phase 3b answer was "Yes" or "Not sure"), you MUST include 5 Infrastructure features (indices 0-4): +1. Database connection established +2. Database schema applied correctly +3. Data persists across server restart +4. No mock data patterns in codebase +5. Backend API queries real database + +These features ensure the coding agent implements a real database, not mock data or in-memory storage. + **How to count features:** For each feature area discussed, estimate the number of discrete, testable behaviors: @@ -225,17 +257,20 @@ For each feature area discussed, estimate the number of discrete, testable behav > "Based on what we discussed, here's my feature breakdown: > +> - **Infrastructure (required)**: 5 features (database setup, persistence verification) > - [Category 1]: ~X features > - [Category 2]: ~Y features > - [Category 3]: ~Z features > - ... > -> **Total: ~N features** +> **Total: ~N features** (including 5 infrastructure) > > Does this seem right, or should I adjust?" Let the user confirm or adjust. This becomes your `feature_count` for the spec. +**Important:** The first 5 features (indices 0-4) created by the initializer MUST be the Infrastructure category with no dependencies. All other features depend on these. + ## Phase 5: Technical Details (DERIVED OR DISCUSSED) **For Quick Mode users:** diff --git a/.claude/templates/coding_prompt.template.md b/.claude/templates/coding_prompt.template.md index bce9a142..2f4a62c6 100644 --- a/.claude/templates/coding_prompt.template.md +++ b/.claude/templates/coding_prompt.template.md @@ -156,6 +156,9 @@ Use browser automation tools: - [ ] Deleted the test data - verified it's gone everywhere - [ ] NO unexplained data appeared (would indicate mock data) - [ ] Dashboard/counts reflect real numbers after my changes +- [ ] **Ran extended mock data grep (STEP 5.6) - no hits in src/ (excluding tests)** +- [ ] **Verified no globalThis, devStore, or dev-store patterns** +- [ ] **Server restart test passed (STEP 5.7) - data persists across restart** #### Navigation Verification @@ -174,10 +177,90 @@ Use browser automation tools: ### STEP 5.6: MOCK DATA DETECTION (Before marking passing) -1. **Search code:** `grep -r "mockData\|fakeData\|TODO\|STUB" --include="*.ts" --include="*.tsx"` -2. **Runtime test:** Create unique data (e.g., "TEST_12345") → verify in UI → delete → verify gone -3. **Check database:** All displayed data must come from real DB queries -4. If unexplained data appears, it's mock data - fix before marking passing. +**Run ALL these grep checks. Any hits in src/ (excluding test files) require investigation:** + +```bash +# 1. In-memory storage patterns (CRITICAL - catches dev-store) +grep -r "globalThis\." --include="*.ts" --include="*.tsx" --include="*.js" src/ +grep -r "dev-store\|devStore\|DevStore\|mock-db\|mockDb" --include="*.ts" --include="*.tsx" --include="*.js" src/ + +# 2. Mock data variables +grep -r "mockData\|fakeData\|sampleData\|dummyData\|testData" --include="*.ts" --include="*.tsx" --include="*.js" src/ + +# 3. TODO/incomplete markers +grep -r "TODO.*real\|TODO.*database\|TODO.*API\|STUB\|MOCK" --include="*.ts" --include="*.tsx" --include="*.js" src/ + +# 4. Development-only conditionals +grep -r "isDevelopment\|isDev\|process\.env\.NODE_ENV.*development" --include="*.ts" --include="*.tsx" --include="*.js" src/ + +# 5. In-memory collections used as data stores (module-level or exported) +# Note: Map/Set are legitimate for local caching - investigate context before flagging +grep -rn "^const.*=.*new Map\(\)\|^let.*=.*new Map\(\)\|^export.*Map\(\)" --include="*.ts" --include="*.tsx" --include="*.js" src/ 2>/dev/null +``` + +**Rule:** If ANY grep returns results in production code → investigate → FIX before marking passing. + +**Runtime verification:** +1. Create unique data (e.g., "TEST_12345") → verify in UI → delete → verify gone +2. Check database directly - all displayed data must come from real DB queries +3. If unexplained data appears, it's mock data - fix before marking passing. + +### STEP 5.7: SERVER RESTART PERSISTENCE TEST (MANDATORY for data features) + +**When required:** Any feature involving CRUD operations or data persistence. + +**This test is NON-NEGOTIABLE. It catches in-memory storage implementations that pass all other tests.** + +**Steps:** + +1. Create unique test data via UI or API (e.g., item named "RESTART_TEST_12345") +2. Verify data appears in UI and API response + +3. **STOP the server completely:** + ```bash + # Kill by port (safer - only kills the dev server, not VS Code/Claude Code/etc.) + # Unix/macOS: + lsof -ti :${PORT:-3000} | xargs kill -TERM 2>/dev/null || true + sleep 3 + lsof -ti :${PORT:-3000} | xargs kill -9 2>/dev/null || true + sleep 2 + + # Windows alternative (use if lsof not available): + # netstat -ano | findstr :${PORT:-3000} | findstr LISTENING + # taskkill /F /PID 2>nul + + # Verify server is stopped + if lsof -ti :${PORT:-3000} > /dev/null 2>&1; then + echo "ERROR: Server still running on port ${PORT:-3000}!" + exit 1 + fi + ``` + +4. **RESTART the server:** + ```bash + ./init.sh & + sleep 15 # Allow server to fully start + # Verify server is responding + if ! curl -f http://localhost:${PORT:-3000}/api/health && ! curl -f http://localhost:${PORT:-3000}; then + echo "ERROR: Server failed to start after restart" + exit 1 + fi + ``` + +5. **Query for test data - it MUST still exist** + - Via UI: Navigate to data location, verify data appears + - Via API: `curl http://localhost:${PORT:-3000}/api/items` - verify data in response + +6. **If data is GONE:** Implementation uses in-memory storage → CRITICAL FAIL + - Run all grep commands from STEP 5.6 to identify the mock pattern + - You MUST fix the in-memory storage implementation before proceeding + - Replace in-memory storage with real database queries + +7. **Clean up test data** after successful verification + +**Why this test exists:** In-memory stores like `globalThis.devStore` pass all other tests because data persists during a single server run. Only a full server restart reveals this bug. Skipping this step WILL allow dev-store implementations to slip through. + +**YOLO Mode Note:** Even in YOLO mode, this verification is MANDATORY for data features. Use curl instead of browser automation. ### STEP 6: UPDATE FEATURE STATUS (CAREFULLY!) @@ -202,17 +285,24 @@ Use the feature_mark_passing tool with feature_id=42 ### STEP 7: COMMIT YOUR PROGRESS -Make a descriptive git commit: +Make a descriptive git commit. + +**Git Commit Rules:** +- ALWAYS use simple `-m` flag for commit messages +- NEVER use heredocs (`cat </dev/null || true && sleep 5 + - Windows: FOR /F "tokens=5" %a IN ('netstat -aon ^| find ":$PORT"') DO taskkill /F /PID %a 2>nul + - Note: Replace $PORT with actual port (e.g., 3000) +4. Verify server is stopped: lsof -ti :$PORT returns nothing (or netstat on Windows) +5. RESTART the server: ./init.sh & sleep 15 +6. Query API again: GET /api/items +7. Verify "RESTART_TEST_12345" still exists +8. If data is GONE → CRITICAL FAILURE (in-memory storage detected) +9. Clean up test data +``` + +**Feature 3 - No mock data patterns in codebase:** +```text +Steps: +1. Run: grep -r "globalThis\." --include="*.ts" --include="*.tsx" --include="*.js" src/ +2. Run: grep -r "dev-store\|devStore\|DevStore\|mock-db\|mockDb" --include="*.ts" --include="*.tsx" --include="*.js" src/ +3. Run: grep -r "mockData\|testData\|fakeData\|sampleData\|dummyData" --include="*.ts" --include="*.tsx" --include="*.js" src/ +4. Run: grep -r "TODO.*real\|TODO.*database\|TODO.*API\|STUB\|MOCK" --include="*.ts" --include="*.tsx" --include="*.js" src/ +5. Run: grep -r "isDevelopment\|isDev\|process\.env\.NODE_ENV.*development" --include="*.ts" --include="*.tsx" --include="*.js" src/ +6. Run: grep -r "new Map\(\)\|new Set\(\)" --include="*.ts" --include="*.tsx" --include="*.js" src/ 2>/dev/null +7. Run: grep -E "json-server|miragejs|msw" package.json +8. ALL grep commands must return empty (exit code 1) +9. If any returns results → investigate and fix before passing +``` + +**Feature 4 - Backend API queries real database:** +```text +Steps: +1. Start server with verbose logging +2. Make API call (e.g., GET /api/items) +3. Check server logs +4. Verify SQL query appears (SELECT, INSERT, etc.) or ORM query log +5. If no DB queries in logs → implementation is using mock data +``` --- @@ -117,6 +201,7 @@ The feature_list.json **MUST** include tests from ALL 20 categories. Minimum cou | Category | Simple | Medium | Complex | | -------------------------------- | ------- | ------- | -------- | +| **0. Infrastructure (REQUIRED)** | 5 | 5 | 5 | | A. Security & Access Control | 5 | 20 | 40 | | B. Navigation Integrity | 15 | 25 | 40 | | C. Real Data Verification | 20 | 30 | 50 | @@ -137,12 +222,14 @@ The feature_list.json **MUST** include tests from ALL 20 categories. Minimum cou | R. Concurrency & Race Conditions | 5 | 8 | 15 | | S. Export/Import | 5 | 6 | 10 | | T. Performance | 5 | 5 | 10 | -| **TOTAL** | **150** | **250** | **400+** | +| **TOTAL** | **165** | **265** | **405+** | --- ### Category Descriptions +**0. Infrastructure (REQUIRED - Priority 0)** - Database connectivity, schema existence, data persistence across server restart, absence of mock patterns. These features MUST pass before any functional features can begin. All tiers require exactly 5 infrastructure features (indices 0-4). + **A. Security & Access Control** - Test unauthorized access blocking, permission enforcement, session management, role-based access, and data isolation between users. **B. Navigation Integrity** - Test all buttons, links, menus, breadcrumbs, deep links, back button behavior, 404 handling, and post-login/logout redirects. @@ -205,6 +292,16 @@ The feature_list.json must include tests that **actively verify real data** and - `setTimeout` simulating API delays with static data - Static returns instead of database queries +**Additional prohibited patterns (in-memory stores):** + +- `globalThis.` (in-memory storage pattern) +- `dev-store`, `devStore`, `DevStore` (development stores) +- `json-server`, `mirage`, `msw` (mock backends) +- `Map()` or `Set()` used as primary data store +- Environment checks like `if (process.env.NODE_ENV === 'development')` for data routing + +**Why this matters:** In-memory stores (like `globalThis.devStore`) will pass simple tests because data persists during a single server run. But data is LOST on server restart, which is unacceptable for production. The Infrastructure features (0-4) specifically test for this by requiring data to survive a full server restart. + --- **CRITICAL INSTRUCTION:** diff --git a/FORK_CHANGELOG.md b/FORK_CHANGELOG.md new file mode 100644 index 00000000..4ab77c66 --- /dev/null +++ b/FORK_CHANGELOG.md @@ -0,0 +1,944 @@ +# Fork Changelog + +All notable changes to this fork are documented in this file. +Format based on [Keep a Changelog](https://keepachangelog.com/). + +## [Unreleased] + +### Added +- Fork documentation (FORK_README.md, FORK_CHANGELOG.md) +- Configuration system via `.autocoder/config.json` + +## [2025-01-25] Infrastructure Features & Mock Data Prevention + +### Problem Addressed +When creating projects, the coding agent could implement in-memory storage (e.g., `dev-store.ts` with `globalThis`) instead of a real database. These implementations passed all tests because data persisted during a single server run, but data was lost on server restart. + +### Added +- **Infrastructure Features (Indices 0-4)** - 5 mandatory features that must pass before any functional features: + - Feature 0: Database connection established + - Feature 1: Database schema applied correctly + - Feature 2: Data persists across server restart (CRITICAL) + - Feature 3: No mock data patterns in codebase + - Feature 4: Backend API queries real database + +- **STEP 5.7: Server Restart Persistence Test** - Mandatory test in coding prompt that catches dev-store implementations by: + 1. Creating test data + 2. Stopping the server completely + 3. Restarting the server + 4. Verifying data still exists + +- **Extended Mock Data Detection (STEP 5.6)** - Comprehensive grep patterns to detect: + - `globalThis.` (in-memory storage) + - `devStore`, `dev-store`, `DevStore`, `mock-db`, `mockDb` + - `mockData`, `fakeData`, `sampleData`, `dummyData`, `testData` + - `TODO.*real`, `TODO.*database`, `STUB`, `MOCK` + - `isDevelopment`, `isDev`, `process.env.NODE_ENV.*development` + - `new Map()`, `new Set()` as data stores + +- **Phase 3b: Database Requirements Question** - Mandatory question in create-spec to determine if project needs database + +### Changed +- **initializer_prompt.template.md**: + - Added Infrastructure category (Priority 0) to category distribution table + - Added MANDATORY INFRASTRUCTURE FEATURES section with detailed test steps + - Extended NO MOCK DATA section with prohibited patterns + - Updated dependency rules - infrastructure features block ALL other features + - Updated example to show infrastructure tier first + - Updated reference tiers: 155/255/405+ (includes 5 infrastructure) + +- **coding_prompt.template.md**: + - Extended STEP 5.6 with comprehensive grep patterns + - Added STEP 5.7: SERVER RESTART PERSISTENCE TEST + - Added checklist items for mock detection and server restart verification + +- **create-spec.md**: + - Added Phase 3b with mandatory database requirements question + - Updated Phase 4L to include infrastructure features in count and breakdown + - Added branching logic for stateless apps vs database apps + +### Files Modified +| File | Changes | +|------|---------| +| `.claude/templates/initializer_prompt.template.md` | Infrastructure category, features 0-4, extended prohibited patterns | +| `.claude/templates/coding_prompt.template.md` | Extended grep, STEP 5.7 server restart test, checklist updates | +| `.claude/commands/create-spec.md` | Database question, infrastructure in feature count | + +### Dependency Pattern +``` +Infrastructure (0-4): NO dependencies - run first +├── Foundation (5-9): depend on [0,1,2,3,4] +│ ├── Auth (10+): depend on [0,1,2,3,4] + foundation +│ │ ├── Core Features: depend on auth + infrastructure +``` + +### Expected Result +- **Before**: Agent could create dev-store.ts and "pass" tests with in-memory data +- **After**: Feature #2 (persist across restart) and Feature #3 (no mock patterns) will FAIL if mock data is used, forcing real database implementation + +### YOLO Mode Compatibility +Infrastructure features work in YOLO mode because: +- Features 0-4 use bash/grep checks, not browser automation +- Feature 2 (server restart) can use curl instead of browser +- Feature 3 (no mock patterns) uses only grep + +--- + +## [2025-01-21] Visual Regression Testing + +### Added +- New module: `visual_regression.py` - Screenshot comparison testing +- New router: `server/routers/visual_regression.py` - REST API for visual testing + +### Features +- **Screenshot capture** via Playwright (chromium) +- **Baseline management** in `.visual-snapshots/baselines/` +- **Diff generation** with pixel-level comparison +- **Multi-viewport support** (desktop 1920x1080, tablet 768x1024, mobile 375x667) +- **Configurable threshold** for acceptable difference (default: 0.1%) +- **Automatic reports** saved to `.visual-snapshots/reports/` + +### Storage Structure +``` +.visual-snapshots/ +├── baselines/ # Baseline screenshots +│ ├── home_desktop.png +│ └── dashboard_mobile.png +├── current/ # Latest test screenshots +├── diffs/ # Diff images (only when failed) +└── reports/ # JSON test reports +``` + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/visual/test` | POST | Run visual tests | +| `/api/visual/baselines/{project}` | GET | List baselines | +| `/api/visual/reports/{project}` | GET | List reports | +| `/api/visual/reports/{project}/{filename}` | GET | Get report | +| `/api/visual/update-baseline` | POST | Accept current as baseline | +| `/api/visual/baselines/{project}/{name}/{viewport}` | DELETE | Delete baseline | +| `/api/visual/snapshot/{project}/{type}/{filename}` | GET | Get snapshot image | + +### Usage +```python +from visual_regression import VisualRegressionTester, run_visual_tests + +# Quick test +report = await run_visual_tests( + project_dir, + base_url="http://localhost:3000", + routes=[ + {"path": "/", "name": "home"}, + {"path": "/dashboard", "name": "dashboard", "wait_for": "#app"}, + ], +) + +# Custom configuration +tester = VisualRegressionTester( + project_dir, + threshold=0.1, + viewports=[Viewport.desktop(), Viewport.mobile()], +) +report = await tester.test_page("http://localhost:3000", "homepage") +``` + +### Configuration +```json +{ + "visual_regression": { + "enabled": true, + "threshold": 0.1, + "capture_on_pass": true, + "viewports": [ + {"name": "desktop", "width": 1920, "height": 1080}, + {"name": "mobile", "width": 375, "height": 667} + ] + } +} +``` + +### Requirements +```bash +pip install playwright Pillow +playwright install chromium +``` + +### How to Disable +```json +{"visual_regression": {"enabled": false}} +``` + +--- + +## [2025-01-21] Design Tokens + +### Added +- New module: `design_tokens.py` - Design tokens management system +- New router: `server/routers/design_tokens.py` - REST API for token management + +### Token Categories +| Category | Description | +|----------|-------------| +| `colors` | Primary, secondary, accent, semantic colors with auto-generated shades | +| `spacing` | Spacing scale (default: 4, 8, 12, 16, 24, 32, 48, 64, 96) | +| `typography` | Font families, sizes, weights, line heights | +| `borders` | Border radii and widths | +| `shadows` | Box shadow definitions | +| `animations` | Durations and easing functions | + +### Generated Files +| File | Description | +|------|-------------| +| `tokens.css` | CSS custom properties with color shades | +| `_tokens.scss` | SCSS variables | +| `tailwind.tokens.js` | Tailwind CSS extend config | + +### Color Shades +Automatically generates 50-950 shades from base colors: +- 50, 100, 200, 300, 400 (lighter) +- 500 (base color) +- 600, 700, 800, 900, 950 (darker) + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/design-tokens/{project}` | GET | Get current tokens | +| `/api/design-tokens/{project}` | PUT | Update tokens | +| `/api/design-tokens/{project}/generate` | POST | Generate token files | +| `/api/design-tokens/{project}/preview/{format}` | GET | Preview output (css/scss/tailwind) | +| `/api/design-tokens/{project}/validate` | POST | Validate tokens | +| `/api/design-tokens/{project}/reset` | POST | Reset to defaults | + +### app_spec.txt Support +```xml + + + #3B82F6 + #6366F1 + #F59E0B + + + [4, 8, 12, 16, 24, 32, 48] + + + Inter, system-ui, sans-serif + + +``` + +### Usage +```python +from design_tokens import DesignTokensManager, generate_design_tokens + +# Quick generation +files = generate_design_tokens(project_dir) + +# Custom management +manager = DesignTokensManager(project_dir) +tokens = manager.load() +manager.generate_css(tokens, output_path) +manager.generate_tailwind_config(tokens, output_path) + +# Validate accessibility +issues = manager.validate_contrast(tokens) +``` + +### Configuration +```json +{ + "design_tokens": { + "enabled": true, + "output_dir": "src/styles", + "generate_on_init": true + } +} +``` + +### How to Disable +```json +{"design_tokens": {"enabled": false}} +``` + +--- + +## [2025-01-21] Auto Documentation + +### Added +- New module: `auto_documentation.py` - Automatic documentation generation +- New router: `server/routers/documentation.py` - REST API for documentation management + +### Generated Files +| File | Location | Description | +|------|----------|-------------| +| `README.md` | Project root | Project overview with features, tech stack, setup | +| `SETUP.md` | `docs/` | Detailed setup guide with prerequisites | +| `API.md` | `docs/` | API endpoint documentation | + +### Documentation Content +- **Project name and description** - From app_spec.txt +- **Tech stack** - Auto-detected from package.json, requirements.txt +- **Features** - From features.db with completion status +- **Setup steps** - From init.sh, package.json scripts +- **Environment variables** - From .env.example +- **API endpoints** - Extracted from Express/FastAPI routes +- **Components** - Extracted from React/Vue components + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/docs/generate` | POST | Generate documentation | +| `/api/docs/{project}` | GET | List documentation files | +| `/api/docs/{project}/{filename}` | GET | Get documentation content | +| `/api/docs/preview` | POST | Preview README without writing | +| `/api/docs/{project}/{filename}` | DELETE | Delete documentation file | + +### Usage +```python +from auto_documentation import DocumentationGenerator, generate_documentation + +# Quick generation +files = generate_documentation(project_dir) + +# Custom generation +generator = DocumentationGenerator(project_dir, output_dir="docs") +docs = generator.generate() +generator.write_readme(docs) +generator.write_api_docs(docs) +generator.write_setup_guide(docs) +``` + +### Configuration +```json +{ + "docs": { + "enabled": true, + "generate_on_init": false, + "generate_on_complete": true, + "output_dir": "docs" + } +} +``` + +### How to Disable +```json +{"docs": {"enabled": false}} +``` + +--- + +## [2025-01-21] Review Agent + +### Added +- New module: `review_agent.py` - Automatic code review with AST-based analysis +- New router: `server/routers/review.py` - REST API for code review operations + +### Issue Categories +| Category | Description | +|----------|-------------| +| `dead_code` | Unused imports, variables, functions | +| `naming` | Naming convention violations | +| `error_handling` | Bare except, silent exception swallowing | +| `security` | eval(), exec(), shell=True, pickle | +| `complexity` | Long functions, too many parameters | +| `documentation` | TODO/FIXME comments | +| `style` | Code style issues | + +### Issue Severities +- **error** - Critical issues that must be fixed +- **warning** - Issues that should be addressed +- **info** - Informational findings +- **style** - Style suggestions + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/review/run` | POST | Run code review | +| `/api/review/reports/{project}` | GET | List review reports | +| `/api/review/reports/{project}/{filename}` | GET | Get specific report | +| `/api/review/create-features` | POST | Create features from issues | +| `/api/review/reports/{project}/{filename}` | DELETE | Delete a report | + +### Python Checks +- Unused imports (AST-based) +- Class naming (PascalCase) +- Function naming (snake_case) +- Bare except clauses +- Empty exception handlers +- Long functions (>50 lines) +- Too many parameters (>7) +- Security patterns (eval, exec, pickle, shell=True) + +### JavaScript/TypeScript Checks +- console.log statements +- TODO/FIXME comments +- Security patterns (eval, innerHTML, dangerouslySetInnerHTML) + +### Usage +```python +from review_agent import ReviewAgent, run_review + +# Quick review +report = run_review(project_dir) + +# Custom review +agent = ReviewAgent( + project_dir, + check_dead_code=True, + check_naming=True, + check_security=True, +) +report = agent.review(commits=["abc123"]) +features = agent.get_issues_as_features() +``` + +### Reports +Reports are saved to `.autocoder/review-reports/review_YYYYMMDD_HHMMSS.json` + +### Configuration +```json +{ + "review": { + "enabled": true, + "trigger_after_features": 5, + "checks": { + "dead_code": true, + "naming": true, + "error_handling": true, + "security": true, + "complexity": true + } + } +} +``` + +### How to Disable +```json +{"review": {"enabled": false}} +``` + +--- + +## [2025-01-21] Import Wizard UI + +### Added +- New hook: `ui/src/hooks/useImportProject.ts` - State management for import workflow +- New component: `ui/src/components/ImportProjectModal.tsx` - Multi-step import wizard + +### Wizard Steps +1. **Folder Selection** - Browse and select existing project folder +2. **Stack Detection** - View detected technologies and confidence scores +3. **Feature Extraction** - Extract features from routes and endpoints +4. **Feature Review** - Select which features to import (toggle individual features) +5. **Registration** - Name and register the project +6. **Completion** - Features created in database + +### Features +- Category-based feature grouping with expand/collapse +- Individual feature selection with checkboxes +- Select All / Deselect All buttons +- Shows source type (route, endpoint, component) +- Shows source file location +- Displays detection confidence scores +- Progress indicators for each step + +### UI Integration +- Added "Import Existing Project" option to NewProjectModal +- Users can choose between "Create New" and "Import Existing" + +### Usage +1. Click "New Project" in the UI +2. Select "Import Existing Project" +3. Browse and select your project folder +4. Review detected tech stack +5. Click "Extract Features" +6. Select features to import +7. Enter project name and complete import + +--- + +## [2025-01-21] Template Library + +### Added +- New module: `templates/` - Project template library +- New router: `server/routers/templates.py` - REST API for templates + +### Available Templates +| Template | Description | Features | +|----------|-------------|----------| +| `saas-starter` | Multi-tenant SaaS with auth, billing | ~45 | +| `ecommerce` | Online store with cart, checkout | ~50 | +| `admin-dashboard` | Admin panel with CRUD, charts | ~40 | +| `blog-cms` | Blog/CMS with posts, comments | ~35 | +| `api-service` | RESTful API with auth, docs | ~30 | + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/templates` | GET | List all templates | +| `/api/templates/{id}` | GET | Get template details | +| `/api/templates/preview` | POST | Preview app_spec.txt | +| `/api/templates/apply` | POST | Apply template to project | +| `/api/templates/{id}/features` | GET | Get template features | + +### Template Format (YAML) +```yaml +name: "Template Name" +description: "Description" +tech_stack: + frontend: "Next.js" + backend: "FastAPI" + database: "PostgreSQL" +feature_categories: + authentication: + - "User login" + - "User registration" +design_tokens: + colors: + primary: "#3B82F6" +estimated_features: 30 +tags: ["saas", "auth"] +``` + +### Usage +```bash +# List templates +curl http://localhost:8888/api/templates + +# Get template details +curl http://localhost:8888/api/templates/saas-starter + +# Preview app_spec.txt +curl -X POST http://localhost:8888/api/templates/preview \ + -H "Content-Type: application/json" \ + -d '{"template_id": "saas-starter", "app_name": "My SaaS"}' + +# Apply template +curl -X POST http://localhost:8888/api/templates/apply \ + -H "Content-Type: application/json" \ + -d '{"template_id": "saas-starter", "project_name": "my-saas", "project_dir": "/path/to/project"}' +``` + +--- + +## [2025-01-21] CI/CD Integration + +### Added +- New module: `integrations/ci/` - CI/CD workflow generation +- New router: `server/routers/cicd.py` - REST API for workflow management + +### Generated Workflows +| Workflow | Filename | Triggers | +|----------|----------|----------| +| CI | `ci.yml` | Push to branches, PRs | +| Security | `security.yml` | Push/PR to main, weekly | +| Deploy | `deploy.yml` | Push to main, manual | + +### CI Workflow Jobs +- **Lint**: ESLint, ruff +- **Type Check**: TypeScript tsc, mypy +- **Test**: npm test, pytest +- **Build**: Production build + +### Security Workflow Jobs +- **NPM Audit**: Dependency vulnerability scan +- **Pip Audit**: Python dependency scan +- **CodeQL**: GitHub code scanning + +### Deploy Workflow Jobs +- **Build**: Create production artifacts +- **Deploy Staging**: Auto-deploy on merge to main +- **Deploy Production**: Manual trigger only + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/cicd/generate` | POST | Generate workflows | +| `/api/cicd/preview` | POST | Preview workflow YAML | +| `/api/cicd/workflows/{project}` | GET | List existing workflows | +| `/api/cicd/workflows/{project}/{filename}` | GET | Get workflow content | + +### Usage +```bash +# Generate all workflows +curl -X POST http://localhost:8888/api/cicd/generate \ + -H "Content-Type: application/json" \ + -d '{"project_name": "my-project"}' + +# Preview CI workflow +curl -X POST http://localhost:8888/api/cicd/preview \ + -H "Content-Type: application/json" \ + -d '{"project_name": "my-project", "workflow_type": "ci"}' +``` + +### Stack Detection +Automatically detects: +- Node.js version from `engines` in package.json +- Package manager (npm, yarn, pnpm, bun) +- TypeScript, React, Next.js, Vue +- Python version from pyproject.toml +- FastAPI, Django + +--- + +## [2025-01-21] Feature Branches Git Workflow + +### Added +- New module: `git_workflow.py` - Git workflow management for feature branches +- New router: `server/routers/git_workflow.py` - REST API for git operations + +### Workflow Modes +| Mode | Description | +|------|-------------| +| `feature_branches` | Create branch per feature, merge on completion | +| `trunk` | All changes on main branch (default) | +| `none` | No git operations | + +### Branch Naming +- Format: `feature/{id}-{slugified-name}` +- Example: `feature/42-user-can-login` + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/git/status/{project}` | GET | Get current git status | +| `/api/git/start-feature` | POST | Start feature (create branch) | +| `/api/git/complete-feature` | POST | Complete feature (merge) | +| `/api/git/abort-feature` | POST | Abort feature | +| `/api/git/commit` | POST | Commit changes | +| `/api/git/branches/{project}` | GET | List feature branches | + +### Configuration +```json +{ + "git_workflow": { + "mode": "feature_branches", + "branch_prefix": "feature/", + "main_branch": "main", + "auto_merge": false + } +} +``` + +### Usage +```python +from git_workflow import get_workflow + +workflow = get_workflow(project_dir) + +# Start working on a feature +result = workflow.start_feature(42, "User can login") + +# Commit progress +result = workflow.commit_feature_progress(42, "Add login form") + +# Complete feature (merge to main if auto_merge enabled) +result = workflow.complete_feature(42) +``` + +--- + +## [2025-01-21] Security Scanning + +### Added +- New module: `security_scanner.py` - Vulnerability detection for code and dependencies +- New router: `server/routers/security.py` - REST API for security scanning + +### Vulnerability Types Detected +| Type | Description | +|------|-------------| +| Dependency | Vulnerable packages via npm audit / pip-audit | +| Secret | Hardcoded API keys, passwords, tokens | +| SQL Injection | String formatting in SQL queries | +| XSS | innerHTML, document.write, dangerouslySetInnerHTML | +| Command Injection | shell=True, exec/eval with concatenation | +| Path Traversal | File operations with string concatenation | +| Insecure Crypto | MD5/SHA1, random.random() | + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/security/scan` | POST | Run security scan | +| `/api/security/reports/{project}` | GET | List scan reports | +| `/api/security/reports/{project}/{filename}` | GET | Get specific report | +| `/api/security/latest/{project}` | GET | Get latest report | + +### Secret Patterns Detected +- AWS Access Keys and Secret Keys +- GitHub Tokens +- Slack Tokens +- Private Keys (RSA, EC, DSA) +- Generic API keys and tokens +- Database connection strings with credentials +- JWT tokens + +### Usage +```python +from security_scanner import scan_project + +result = scan_project(project_dir) +print(f"Found {result.summary['total_issues']} issues") +print(f"Critical: {result.summary['critical']}") +print(f"High: {result.summary['high']}") +``` + +### Reports +Reports are saved to `.autocoder/security-reports/security_scan_YYYYMMDD_HHMMSS.json` + +--- + +## [2025-01-21] Enhanced Logging System + +### Added +- New module: `structured_logging.py` - Structured JSON logging with SQLite storage +- New router: `server/routers/logs.py` - REST API for log querying and export + +### Log Format +```json +{ + "timestamp": "2025-01-21T10:30:00.000Z", + "level": "info|warn|error", + "agent_id": "coding-42", + "feature_id": 42, + "tool_name": "feature_mark_passing", + "duration_ms": 150, + "message": "Feature marked as passing" +} +``` + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/logs/{project_name}` | GET | Query logs with filters | +| `/api/logs/{project_name}/timeline` | GET | Get activity timeline | +| `/api/logs/{project_name}/stats` | GET | Get per-agent statistics | +| `/api/logs/export` | POST | Export logs to file | +| `/api/logs/{project_name}/download/{filename}` | GET | Download exported file | + +### Features +- Filter by level, agent, feature, tool +- Full-text search in messages +- Timeline view bucketed by configurable intervals +- Per-agent statistics (info/warn/error counts) +- Export to JSON, JSONL, CSV formats +- Auto-cleanup old logs (configurable max entries) + +### Usage +```python +from structured_logging import get_logger, get_log_query + +# Create logger for an agent +logger = get_logger(project_dir, agent_id="coding-1") +logger.info("Starting feature", feature_id=42) +logger.error("Test failed", feature_id=42, tool_name="playwright") + +# Query logs +query = get_log_query(project_dir) +logs = query.query(level="error", agent_id="coding-1", limit=50) +timeline = query.get_timeline(since_hours=24) +stats = query.get_agent_stats() +``` + +--- + +## [2025-01-21] Import Project API (Import Projects - Phase 2) + +### Added +- New router: `server/routers/import_project.py` - REST API for project import +- New module: `analyzers/feature_extractor.py` - Transform routes to features + +### API Endpoints +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/api/import/analyze` | POST | Analyze directory, detect stack | +| `/api/import/extract-features` | POST | Generate features from analysis | +| `/api/import/create-features` | POST | Create features in database | +| `/api/import/quick-detect` | GET | Quick stack preview | + +### Feature Extraction +- Routes -> "View X page" navigation features +- API endpoints -> "API: Create/List/Update/Delete X" features +- Infrastructure -> Startup, health check features +- Each feature includes category, name, description, steps + +### Usage +```bash +# 1. Analyze project +curl -X POST http://localhost:8888/api/import/analyze \ + -H "Content-Type: application/json" \ + -d '{"path": "/path/to/existing/project"}' + +# 2. Extract features +curl -X POST http://localhost:8888/api/import/extract-features \ + -H "Content-Type: application/json" \ + -d '{"path": "/path/to/existing/project"}' + +# 3. Create features in registered project +curl -X POST http://localhost:8888/api/import/create-features \ + -H "Content-Type: application/json" \ + -d '{"project_name": "my-project", "features": [...]}' +``` + +--- + +## [2025-01-21] Stack Detector (Import Projects - Phase 1) + +### Added +- New module: `analyzers/` - Codebase analysis for project import +- `analyzers/base_analyzer.py` - Abstract base class with TypedDicts +- `analyzers/stack_detector.py` - Orchestrator for running all analyzers +- `analyzers/react_analyzer.py` - React, Vite, Next.js detection +- `analyzers/node_analyzer.py` - Express, NestJS, Fastify detection +- `analyzers/python_analyzer.py` - FastAPI, Django, Flask detection +- `analyzers/vue_analyzer.py` - Vue.js, Nuxt detection + +### Features +- Auto-detect tech stack from package.json, requirements.txt, config files +- Extract routes from React Router, Next.js file-based, Vue Router +- Extract API endpoints from Express, FastAPI, Django, NestJS +- Extract components from components/, views/, models/ directories +- Confidence scoring for each detected stack + +### Usage +```python +from analyzers import StackDetector + +detector = StackDetector(project_dir) +result = detector.detect() # Full analysis +quick = detector.detect_quick() # Fast preview +``` + +### Supported Stacks +| Stack | Indicators | +|-------|-----------| +| React | "react" in package.json, src/App.tsx | +| Next.js | next.config.js, pages/ or app/ dirs | +| Vue.js | "vue" in package.json, src/App.vue | +| Nuxt | nuxt.config.js, pages/ | +| Express | "express" in package.json, routes/ | +| NestJS | "@nestjs/core" in package.json | +| FastAPI | "from fastapi import" in main.py | +| Django | manage.py in root | +| Flask | "from flask import" in app.py | + +--- + +## [2025-01-21] Quality Gates + +### Added +- New module: `quality_gates.py` - Quality checking logic (lint, type-check, custom scripts) +- New MCP tool: `feature_verify_quality` - Run quality checks on demand +- Auto-detection of linters: ESLint, Biome, ruff, flake8 +- Auto-detection of type checkers: TypeScript (tsc), Python (mypy) +- Support for custom quality scripts via `.autocoder/quality-checks.sh` + +### Changed +- Modified `feature_mark_passing` - Now enforces quality checks in strict mode +- In strict mode, `feature_mark_passing` BLOCKS if lint or type-check fails +- Quality results are stored in the `quality_result` DB column + +### Configuration +- `quality_gates.enabled`: Enable/disable quality gates (default: true) +- `quality_gates.strict_mode`: Block feature_mark_passing on failure (default: true) +- `quality_gates.checks.lint`: Run lint check (default: true) +- `quality_gates.checks.type_check`: Run type check (default: true) +- `quality_gates.checks.custom_script`: Path to custom script (optional) + +### How to Disable +```json +{"quality_gates": {"enabled": false}} +``` +Or for non-blocking mode: +```json +{"quality_gates": {"strict_mode": false}} +``` + +### Related Issues +- Addresses #68 (Agent skips features without testing) +- Addresses #69 (Test evidence storage) + +--- + +## [2025-01-21] Error Recovery + +### Added +- New DB columns: `failure_reason`, `failure_count`, `last_failure_at`, `quality_result` in Feature model +- New MCP tool: `feature_report_failure` - Report failures with escalation recommendations +- New MCP tool: `feature_get_stuck` - Get all features that have failed at least once +- New MCP tool: `feature_clear_all_in_progress` - Clear all stuck features at once +- New MCP tool: `feature_reset_failure` - Reset failure tracking for a feature +- New helper: `clear_stuck_features()` in `progress.py` - Auto-clear on agent startup +- Auto-recovery on agent startup: Clears stuck features from interrupted sessions + +### Changed +- Modified `api/database.py` - Added error recovery and quality result columns with auto-migration +- Modified `agent.py` - Calls `clear_stuck_features()` on startup +- Modified `mcp_server/feature_mcp.py` - Added error recovery MCP tools + +### Configuration +- New config section: `error_recovery` with `max_retries`, `skip_threshold`, `escalate_threshold`, `auto_clear_on_startup` + +### How to Disable +```json +{"error_recovery": {"auto_clear_on_startup": false}} +``` + +### Related Issues +- Fixes features stuck after stop (common issue when agents are interrupted) + +--- + +## Entry Template + +When adding a new feature, use this template: + +```markdown +## [YYYY-MM-DD] Feature Name + +### Added +- New file: `path/to/file.py` - Description +- New component: `ComponentName` - Description + +### Changed +- Modified `file.py` - What changed and why + +### Configuration +- New config option: `config.key` - What it does + +### How to Disable +\`\`\`json +{"feature_name": {"enabled": false}} +\`\`\` + +### Related Issues +- Closes #XX (upstream issue) +``` + +--- + +## Planned Features + +The following features are planned for implementation: + +### Phase 1: Foundation (Quick Wins) +- [x] Enhanced Logging - Structured logs with filtering ✅ +- [x] Quality Gates - Lint/type-check before marking passing ✅ +- [x] Security Scanning - Detect vulnerabilities ✅ + +### Phase 2: Import Projects +- [x] Stack Detector - Detect React, Next.js, Express, FastAPI, Django, Vue.js ✅ +- [x] Feature Extractor - Reverse-engineer features from routes/endpoints ✅ +- [x] Import Wizard API - REST endpoints for import flow ✅ +- [x] Import Wizard UI - Chat-based project import (UI component) ✅ + +### Phase 3: Workflow Improvements +- [x] Feature Branches - Git workflow with feature branches ✅ +- [x] Error Recovery - Handle stuck features, auto-clear on startup ✅ +- [x] Review Agent - Automatic code review ✅ +- [x] CI/CD Integration - GitHub Actions generation ✅ + +### Phase 4: Polish & Ecosystem +- [x] Template Library - SaaS, e-commerce, dashboard templates ✅ +- [x] Auto Documentation - README, API docs generation ✅ +- [x] Design Tokens - Consistent styling ✅ +- [x] Visual Regression - Screenshot comparison testing ✅ diff --git a/FORK_README.md b/FORK_README.md new file mode 100644 index 00000000..73974ff1 --- /dev/null +++ b/FORK_README.md @@ -0,0 +1,135 @@ +# Autocoder Fork - Enhanced Features + +This is a fork of [leonvanzyl/autocoder](https://github.com/leonvanzyl/autocoder) +with additional features for improved developer experience. + +## What's Different in This Fork + +### New Features + +- **Import Existing Projects** - Import existing codebases and continue development with Autocoder +- **Quality Gates** - Automatic code quality checks (lint, type-check) before marking features as passing +- **Enhanced Logging** - Better debugging with filterable, searchable, structured logs +- **Security Scanning** - Detect vulnerabilities in generated code (secrets, injection patterns) +- **Feature Branches** - Professional git workflow with automatic feature branch creation +- **Error Recovery** - Better handling of stuck features with auto-clear on startup +- **Template Library** - Pre-made templates for common app types (SaaS, e-commerce, dashboard) +- **CI/CD Integration** - GitHub Actions workflows generated automatically + +### Configuration + +All new features can be configured via `.autocoder/config.json`. +See [Configuration Guide](#configuration) for details. + +## Configuration + +Create a `.autocoder/config.json` file in your project directory: + +```json +{ + "version": "1.0", + + "quality_gates": { + "enabled": true, + "strict_mode": true, + "checks": { + "lint": true, + "type_check": true, + "unit_tests": false, + "custom_script": ".autocoder/quality-checks.sh" + } + }, + + "git_workflow": { + "mode": "feature_branches", + "branch_prefix": "feature/", + "auto_merge": false + }, + + "error_recovery": { + "max_retries": 3, + "skip_threshold": 5, + "escalate_threshold": 7 + }, + + "completion": { + "auto_stop_at_100": true, + "max_regression_cycles": 3 + }, + + "ci_cd": { + "provider": "github", + "environments": { + "staging": {"url": "", "auto_deploy": true}, + "production": {"url": "", "auto_deploy": false} + } + }, + + "import": { + "default_feature_status": "pending", + "auto_detect_stack": true + } +} +``` + +### Disabling Features + +Each feature can be disabled individually: + +```json +{ + "quality_gates": { + "enabled": false + }, + "git_workflow": { + "mode": "none" + } +} +``` + +## Staying Updated with Upstream + +This fork regularly syncs with upstream. To get latest upstream changes: + +```bash +git fetch upstream +git checkout master && git merge upstream/master +git checkout my-features && git merge master +``` + +## Reverting Changes + +### Revert to Original + +```bash +# Option 1: Full reset to upstream +git checkout my-features +git reset --hard upstream/master +git push origin my-features --force + +# Option 2: Revert specific commits +git log --oneline # find commit to revert +git revert + +# Option 3: Checkout specific files from upstream +git checkout upstream/master -- path/to/file.py +``` + +### Safety Checkpoint + +Before major changes, create a tag: + +```bash +git tag before-feature-name +# If something goes wrong: +git reset --hard before-feature-name +``` + +## Contributing Back + +Features that could benefit the original project are submitted as PRs to upstream. +See [FORK_CHANGELOG.md](./FORK_CHANGELOG.md) for detailed change history. + +## License + +Same license as the original [leonvanzyl/autocoder](https://github.com/leonvanzyl/autocoder) project. diff --git a/agent.py b/agent.py index 7d904736..f23cdbb4 100644 --- a/agent.py +++ b/agent.py @@ -7,6 +7,7 @@ import asyncio import io +import os import re import sys from datetime import datetime, timedelta @@ -16,6 +17,8 @@ from claude_agent_sdk import ClaudeSDKClient +from structured_logging import get_logger + # Fix Windows console encoding for Unicode characters (emoji, etc.) # Without this, print() crashes when Claude outputs emoji like ✅ if sys.platform == "win32": @@ -23,7 +26,13 @@ sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace", line_buffering=True) from client import create_client -from progress import count_passing_tests, has_features, print_progress_summary, print_session_header +from progress import ( + clear_stuck_features, + count_passing_tests, + has_features, + print_progress_summary, + print_session_header, +) from prompts import ( copy_spec_to_project, get_coding_prompt, @@ -31,6 +40,11 @@ get_single_feature_prompt, get_testing_prompt, ) +from rate_limit_utils import ( + RATE_LIMIT_PATTERNS, + is_rate_limit_error, + parse_retry_after, +) # Configuration AUTO_CONTINUE_DELAY_SECONDS = 3 @@ -40,6 +54,7 @@ async def run_agent_session( client: ClaudeSDKClient, message: str, project_dir: Path, + logger=None, ) -> tuple[str, str]: """ Run a single agent session using Claude Agent SDK. @@ -48,6 +63,7 @@ async def run_agent_session( client: Claude SDK client message: The prompt to send project_dir: Project directory path + logger: Optional structured logger for this session Returns: (status, response_text) where status is: @@ -55,6 +71,8 @@ async def run_agent_session( - "error" if an error occurred """ print("Sending prompt to Claude Agent SDK...\n") + if logger: + logger.info("Starting agent session", prompt_length=len(message)) try: # Send the query @@ -81,6 +99,8 @@ async def run_agent_session( print(f" Input: {input_str[:200]}...", flush=True) else: print(f" Input: {input_str}", flush=True) + if logger: + logger.debug("Tool used", tool_name=block.name, input_size=len(str(getattr(block, "input", "")))) # Handle UserMessage (tool results) elif msg_type == "UserMessage" and hasattr(msg, "content"): @@ -94,20 +114,41 @@ async def run_agent_session( # Check if command was blocked by security hook if "blocked" in str(result_content).lower(): print(f" [BLOCKED] {result_content}", flush=True) + if logger: + logger.error("Security: command blocked", content=str(result_content)[:200]) elif is_error: # Show errors (truncated) error_str = str(result_content)[:500] print(f" [Error] {error_str}", flush=True) + if logger: + logger.error("Tool execution error", error=error_str[:200]) else: # Tool succeeded - just show brief confirmation print(" [Done]", flush=True) print("\n" + "-" * 70 + "\n") + if logger: + logger.info("Agent session completed", response_length=len(response_text)) return "continue", response_text except Exception as e: - print(f"Error during agent session: {e}") - return "error", str(e) + error_str = str(e) + print(f"Error during agent session: {error_str}") + if logger: + logger.error("Agent session error", error_type=type(e).__name__, message=error_str[:200]) + + # Detect rate limit errors from exception message + if is_rate_limit_error(error_str): + # Try to extract retry-after time from error + retry_seconds = parse_retry_after(error_str) + if logger: + logger.warning("Rate limit detected", retry_seconds=retry_seconds) + if retry_seconds is not None: + return "rate_limit", str(retry_seconds) + else: + return "rate_limit", "unknown" + + return "error", error_str async def run_autonomous_agent( @@ -131,6 +172,27 @@ async def run_autonomous_agent( agent_type: Type of agent: "initializer", "coding", "testing", or None (auto-detect) testing_feature_id: For testing agents, the pre-claimed feature ID to test """ + # Initialize structured logger for this agent session + # Agent ID format: "initializer", "coding-", "testing-" + if agent_type == "testing": + agent_id = f"testing-{os.getpid()}" + elif feature_id: + agent_id = f"coding-{feature_id}" + elif agent_type == "initializer": + agent_id = "initializer" + else: + agent_id = "coding-main" + + logger = get_logger(project_dir, agent_id=agent_id, console_output=False) + logger.info( + "Autonomous agent started", + agent_type=agent_type or "auto-detect", + model=model, + yolo_mode=yolo_mode, + max_iterations=max_iterations, + feature_id=feature_id, + ) + print("\n" + "=" * 70) print(" AUTONOMOUS CODING AGENT") print("=" * 70) @@ -151,6 +213,29 @@ async def run_autonomous_agent( # Create project directory project_dir.mkdir(parents=True, exist_ok=True) + # IMPORTANT: Do NOT clear stuck features in parallel mode! + # The orchestrator manages feature claiming atomically. + # Clearing here causes race conditions where features are marked in_progress + # by the orchestrator but immediately cleared by the agent subprocess on startup. + # + # For single-agent mode or manual runs, clearing is still safe because + # there's only one agent at a time and it happens before claiming any features. + # + # Only clear if we're NOT in a parallel orchestrator context + # (detected by checking if this agent is a subprocess spawned by orchestrator) + import psutil + try: + parent_process = psutil.Process().parent() + parent_name = parent_process.name() if parent_process else "" + + # Only clear if parent is NOT python (i.e., we're running manually, not from orchestrator) + if "python" not in parent_name.lower(): + clear_stuck_features(project_dir) + except Exception: + # If parent process check fails, do NOT clear features to avoid race conditions + # in parallel mode. The orchestrator handles clearing stuck features safely. + pass + # Determine agent type if not explicitly set if agent_type is None: # Auto-detect based on whether we have features @@ -183,6 +268,8 @@ async def run_autonomous_agent( # Main loop iteration = 0 + rate_limit_retries = 0 # Track consecutive rate limit errors for exponential backoff + error_retries = 0 # Track consecutive non-rate-limit errors while True: iteration += 1 @@ -192,6 +279,7 @@ async def run_autonomous_agent( if not is_initializer and iteration == 1: passing, in_progress, total = count_passing_tests(project_dir) if total > 0 and passing == total: + logger.info("Project complete on startup", passing=passing, total=total) print("\n" + "=" * 70) print(" ALL FEATURES ALREADY COMPLETE!") print("=" * 70) @@ -208,15 +296,14 @@ async def run_autonomous_agent( print_session_header(iteration, is_initializer) # Create client (fresh context) - # Pass agent_id for browser isolation in multi-agent scenarios - import os + # Pass client_agent_id for browser isolation in multi-agent scenarios if agent_type == "testing": - agent_id = f"testing-{os.getpid()}" # Unique ID for testing agents + client_agent_id = f"testing-{os.getpid()}" # Unique ID for testing agents elif feature_id: - agent_id = f"feature-{feature_id}" + client_agent_id = f"feature-{feature_id}" else: - agent_id = None - client = create_client(project_dir, model, yolo_mode=yolo_mode, agent_id=agent_id) + client_agent_id = None + client = create_client(project_dir, model, yolo_mode=yolo_mode, agent_id=client_agent_id) # Choose prompt based on agent type if agent_type == "initializer": @@ -234,9 +321,10 @@ async def run_autonomous_agent( # Wrap in try/except to handle MCP server startup failures gracefully try: async with client: - status, response = await run_agent_session(client, prompt, project_dir) + status, response = await run_agent_session(client, prompt, project_dir, logger=logger) except Exception as e: print(f"Client/MCP server error: {e}") + logger.error("Client/MCP server error", error_type=type(e).__name__, message=str(e)[:200]) # Don't crash - return error status so the loop can retry status, response = "error", str(e) @@ -250,13 +338,32 @@ async def run_autonomous_agent( # Handle status if status == "continue": + # Reset error retries on success; rate-limit retries reset only if no signal + error_retries = 0 + reset_rate_limit_retries = True + delay_seconds = AUTO_CONTINUE_DELAY_SECONDS target_time_str = None - if "limit reached" in response.lower(): - print("Claude Agent SDK indicated limit reached.") + # Check for rate limit indicators in response text + response_lower = response.lower() + if any(pattern in response_lower for pattern in RATE_LIMIT_PATTERNS): + print("Claude Agent SDK indicated rate limit reached.") + logger.warning("Rate limit signal in response") + reset_rate_limit_retries = False + + # Try to extract retry-after from response text first + retry_seconds = parse_retry_after(response) + if retry_seconds is not None: + delay_seconds = retry_seconds + logger.warning("Rate limit signal in response", delay_seconds=delay_seconds, source="retry-after") + else: + # Use exponential backoff when retry-after unknown + delay_seconds = min(60 * (2 ** rate_limit_retries), 3600) + rate_limit_retries += 1 + logger.warning("Rate limit signal in response", delay_seconds=delay_seconds, source="exponential-backoff", attempt=rate_limit_retries) - # Try to parse reset time from response + # Try to parse reset time from response (more specific format) match = re.search( r"(?i)\bresets(?:\s+at)?\s+(\d+)(?::(\d+))?\s*(am|pm)\s*\(([^)]+)\)", response, @@ -324,19 +431,50 @@ async def run_autonomous_agent( print(f"\nSingle-feature mode: Feature #{feature_id} session complete.") break + # Reset rate limit retries only if no rate limit signal was detected + if reset_rate_limit_retries: + rate_limit_retries = 0 + + await asyncio.sleep(delay_seconds) + + elif status == "rate_limit": + # Smart rate limit handling with exponential backoff + if response != "unknown": + delay_seconds = int(response) + print(f"\nRate limit hit. Waiting {delay_seconds} seconds before retry...") + logger.warning("Rate limit backoff", delay_seconds=delay_seconds, source="known") + else: + # Use exponential backoff when retry-after unknown + delay_seconds = min(60 * (2 ** rate_limit_retries), 3600) # Max 1 hour + rate_limit_retries += 1 + print(f"\nRate limit hit. Backoff wait: {delay_seconds} seconds (attempt #{rate_limit_retries})...") + logger.warning("Rate limit backoff", delay_seconds=delay_seconds, source="exponential", attempt=rate_limit_retries) + await asyncio.sleep(delay_seconds) elif status == "error": + # Non-rate-limit errors: linear backoff capped at 5 minutes + error_retries += 1 + delay_seconds = min(30 * error_retries, 300) # Max 5 minutes print("\nSession encountered an error") - print("Will retry with a fresh session...") - await asyncio.sleep(AUTO_CONTINUE_DELAY_SECONDS) + print(f"Will retry in {delay_seconds}s (attempt #{error_retries})...") + logger.error("Session error, retrying", attempt=error_retries, delay_seconds=delay_seconds) + await asyncio.sleep(delay_seconds) - # Small delay between sessions + # Small delay between sessions (3 seconds as per CLAUDE.md doc) if max_iterations is None or iteration < max_iterations: print("\nPreparing next session...\n") - await asyncio.sleep(1) + await asyncio.sleep(3) # Final summary + passing, in_progress, total = count_passing_tests(project_dir) + logger.info( + "Agent session complete", + iterations=iteration, + passing=passing, + in_progress=in_progress, + total=total, + ) print("\n" + "=" * 70) print(" SESSION COMPLETE") print("=" * 70) diff --git a/analyzers/__init__.py b/analyzers/__init__.py new file mode 100644 index 00000000..5040ec7f --- /dev/null +++ b/analyzers/__init__.py @@ -0,0 +1,35 @@ +""" +Codebase Analyzers +================== + +Modules for analyzing existing codebases to detect tech stack, +extract features, and prepare for import into Autocoder. + +Main entry points: +- StackDetector: Detect tech stack and extract routes/endpoints +- extract_features: Transform detection result into Autocoder features +- extract_from_project: One-step detection and feature extraction +""" + +from .base_analyzer import BaseAnalyzer +from .feature_extractor import ( + DetectedFeature, + FeatureExtractionResult, + extract_features, + extract_from_project, + features_to_bulk_create_format, +) +from .stack_detector import StackDetectionResult, StackDetector + +__all__ = [ + # Stack Detection + "StackDetector", + "StackDetectionResult", + "BaseAnalyzer", + # Feature Extraction + "DetectedFeature", + "FeatureExtractionResult", + "extract_features", + "extract_from_project", + "features_to_bulk_create_format", +] diff --git a/analyzers/base_analyzer.py b/analyzers/base_analyzer.py new file mode 100644 index 00000000..9bb31de2 --- /dev/null +++ b/analyzers/base_analyzer.py @@ -0,0 +1,152 @@ +""" +Base Analyzer +============= + +Abstract base class for all stack analyzers. +Each analyzer detects a specific tech stack and extracts relevant information. +""" + +from abc import ABC, abstractmethod +from pathlib import Path +from typing import TypedDict + + +class RouteInfo(TypedDict): + """Information about a detected route.""" + path: str + method: str # GET, POST, PUT, DELETE, etc. + handler: str # Function or component name + file: str # Source file path + + +class ComponentInfo(TypedDict): + """Information about a detected component.""" + name: str + file: str + type: str # page, component, layout, etc. + + +class EndpointInfo(TypedDict): + """Information about an API endpoint.""" + path: str + method: str + handler: str + file: str + description: str # Generated description + + +class AnalysisResult(TypedDict): + """Result of analyzing a codebase with a specific analyzer.""" + stack_name: str + confidence: float # 0.0 to 1.0 + routes: list[RouteInfo] + components: list[ComponentInfo] + endpoints: list[EndpointInfo] + entry_point: str | None + config_files: list[str] + dependencies: dict[str, str] # name: version + metadata: dict # Additional stack-specific info + + +class BaseAnalyzer(ABC): + """ + Abstract base class for stack analyzers. + + Each analyzer is responsible for: + 1. Detecting if a codebase uses its stack (can_analyze) + 2. Extracting routes, components, and endpoints (analyze) + """ + + def __init__(self, project_dir: Path): + """ + Initialize the analyzer. + + Args: + project_dir: Path to the project directory to analyze + """ + self.project_dir = project_dir + + @property + @abstractmethod + def stack_name(self) -> str: + """The name of the stack this analyzer handles (e.g., 'react', 'nextjs').""" + pass + + @abstractmethod + def can_analyze(self) -> tuple[bool, float]: + """ + Check if this analyzer can handle the codebase. + + Returns: + (can_handle, confidence) where: + - can_handle: True if the analyzer recognizes the stack + - confidence: 0.0 to 1.0 indicating how confident the detection is + """ + pass + + @abstractmethod + def analyze(self) -> AnalysisResult: + """ + Analyze the codebase and extract information. + + Returns: + AnalysisResult with detected routes, components, endpoints, etc. + """ + pass + + def _read_file_safe(self, path: Path, max_size: int = 1024 * 1024) -> str | None: + """ + Safely read a file, returning None if it doesn't exist or is too large. + + Args: + path: Path to the file + max_size: Maximum file size in bytes (default 1MB) + + Returns: + File contents or None + """ + if not path.exists(): + return None + + try: + if path.stat().st_size > max_size: + return None + return path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + return None + + def _find_files(self, pattern: str, exclude_dirs: list[str] | None = None) -> list[Path]: + """ + Find files matching a glob pattern, excluding common non-source directories. + + Args: + pattern: Glob pattern (e.g., "**/*.tsx") + exclude_dirs: Additional directories to exclude + + Returns: + List of matching file paths + """ + default_exclude = [ + "node_modules", + "venv", + ".venv", + "__pycache__", + ".git", + "dist", + "build", + ".next", + ".nuxt", + "coverage", + ] + + if exclude_dirs: + default_exclude.extend(exclude_dirs) + + results = [] + for path in self.project_dir.glob(pattern): + # Check if any parent is in exclude list + parts = path.relative_to(self.project_dir).parts + if not any(part in default_exclude for part in parts): + results.append(path) + + return results diff --git a/analyzers/feature_extractor.py b/analyzers/feature_extractor.py new file mode 100644 index 00000000..4bc41ae4 --- /dev/null +++ b/analyzers/feature_extractor.py @@ -0,0 +1,445 @@ +""" +Feature Extractor +================= + +Transforms detected routes, endpoints, and components into Autocoder features. +Each feature is marked as pending (passes=False) for verification. + +Generates features in the format expected by feature_create_bulk MCP tool. +""" + +from pathlib import Path +from typing import TypedDict + +from .stack_detector import StackDetectionResult + + +class DetectedFeature(TypedDict): + """A feature extracted from codebase analysis.""" + category: str + name: str + description: str + steps: list[str] + source_type: str # "route", "endpoint", "component", "inferred" + source_file: str | None + confidence: float # 0.0 to 1.0 + + +class FeatureExtractionResult(TypedDict): + """Result of feature extraction.""" + features: list[DetectedFeature] + count: int + by_category: dict[str, int] + summary: str + + +def _route_to_feature_name(path: str, method: str = "GET") -> str: + """ + Convert a route path to a human-readable feature name. + + Examples: + "/" -> "View home page" + "/users" -> "View users page" + "/users/:id" -> "View user details page" + "/api/users" -> "API: List users" + """ + # Clean up path + path = path.strip("/") + + if not path: + return "View home page" + + # Handle API routes + if path.startswith("api/"): + api_path = path[4:] # Remove "api/" + parts = api_path.split("/") + + # Handle dynamic segments + parts = [p for p in parts if not p.startswith(":") and not p.startswith("[")] + + if not parts: + return "API: Root endpoint" + + resource = parts[-1].replace("-", " ").replace("_", " ").title() + + if method == "GET": + if any(p.startswith(":") or p.startswith("[") for p in api_path.split("/")): + return f"API: Get {resource} details" + return f"API: List {resource}" + elif method == "POST": + return f"API: Create {resource}" + elif method == "PUT" or method == "PATCH": + return f"API: Update {resource}" + elif method == "DELETE": + return f"API: Delete {resource}" + else: + return f"API: {resource} endpoint" + + # Handle page routes + parts = path.split("/") + + # Handle dynamic segments (remove them from naming) + clean_parts = [p for p in parts if not p.startswith(":") and not p.startswith("[")] + + if not clean_parts: + return "View dynamic page" + + # Build name from path parts + page_name = " ".join(p.replace("-", " ").replace("_", " ") for p in clean_parts) + page_name = page_name.title() + + # Check if it's a detail page (has dynamic segment) + has_dynamic = any(p.startswith(":") or p.startswith("[") for p in parts) + + if has_dynamic: + return f"View {page_name} details page" + + return f"View {page_name} page" + + +def _generate_page_steps(path: str, stack: str | None) -> list[str]: + """Generate test steps for a page route.""" + clean_path = path + + # Replace dynamic segments with example values + if ":id" in clean_path or "[id]" in clean_path: + clean_path = clean_path.replace(":id", "123").replace("[id]", "123") + + # Generate steps + steps = [ + f"Navigate to {clean_path}", + "Verify the page loads without errors", + "Verify the page title and main content are visible", + ] + + # Add stack-specific checks + if stack in ("react", "nextjs", "vue", "nuxt"): + steps.append("Verify no console errors in browser developer tools") + steps.append("Verify responsive layout at mobile and desktop widths") + + return steps + + +def _generate_api_steps(path: str, method: str) -> list[str]: + """Generate test steps for an API endpoint.""" + # Replace dynamic segments with example values + test_path = path.replace(":id", "123").replace("[id]", "123") + + steps = [] + + if method == "GET": + steps = [ + f"Send GET request to {test_path}", + "Verify response status code is 200", + "Verify response body contains expected data structure", + ] + elif method == "POST": + steps = [ + f"Send POST request to {test_path} with valid payload", + "Verify response status code is 201 (created)", + "Verify response contains the created resource", + f"Send POST request to {test_path} with invalid payload", + "Verify response status code is 400 (bad request)", + ] + elif method in ("PUT", "PATCH"): + steps = [ + f"Send {method} request to {test_path} with valid payload", + "Verify response status code is 200", + "Verify response contains the updated resource", + "Verify the resource was actually updated", + ] + elif method == "DELETE": + steps = [ + f"Send DELETE request to {test_path}", + "Verify response status code is 200 or 204", + "Verify the resource no longer exists", + ] + else: + steps = [ + f"Send {method} request to {test_path}", + "Verify response status code is appropriate", + ] + + return steps + + +def _generate_component_steps(name: str, comp_type: str) -> list[str]: + """Generate test steps for a component.""" + if comp_type == "page": + return [ + f"Navigate to the {name} page", + "Verify all UI elements render correctly", + "Test user interactions (buttons, forms, etc.)", + "Verify data is fetched and displayed", + ] + elif comp_type == "model": + return [ + f"Verify {name} model schema matches expected fields", + "Test CRUD operations on the model", + "Verify validation rules work correctly", + ] + elif comp_type == "middleware": + return [ + f"Verify {name} middleware processes requests correctly", + "Test edge cases and error handling", + ] + elif comp_type == "service": + return [ + f"Verify {name} service methods work correctly", + "Test error handling in service layer", + ] + else: + return [ + f"Verify {name} component renders correctly", + "Test component props and state", + "Verify component interactions work", + ] + + +def extract_features(detection_result: StackDetectionResult) -> FeatureExtractionResult: + """ + Extract features from a stack detection result. + + Converts routes, endpoints, and components into Autocoder features. + Each feature is ready to be created via feature_create_bulk. + + Args: + detection_result: Result from StackDetector.detect() + + Returns: + FeatureExtractionResult with list of features + """ + features: list[DetectedFeature] = [] + primary_frontend = detection_result.get("primary_frontend") + + # Track unique features to avoid duplicates + seen_features: set[str] = set() + + # Extract features from routes (frontend pages) + for route in detection_result.get("all_routes", []): + path = route.get("path", "") + method = route.get("method", "GET") + source_file = route.get("file") + + feature_name = _route_to_feature_name(path, method) + + # Skip duplicates + feature_key = f"route:{path}:{method}" + if feature_key in seen_features: + continue + seen_features.add(feature_key) + + features.append({ + "category": "Navigation", + "name": feature_name, + "description": f"User can navigate to and view the {path or '/'} page. The page should load correctly and display the expected content.", + "steps": _generate_page_steps(path, primary_frontend), + "source_type": "route", + "source_file": source_file, + "confidence": 0.8, + }) + + # Extract features from API endpoints + for endpoint in detection_result.get("all_endpoints", []): + path = endpoint.get("path", "") + method = endpoint.get("method", "ALL") + source_file = endpoint.get("file") + + # Handle ALL method by creating GET endpoint + if method == "ALL": + method = "GET" + + feature_name = _route_to_feature_name(path, method) + + # Skip duplicates + feature_key = f"endpoint:{path}:{method}" + if feature_key in seen_features: + continue + seen_features.add(feature_key) + + # Determine category based on path + category = "API" + path_lower = path.lower() + if "auth" in path_lower or "login" in path_lower or "register" in path_lower: + category = "Authentication" + elif "user" in path_lower or "profile" in path_lower: + category = "User Management" + elif "admin" in path_lower: + category = "Administration" + + features.append({ + "category": category, + "name": feature_name, + "description": f"{method} endpoint at {path}. Should handle requests appropriately and return correct responses.", + "steps": _generate_api_steps(path, method), + "source_type": "endpoint", + "source_file": source_file, + "confidence": 0.85, + }) + + # Extract features from components (with lower priority) + component_features: list[DetectedFeature] = [] + for component in detection_result.get("all_components", []): + name = component.get("name", "") + comp_type = component.get("type", "component") + source_file = component.get("file") + + # Skip common/generic components + skip_names = ["index", "app", "main", "layout", "_app", "_document"] + if name.lower() in skip_names: + continue + + # Skip duplicates + feature_key = f"component:{name}:{comp_type}" + if feature_key in seen_features: + continue + seen_features.add(feature_key) + + # Only include significant components + if comp_type in ("page", "view", "model", "service"): + clean_name = name.replace("-", " ").replace("_", " ").title() + + # Determine category + if comp_type == "model": + category = "Data Models" + elif comp_type == "service": + category = "Services" + elif comp_type in ("page", "view"): + category = "Pages" + else: + category = "Components" + + component_features.append({ + "category": category, + "name": f"{clean_name} {comp_type.title()}", + "description": f"The {clean_name} {comp_type} should function correctly and handle all expected use cases.", + "steps": _generate_component_steps(name, comp_type), + "source_type": "component", + "source_file": source_file, + "confidence": 0.6, # Lower confidence for component-based features + }) + + # Add component features if we don't have many from routes/endpoints + if len(features) < 10: + features.extend(component_features[:10]) # Limit to 10 component features + + # Add basic infrastructure features + basic_features = _generate_basic_features(detection_result) + features.extend(basic_features) + + # Count by category + by_category: dict[str, int] = {} + for f in features: + cat = f["category"] + by_category[cat] = by_category.get(cat, 0) + 1 + + # Build summary + summary = f"Extracted {len(features)} features from {len(detection_result.get('detected_stacks', []))} detected stack(s)" + + return { + "features": features, + "count": len(features), + "by_category": by_category, + "summary": summary, + } + + +def _generate_basic_features(detection_result: StackDetectionResult) -> list[DetectedFeature]: + """Generate basic infrastructure features based on detected stack.""" + features: list[DetectedFeature] = [] + + primary_frontend = detection_result.get("primary_frontend") + primary_backend = detection_result.get("primary_backend") + + # Application startup feature + if primary_frontend or primary_backend: + features.append({ + "category": "Infrastructure", + "name": "Application starts successfully", + "description": "The application should start without errors and be accessible.", + "steps": [ + "Run the application start command", + "Verify the server starts without errors", + "Access the application URL", + "Verify the main page loads", + ], + "source_type": "inferred", + "source_file": None, + "confidence": 1.0, + }) + + # Frontend-specific features + if primary_frontend in ("react", "nextjs", "vue", "nuxt"): + features.append({ + "category": "Infrastructure", + "name": "No console errors on page load", + "description": "The application should load without JavaScript errors in the browser console.", + "steps": [ + "Open browser developer tools", + "Navigate to the home page", + "Check the console for errors", + "Navigate to other pages and repeat", + ], + "source_type": "inferred", + "source_file": None, + "confidence": 0.9, + }) + + # Backend-specific features + if primary_backend in ("express", "fastapi", "django", "flask", "nestjs"): + features.append({ + "category": "Infrastructure", + "name": "Health check endpoint responds", + "description": "The API should have a health check endpoint that responds correctly.", + "steps": [ + "Send GET request to /health or /api/health", + "Verify response status is 200", + "Verify response indicates healthy status", + ], + "source_type": "inferred", + "source_file": None, + "confidence": 0.7, + }) + + return features + + +def features_to_bulk_create_format(features: list[DetectedFeature]) -> list[dict]: + """ + Convert extracted features to the format expected by feature_create_bulk. + + Removes source_type, source_file, and confidence fields. + Returns a list ready for MCP tool consumption. + + Args: + features: List of DetectedFeature objects + + Returns: + List of dicts with category, name, description, steps + """ + return [ + { + "category": f["category"], + "name": f["name"], + "description": f["description"], + "steps": f["steps"], + } + for f in features + ] + + +def extract_from_project(project_dir: str | Path) -> FeatureExtractionResult: + """ + Convenience function to detect stack and extract features in one step. + + Args: + project_dir: Path to the project directory + + Returns: + FeatureExtractionResult with extracted features + """ + from .stack_detector import StackDetector + + detector = StackDetector(Path(project_dir)) + detection_result = detector.detect() + return extract_features(detection_result) diff --git a/analyzers/node_analyzer.py b/analyzers/node_analyzer.py new file mode 100644 index 00000000..8ceb96e4 --- /dev/null +++ b/analyzers/node_analyzer.py @@ -0,0 +1,352 @@ +""" +Node.js Analyzer +================ + +Detects Node.js/Express/NestJS projects. +Extracts API endpoints from Express router definitions. +""" + +import json +import re +from pathlib import Path + +from .base_analyzer import ( + AnalysisResult, + BaseAnalyzer, + ComponentInfo, + EndpointInfo, + RouteInfo, +) + + +class NodeAnalyzer(BaseAnalyzer): + """Analyzer for Node.js/Express/NestJS projects.""" + + @property + def stack_name(self) -> str: + return self._detected_stack + + def __init__(self, project_dir: Path): + super().__init__(project_dir) + self._detected_stack = "nodejs" # Default, may change to "express" or "nestjs" + + def can_analyze(self) -> tuple[bool, float]: + """Detect if this is a Node.js/Express/NestJS project.""" + confidence = 0.0 + + # Check package.json + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + deps = { + **data.get("dependencies", {}), + **data.get("devDependencies", {}), + } + + # Check for NestJS first (more specific) + if "@nestjs/core" in deps: + self._detected_stack = "nestjs" + confidence = 0.95 + return True, confidence + + # Check for Express + if "express" in deps: + self._detected_stack = "express" + confidence = 0.85 + + # Bonus for having typical Express structure + if (self.project_dir / "routes").exists() or \ + (self.project_dir / "src" / "routes").exists(): + confidence = 0.9 + + return True, confidence + + # Check for Fastify + if "fastify" in deps: + self._detected_stack = "fastify" + confidence = 0.85 + return True, confidence + + # Check for Koa + if "koa" in deps: + self._detected_stack = "koa" + confidence = 0.85 + return True, confidence + + # Generic Node.js (has node-specific files but no specific framework) + if "type" in data and data["type"] == "module": + self._detected_stack = "nodejs" + confidence = 0.5 + return True, confidence + + except (json.JSONDecodeError, OSError): + pass + + # Check for common Node.js files + common_files = ["app.js", "server.js", "index.js", "src/app.js", "src/server.js"] + for file in common_files: + if (self.project_dir / file).exists(): + self._detected_stack = "nodejs" + return True, 0.5 + + return False, 0.0 + + def analyze(self) -> AnalysisResult: + """Analyze the Node.js project.""" + routes: list[RouteInfo] = [] + components: list[ComponentInfo] = [] + endpoints: list[EndpointInfo] = [] + config_files: list[str] = [] + dependencies: dict[str, str] = {} + entry_point: str | None = None + + # Load dependencies from package.json + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + dependencies = { + **data.get("dependencies", {}), + **data.get("devDependencies", {}), + } + + # Detect entry point from package.json + entry_point = data.get("main") + if not entry_point: + scripts = data.get("scripts", {}) + start_script = scripts.get("start", "") + if "node" in start_script: + # Extract file from "node src/index.js" etc. + match = re.search(r"node\s+(\S+)", start_script) + if match: + entry_point = match.group(1) + + except (json.JSONDecodeError, OSError): + pass + + # Collect config files + for config_name in [ + "tsconfig.json", ".eslintrc.js", ".eslintrc.json", + "jest.config.js", "nodemon.json", ".env.example", + ]: + if (self.project_dir / config_name).exists(): + config_files.append(config_name) + + # Detect entry point if not found + if not entry_point: + for candidate in ["src/index.js", "src/index.ts", "src/app.js", "src/app.ts", + "index.js", "app.js", "server.js"]: + if (self.project_dir / candidate).exists(): + entry_point = candidate + break + + # Extract endpoints based on stack type + if self._detected_stack == "express": + endpoints = self._extract_express_routes() + elif self._detected_stack == "nestjs": + endpoints = self._extract_nestjs_routes() + elif self._detected_stack == "fastify": + endpoints = self._extract_fastify_routes() + else: + # Generic Node.js - try Express patterns + endpoints = self._extract_express_routes() + + # Extract middleware/components + components = self._extract_components() + + return { + "stack_name": self._detected_stack, + "confidence": 0.85, + "routes": routes, + "components": components, + "endpoints": endpoints, + "entry_point": entry_point, + "config_files": config_files, + "dependencies": dependencies, + "metadata": { + "has_typescript": "typescript" in dependencies, + "has_prisma": "prisma" in dependencies or "@prisma/client" in dependencies, + "has_mongoose": "mongoose" in dependencies, + "has_sequelize": "sequelize" in dependencies, + }, + } + + def _extract_express_routes(self) -> list[EndpointInfo]: + """Extract routes from Express router definitions.""" + endpoints: list[EndpointInfo] = [] + + # Find route files + route_files = ( + self._find_files("**/routes/**/*.js") + + self._find_files("**/routes/**/*.ts") + + self._find_files("**/router/**/*.js") + + self._find_files("**/router/**/*.ts") + + self._find_files("**/controllers/**/*.js") + + self._find_files("**/controllers/**/*.ts") + ) + + # Also check main files + for main_file in ["app.js", "app.ts", "server.js", "server.ts", + "src/app.js", "src/app.ts", "index.js", "index.ts"]: + main_path = self.project_dir / main_file + if main_path.exists(): + route_files.append(main_path) + + # Pattern for Express routes + # router.get('/path', handler) + # app.post('/path', handler) + route_pattern = re.compile( + r'(?:router|app)\.(get|post|put|patch|delete|all)\s*\(\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + for file in route_files: + content = self._read_file_safe(file) + if content is None: + continue + + for match in route_pattern.finditer(content): + method = match.group(1).upper() + path = match.group(2) + + endpoints.append({ + "path": path, + "method": method, + "handler": "handler", + "file": str(file.relative_to(self.project_dir)), + "description": f"{method} {path}", + }) + + return endpoints + + def _extract_nestjs_routes(self) -> list[EndpointInfo]: + """Extract routes from NestJS controllers.""" + endpoints: list[EndpointInfo] = [] + + # Find controller files + controller_files = ( + self._find_files("**/*.controller.ts") + + self._find_files("**/*.controller.js") + ) + + # Pattern for NestJS decorators + # @Get('/path'), @Post(), etc. + decorator_pattern = re.compile( + r'@(Get|Post|Put|Patch|Delete|All)\s*\(\s*["\']?([^"\')\s]*)["\']?\s*\)', + re.IGNORECASE + ) + + # Pattern for controller path + controller_pattern = re.compile( + r'@Controller\s*\(\s*["\']?([^"\')\s]*)["\']?\s*\)', + re.IGNORECASE + ) + + for file in controller_files: + content = self._read_file_safe(file) + if content is None: + continue + + # Get controller base path + controller_match = controller_pattern.search(content) + base_path = "/" + controller_match.group(1) if controller_match else "" + + for match in decorator_pattern.finditer(content): + method = match.group(1).upper() + path = match.group(2) or "" + + full_path = base_path + if path: + full_path = f"{base_path}/{path}".replace("//", "/") + + endpoints.append({ + "path": full_path or "/", + "method": method, + "handler": "controller", + "file": str(file.relative_to(self.project_dir)), + "description": f"{method} {full_path or '/'}", + }) + + return endpoints + + def _extract_fastify_routes(self) -> list[EndpointInfo]: + """Extract routes from Fastify route definitions.""" + endpoints: list[EndpointInfo] = [] + + # Find route files + route_files = ( + self._find_files("**/routes/**/*.js") + + self._find_files("**/routes/**/*.ts") + + self._find_files("**/*.routes.js") + + self._find_files("**/*.routes.ts") + ) + + # Pattern for Fastify routes + # fastify.get('/path', handler) + route_pattern = re.compile( + r'(?:fastify|server|app)\.(get|post|put|patch|delete|all)\s*\(\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + for file in route_files: + content = self._read_file_safe(file) + if content is None: + continue + + for match in route_pattern.finditer(content): + method = match.group(1).upper() + path = match.group(2) + + endpoints.append({ + "path": path, + "method": method, + "handler": "handler", + "file": str(file.relative_to(self.project_dir)), + "description": f"{method} {path}", + }) + + return endpoints + + def _extract_components(self) -> list[ComponentInfo]: + """Extract middleware and service components.""" + components: list[ComponentInfo] = [] + + # Find middleware files + middleware_files = self._find_files("**/middleware/**/*.js") + \ + self._find_files("**/middleware/**/*.ts") + + for file in middleware_files: + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "middleware", + }) + + # Find service files + service_files = self._find_files("**/services/**/*.js") + \ + self._find_files("**/services/**/*.ts") + \ + self._find_files("**/*.service.js") + \ + self._find_files("**/*.service.ts") + + for file in service_files: + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "service", + }) + + # Find model files + model_files = self._find_files("**/models/**/*.js") + \ + self._find_files("**/models/**/*.ts") + \ + self._find_files("**/*.model.js") + \ + self._find_files("**/*.model.ts") + + for file in model_files: + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "model", + }) + + return components diff --git a/analyzers/python_analyzer.py b/analyzers/python_analyzer.py new file mode 100644 index 00000000..7f50df77 --- /dev/null +++ b/analyzers/python_analyzer.py @@ -0,0 +1,395 @@ +""" +Python Analyzer +=============== + +Detects FastAPI, Django, and Flask projects. +Extracts API endpoints from route/view definitions. +""" + +import re +from pathlib import Path + +from .base_analyzer import ( + AnalysisResult, + BaseAnalyzer, + ComponentInfo, + EndpointInfo, + RouteInfo, +) + + +class PythonAnalyzer(BaseAnalyzer): + """Analyzer for FastAPI, Django, and Flask projects.""" + + @property + def stack_name(self) -> str: + return self._detected_stack + + def __init__(self, project_dir: Path): + super().__init__(project_dir) + self._detected_stack = "python" # Default, may change + + def can_analyze(self) -> tuple[bool, float]: + """Detect if this is a Python web framework project.""" + confidence = 0.0 + + # Check for Django first + if (self.project_dir / "manage.py").exists(): + self._detected_stack = "django" + confidence = 0.95 + return True, confidence + + # Check requirements.txt + requirements = self.project_dir / "requirements.txt" + if requirements.exists(): + try: + content = requirements.read_text().lower() + + if "fastapi" in content: + self._detected_stack = "fastapi" + confidence = 0.9 + return True, confidence + + if "flask" in content: + self._detected_stack = "flask" + confidence = 0.85 + return True, confidence + + if "django" in content: + self._detected_stack = "django" + confidence = 0.85 + return True, confidence + + except OSError: + pass + + # Check pyproject.toml + pyproject = self.project_dir / "pyproject.toml" + if pyproject.exists(): + try: + content = pyproject.read_text().lower() + + if "fastapi" in content: + self._detected_stack = "fastapi" + confidence = 0.9 + return True, confidence + + if "flask" in content: + self._detected_stack = "flask" + confidence = 0.85 + return True, confidence + + if "django" in content: + self._detected_stack = "django" + confidence = 0.85 + return True, confidence + + except OSError: + pass + + # Check for common FastAPI patterns + main_py = self.project_dir / "main.py" + if main_py.exists(): + content = self._read_file_safe(main_py) + if content and "from fastapi import" in content: + self._detected_stack = "fastapi" + return True, 0.9 + + # Check for Flask patterns + app_py = self.project_dir / "app.py" + if app_py.exists(): + content = self._read_file_safe(app_py) + if content and "from flask import" in content: + self._detected_stack = "flask" + return True, 0.85 + + return False, 0.0 + + def analyze(self) -> AnalysisResult: + """Analyze the Python project.""" + routes: list[RouteInfo] = [] + components: list[ComponentInfo] = [] + endpoints: list[EndpointInfo] = [] + config_files: list[str] = [] + dependencies: dict[str, str] = {} + entry_point: str | None = None + + # Load dependencies from requirements.txt + requirements = self.project_dir / "requirements.txt" + if requirements.exists(): + try: + for line in requirements.read_text().splitlines(): + line = line.strip() + if line and not line.startswith("#"): + # Parse package==version or package>=version etc. + match = re.match(r"([a-zA-Z0-9_-]+)(?:[=<>!~]+(.+))?", line) + if match: + dependencies[match.group(1)] = match.group(2) or "*" + except OSError: + pass + + # Collect config files + for config_name in [ + "pyproject.toml", "setup.py", "setup.cfg", + "requirements.txt", "requirements-dev.txt", + ".env.example", "alembic.ini", "pytest.ini", + ]: + if (self.project_dir / config_name).exists(): + config_files.append(config_name) + + # Extract endpoints based on framework + if self._detected_stack == "fastapi": + endpoints = self._extract_fastapi_routes() + entry_point = "main.py" + elif self._detected_stack == "django": + endpoints = self._extract_django_routes() + entry_point = "manage.py" + elif self._detected_stack == "flask": + endpoints = self._extract_flask_routes() + entry_point = "app.py" + + # Find entry point if not set + if not entry_point or not (self.project_dir / entry_point).exists(): + for candidate in ["main.py", "app.py", "server.py", "run.py", "src/main.py"]: + if (self.project_dir / candidate).exists(): + entry_point = candidate + break + + # Extract components (models, services, etc.) + components = self._extract_components() + + return { + "stack_name": self._detected_stack, + "confidence": 0.85, + "routes": routes, + "components": components, + "endpoints": endpoints, + "entry_point": entry_point, + "config_files": config_files, + "dependencies": dependencies, + "metadata": { + "has_sqlalchemy": "sqlalchemy" in dependencies, + "has_alembic": "alembic" in dependencies, + "has_pytest": "pytest" in dependencies, + "has_celery": "celery" in dependencies, + }, + } + + def _extract_fastapi_routes(self) -> list[EndpointInfo]: + """Extract routes from FastAPI decorators.""" + endpoints: list[EndpointInfo] = [] + + # Find Python files + py_files = self._find_files("**/*.py") + + # Pattern for FastAPI routes + # @app.get("/path") + # @router.post("/path") + route_pattern = re.compile( + r'@(?:app|router)\.(get|post|put|patch|delete)\s*\(\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + # Pattern for APIRouter prefix + router_prefix_pattern = re.compile( + r'APIRouter\s*\([^)]*prefix\s*=\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + for file in py_files: + content = self._read_file_safe(file) + if content is None: + continue + + # Skip if not a route file + if "@app." not in content and "@router." not in content: + continue + + # Try to find router prefix + prefix = "" + prefix_match = router_prefix_pattern.search(content) + if prefix_match: + prefix = prefix_match.group(1) + + for match in route_pattern.finditer(content): + method = match.group(1).upper() + path = match.group(2) + + full_path = prefix + path if prefix else path + + endpoints.append({ + "path": full_path, + "method": method, + "handler": "handler", + "file": str(file.relative_to(self.project_dir)), + "description": f"{method} {full_path}", + }) + + return endpoints + + def _extract_django_routes(self) -> list[EndpointInfo]: + """Extract routes from Django URL patterns.""" + endpoints: list[EndpointInfo] = [] + + # Find urls.py files + url_files = self._find_files("**/urls.py") + + # Pattern for Django URL patterns + # path('api/users/', views.user_list) + # path('api/users//', views.user_detail) + path_pattern = re.compile( + r'path\s*\(\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + # Pattern for re_path + re_path_pattern = re.compile( + r're_path\s*\(\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + for file in url_files: + content = self._read_file_safe(file) + if content is None: + continue + + for match in path_pattern.finditer(content): + path = "/" + match.group(1).rstrip("/") + if path == "/": + path = "/" + + # Django uses for params, convert to :name + path = re.sub(r"<\w+:(\w+)>", r":\1", path) + path = re.sub(r"<(\w+)>", r":\1", path) + + endpoints.append({ + "path": path, + "method": "ALL", # Django views typically handle multiple methods + "handler": "view", + "file": str(file.relative_to(self.project_dir)), + "description": f"Django view at {path}", + }) + + for match in re_path_pattern.finditer(content): + # re_path uses regex, just record the pattern + path = "/" + match.group(1) + + endpoints.append({ + "path": path, + "method": "ALL", + "handler": "view", + "file": str(file.relative_to(self.project_dir)), + "description": "Django regex route", + }) + + return endpoints + + def _extract_flask_routes(self) -> list[EndpointInfo]: + """Extract routes from Flask decorators.""" + endpoints: list[EndpointInfo] = [] + + # Find Python files + py_files = self._find_files("**/*.py") + + # Pattern for Flask routes + # @app.route('/path', methods=['GET', 'POST']) + # @bp.route('/path') + route_pattern = re.compile( + r'@(?:app|bp|blueprint)\s*\.\s*route\s*\(\s*["\']([^"\']+)["\'](?:\s*,\s*methods\s*=\s*\[([^\]]+)\])?', + re.IGNORECASE + ) + + # Pattern for Blueprint prefix + blueprint_pattern = re.compile( + r'Blueprint\s*\(\s*[^,]+\s*,\s*[^,]+\s*(?:,\s*url_prefix\s*=\s*["\']([^"\']+)["\'])?', + re.IGNORECASE + ) + + for file in py_files: + content = self._read_file_safe(file) + if content is None: + continue + + # Skip if not a route file + if "@app." not in content and "@bp." not in content and "@blueprint" not in content.lower(): + continue + + # Try to find blueprint prefix + prefix = "" + prefix_match = blueprint_pattern.search(content) + if prefix_match and prefix_match.group(1): + prefix = prefix_match.group(1) + + for match in route_pattern.finditer(content): + path = match.group(1) + methods_str = match.group(2) + + full_path = prefix + path if prefix else path + + # Parse methods + methods = ["GET"] # Default + if methods_str: + # Parse ['GET', 'POST'] format + methods = re.findall(r"['\"](\w+)['\"]", methods_str) + + for method in methods: + endpoints.append({ + "path": full_path, + "method": method.upper(), + "handler": "view", + "file": str(file.relative_to(self.project_dir)), + "description": f"{method.upper()} {full_path}", + }) + + return endpoints + + def _extract_components(self) -> list[ComponentInfo]: + """Extract models, services, and other components.""" + components: list[ComponentInfo] = [] + + # Find model files + model_files = ( + self._find_files("**/models.py") + + self._find_files("**/models/**/*.py") + + self._find_files("**/*_model.py") + ) + + for file in model_files: + if file.name != "__init__.py": + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "model", + }) + + # Find view/controller files + view_files = ( + self._find_files("**/views.py") + + self._find_files("**/views/**/*.py") + + self._find_files("**/routers/**/*.py") + + self._find_files("**/api/**/*.py") + ) + + for file in view_files: + if file.name != "__init__.py": + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "view", + }) + + # Find service files + service_files = ( + self._find_files("**/services/**/*.py") + + self._find_files("**/*_service.py") + ) + + for file in service_files: + if file.name != "__init__.py": + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "service", + }) + + return components diff --git a/analyzers/react_analyzer.py b/analyzers/react_analyzer.py new file mode 100644 index 00000000..9d125e3c --- /dev/null +++ b/analyzers/react_analyzer.py @@ -0,0 +1,418 @@ +""" +React Analyzer +============== + +Detects React, Vite, and Next.js projects. +Extracts routes from React Router and Next.js file-based routing. +""" + +import json +import re +from pathlib import Path + +from .base_analyzer import ( + AnalysisResult, + BaseAnalyzer, + ComponentInfo, + EndpointInfo, + RouteInfo, +) + + +class ReactAnalyzer(BaseAnalyzer): + """Analyzer for React, Vite, and Next.js projects.""" + + @property + def stack_name(self) -> str: + return self._detected_stack + + def __init__(self, project_dir: Path): + super().__init__(project_dir) + self._detected_stack = "react" # Default, may change to "nextjs" + + def can_analyze(self) -> tuple[bool, float]: + """Detect if this is a React/Next.js project.""" + confidence = 0.0 + + # Check package.json + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + deps = { + **data.get("dependencies", {}), + **data.get("devDependencies", {}), + } + + # Check for Next.js first (more specific) + if "next" in deps: + self._detected_stack = "nextjs" + confidence = 0.95 + return True, confidence + + # Check for React + if "react" in deps: + confidence = 0.85 + + # Check for Vite + if "vite" in deps: + self._detected_stack = "react-vite" + confidence = 0.9 + + # Check for Create React App + if "react-scripts" in deps: + self._detected_stack = "react-cra" + confidence = 0.9 + + return True, confidence + + except (json.JSONDecodeError, OSError): + pass + + # Check for Next.js config + if (self.project_dir / "next.config.js").exists() or \ + (self.project_dir / "next.config.mjs").exists() or \ + (self.project_dir / "next.config.ts").exists(): + self._detected_stack = "nextjs" + return True, 0.95 + + # Check for common React files + if (self.project_dir / "src" / "App.tsx").exists() or \ + (self.project_dir / "src" / "App.jsx").exists(): + return True, 0.7 + + return False, 0.0 + + def analyze(self) -> AnalysisResult: + """Analyze the React/Next.js project.""" + routes: list[RouteInfo] = [] + components: list[ComponentInfo] = [] + endpoints: list[EndpointInfo] = [] + config_files: list[str] = [] + dependencies: dict[str, str] = {} + entry_point: str | None = None + + # Load dependencies from package.json + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + dependencies = { + **data.get("dependencies", {}), + **data.get("devDependencies", {}), + } + except (json.JSONDecodeError, OSError): + pass + + # Collect config files + for config_name in [ + "next.config.js", "next.config.mjs", "next.config.ts", + "vite.config.js", "vite.config.ts", + "tsconfig.json", "tailwind.config.js", "tailwind.config.ts", + ]: + if (self.project_dir / config_name).exists(): + config_files.append(config_name) + + # Detect entry point + for entry in ["src/main.tsx", "src/main.jsx", "src/index.tsx", "src/index.jsx", "pages/_app.tsx", "app/layout.tsx"]: + if (self.project_dir / entry).exists(): + entry_point = entry + break + + # Extract routes based on stack type + if self._detected_stack == "nextjs": + routes = self._extract_nextjs_routes() + endpoints = self._extract_nextjs_api_routes() + else: + routes = self._extract_react_router_routes() + + # Extract components + components = self._extract_components() + + return { + "stack_name": self._detected_stack, + "confidence": 0.9, + "routes": routes, + "components": components, + "endpoints": endpoints, + "entry_point": entry_point, + "config_files": config_files, + "dependencies": dependencies, + "metadata": { + "has_typescript": "typescript" in dependencies, + "has_tailwind": "tailwindcss" in dependencies, + "has_react_router": "react-router-dom" in dependencies, + }, + } + + def _extract_nextjs_routes(self) -> list[RouteInfo]: + """Extract routes from Next.js file-based routing.""" + routes: list[RouteInfo] = [] + + # Check for App Router (Next.js 13+) + app_dir = self.project_dir / "app" + if app_dir.exists(): + routes.extend(self._extract_app_router_routes(app_dir)) + + # Check for Pages Router + pages_dir = self.project_dir / "pages" + if pages_dir.exists(): + routes.extend(self._extract_pages_router_routes(pages_dir)) + + # Also check src/app and src/pages + src_app = self.project_dir / "src" / "app" + if src_app.exists(): + routes.extend(self._extract_app_router_routes(src_app)) + + src_pages = self.project_dir / "src" / "pages" + if src_pages.exists(): + routes.extend(self._extract_pages_router_routes(src_pages)) + + return routes + + def _extract_app_router_routes(self, app_dir: Path) -> list[RouteInfo]: + """Extract routes from Next.js App Router.""" + routes: list[RouteInfo] = [] + + for page_file in app_dir.rglob("page.tsx"): + rel_path = page_file.relative_to(app_dir) + route_path = "/" + "/".join(rel_path.parent.parts) + + # Handle dynamic routes: [id] -> :id + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + + # Clean up + if route_path == "/.": + route_path = "/" + route_path = route_path.replace("//", "/") + + routes.append({ + "path": route_path, + "method": "GET", + "handler": "Page", + "file": str(page_file.relative_to(self.project_dir)), + }) + + # Also check .jsx files + for page_file in app_dir.rglob("page.jsx"): + rel_path = page_file.relative_to(app_dir) + route_path = "/" + "/".join(rel_path.parent.parts) + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + if route_path == "/.": + route_path = "/" + route_path = route_path.replace("//", "/") + + routes.append({ + "path": route_path, + "method": "GET", + "handler": "Page", + "file": str(page_file.relative_to(self.project_dir)), + }) + + return routes + + def _extract_pages_router_routes(self, pages_dir: Path) -> list[RouteInfo]: + """Extract routes from Next.js Pages Router.""" + routes: list[RouteInfo] = [] + + for page_file in pages_dir.rglob("*.tsx"): + if page_file.name.startswith("_"): # Skip _app.tsx, _document.tsx + continue + if "api" in page_file.parts: # Skip API routes + continue + + rel_path = page_file.relative_to(pages_dir) + route_path = "/" + str(rel_path.with_suffix("")) + + # Handle index files + route_path = route_path.replace("/index", "") + if not route_path: + route_path = "/" + + # Handle dynamic routes + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + + routes.append({ + "path": route_path, + "method": "GET", + "handler": page_file.stem, + "file": str(page_file.relative_to(self.project_dir)), + }) + + # Also check .jsx files + for page_file in pages_dir.rglob("*.jsx"): + if page_file.name.startswith("_"): + continue + if "api" in page_file.parts: + continue + + rel_path = page_file.relative_to(pages_dir) + route_path = "/" + str(rel_path.with_suffix("")) + route_path = route_path.replace("/index", "") + if not route_path: + route_path = "/" + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + + routes.append({ + "path": route_path, + "method": "GET", + "handler": page_file.stem, + "file": str(page_file.relative_to(self.project_dir)), + }) + + return routes + + def _extract_nextjs_api_routes(self) -> list[EndpointInfo]: + """Extract API routes from Next.js.""" + endpoints: list[EndpointInfo] = [] + + # Check pages/api (Pages Router) + api_dirs = [ + self.project_dir / "pages" / "api", + self.project_dir / "src" / "pages" / "api", + ] + + for api_dir in api_dirs: + if api_dir.exists(): + for api_file in api_dir.rglob("*.ts"): + endpoints.extend(self._parse_api_route(api_file, api_dir)) + for api_file in api_dir.rglob("*.js"): + endpoints.extend(self._parse_api_route(api_file, api_dir)) + + # Check app/api (App Router - route.ts files) + app_api_dirs = [ + self.project_dir / "app" / "api", + self.project_dir / "src" / "app" / "api", + ] + + for app_api in app_api_dirs: + if app_api.exists(): + for route_file in app_api.rglob("route.ts"): + endpoints.extend(self._parse_app_router_api(route_file, app_api)) + for route_file in app_api.rglob("route.js"): + endpoints.extend(self._parse_app_router_api(route_file, app_api)) + + return endpoints + + def _parse_api_route(self, api_file: Path, api_dir: Path) -> list[EndpointInfo]: + """Parse a Pages Router API route file.""" + rel_path = api_file.relative_to(api_dir) + route_path = "/api/" + str(rel_path.with_suffix("")) + route_path = route_path.replace("/index", "") + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + + return [{ + "path": route_path, + "method": "ALL", # Default export handles all methods + "handler": "handler", + "file": str(api_file.relative_to(self.project_dir)), + "description": f"API endpoint at {route_path}", + }] + + def _parse_app_router_api(self, route_file: Path, api_dir: Path) -> list[EndpointInfo]: + """Parse an App Router API route file.""" + rel_path = route_file.relative_to(api_dir) + route_path = "/api/" + "/".join(rel_path.parent.parts) + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + if route_path.endswith("/"): + route_path = route_path[:-1] + + # Try to detect which methods are exported + content = self._read_file_safe(route_file) + methods = [] + if content: + for method in ["GET", "POST", "PUT", "PATCH", "DELETE"]: + if f"export async function {method}" in content or \ + f"export function {method}" in content: + methods.append(method) + + if not methods: + methods = ["ALL"] + + return [ + { + "path": route_path, + "method": method, + "handler": method, + "file": str(route_file.relative_to(self.project_dir)), + "description": f"{method} {route_path}", + } + for method in methods + ] + + def _extract_react_router_routes(self) -> list[RouteInfo]: + """Extract routes from React Router configuration.""" + routes: list[RouteInfo] = [] + + # Look for route definitions in common files + route_files = self._find_files("**/*.tsx") + self._find_files("**/*.jsx") + + # Pattern for React Router elements + route_pattern = re.compile( + r']*path=["\']([^"\']+)["\'][^>]*>', + re.IGNORECASE + ) + + # Pattern for createBrowserRouter routes + browser_router_pattern = re.compile( + r'{\s*path:\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + for file in route_files: + content = self._read_file_safe(file) + if content is None: + continue + + # Skip if not likely a routing file + if "Route" not in content and "createBrowserRouter" not in content: + continue + + # Extract routes from JSX + for match in route_pattern.finditer(content): + routes.append({ + "path": match.group(1), + "method": "GET", + "handler": "Route", + "file": str(file.relative_to(self.project_dir)), + }) + + # Extract routes from createBrowserRouter + for match in browser_router_pattern.finditer(content): + routes.append({ + "path": match.group(1), + "method": "GET", + "handler": "RouterRoute", + "file": str(file.relative_to(self.project_dir)), + }) + + return routes + + def _extract_components(self) -> list[ComponentInfo]: + """Extract React components.""" + components: list[ComponentInfo] = [] + + # Find component files + component_files = self._find_files("**/components/**/*.tsx") + \ + self._find_files("**/components/**/*.jsx") + + for file in component_files: + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "component", + }) + + # Find page files + page_files = self._find_files("**/pages/**/*.tsx") + \ + self._find_files("**/pages/**/*.jsx") + + for file in page_files: + if not file.name.startswith("_"): + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "page", + }) + + return components diff --git a/analyzers/stack_detector.py b/analyzers/stack_detector.py new file mode 100644 index 00000000..09bed2d2 --- /dev/null +++ b/analyzers/stack_detector.py @@ -0,0 +1,216 @@ +""" +Stack Detector +============== + +Orchestrates detection of tech stacks in a codebase. +Uses multiple analyzers to detect frontend, backend, and database technologies. +""" + +import json +from pathlib import Path +from typing import TypedDict + +from .base_analyzer import AnalysisResult + + +class StackInfo(TypedDict): + """Information about a detected stack.""" + name: str + category: str # frontend, backend, database, other + confidence: float + analysis: AnalysisResult | None + + +class StackDetectionResult(TypedDict): + """Complete result of stack detection.""" + project_dir: str + detected_stacks: list[StackInfo] + primary_frontend: str | None + primary_backend: str | None + database: str | None + routes_count: int + components_count: int + endpoints_count: int + all_routes: list[dict] + all_endpoints: list[dict] + all_components: list[dict] + summary: str + + +class StackDetector: + """ + Detects tech stacks in a codebase by running multiple analyzers. + + Usage: + detector = StackDetector(project_dir) + result = detector.detect() + """ + + def __init__(self, project_dir: Path): + """ + Initialize the stack detector. + + Args: + project_dir: Path to the project directory to analyze + """ + self.project_dir = Path(project_dir).resolve() + self._analyzers = [] + self._load_analyzers() + + def _load_analyzers(self) -> None: + """Load all available analyzers.""" + # Import analyzers here to avoid circular imports + from .node_analyzer import NodeAnalyzer + from .python_analyzer import PythonAnalyzer + from .react_analyzer import ReactAnalyzer + from .vue_analyzer import VueAnalyzer + + # Order matters: more specific analyzers first (Next.js before React) + self._analyzers = [ + ReactAnalyzer(self.project_dir), + VueAnalyzer(self.project_dir), + NodeAnalyzer(self.project_dir), + PythonAnalyzer(self.project_dir), + ] + + def detect(self) -> StackDetectionResult: + """ + Run all analyzers and compile results. + + Returns: + StackDetectionResult with all detected stacks and extracted information + """ + detected_stacks: list[StackInfo] = [] + all_routes: list[dict] = [] + all_endpoints: list[dict] = [] + all_components: list[dict] = [] + + for analyzer in self._analyzers: + can_analyze, confidence = analyzer.can_analyze() + + if can_analyze and confidence > 0.3: # Minimum confidence threshold + try: + analysis = analyzer.analyze() + + # Determine category + stack_name = analyzer.stack_name.lower() + if stack_name in ("react", "nextjs", "vue", "nuxt", "angular"): + category = "frontend" + elif stack_name in ("express", "fastapi", "django", "flask", "nestjs"): + category = "backend" + elif stack_name in ("postgres", "mysql", "mongodb", "sqlite"): + category = "database" + else: + category = "other" + + detected_stacks.append({ + "name": analyzer.stack_name, + "category": category, + "confidence": confidence, + "analysis": analysis, + }) + + # Collect all routes, endpoints, components + all_routes.extend(analysis.get("routes", [])) + all_endpoints.extend(analysis.get("endpoints", [])) + all_components.extend(analysis.get("components", [])) + + except Exception as e: + # Log but don't fail - continue with other analyzers + print(f"Warning: {analyzer.stack_name} analyzer failed: {e}") + + # Sort by confidence + detected_stacks.sort(key=lambda x: x["confidence"], reverse=True) + + # Determine primary frontend and backend + primary_frontend = None + primary_backend = None + database = None + + for stack in detected_stacks: + if stack["category"] == "frontend" and primary_frontend is None: + primary_frontend = stack["name"] + elif stack["category"] == "backend" and primary_backend is None: + primary_backend = stack["name"] + elif stack["category"] == "database" and database is None: + database = stack["name"] + + # Build summary + stack_names = [s["name"] for s in detected_stacks] + if stack_names: + summary = f"Detected: {', '.join(stack_names)}" + else: + summary = "No recognized tech stack detected" + + if all_routes: + summary += f" | {len(all_routes)} routes" + if all_endpoints: + summary += f" | {len(all_endpoints)} endpoints" + if all_components: + summary += f" | {len(all_components)} components" + + return { + "project_dir": str(self.project_dir), + "detected_stacks": detected_stacks, + "primary_frontend": primary_frontend, + "primary_backend": primary_backend, + "database": database, + "routes_count": len(all_routes), + "components_count": len(all_components), + "endpoints_count": len(all_endpoints), + "all_routes": all_routes, + "all_endpoints": all_endpoints, + "all_components": all_components, + "summary": summary, + } + + def detect_quick(self) -> dict: + """ + Quick detection without full analysis. + + Returns a simplified result with just stack names and confidence. + Useful for UI display before full analysis. + """ + results = [] + + for analyzer in self._analyzers: + can_analyze, confidence = analyzer.can_analyze() + if can_analyze and confidence > 0.3: + results.append({ + "name": analyzer.stack_name, + "confidence": confidence, + }) + + results.sort(key=lambda x: x["confidence"], reverse=True) + + return { + "project_dir": str(self.project_dir), + "stacks": results, + "primary": results[0]["name"] if results else None, + } + + def to_json(self, result: StackDetectionResult) -> str: + """Convert detection result to JSON string.""" + # Remove analysis objects for cleaner output + clean_result = { + **result, + "detected_stacks": [ + {k: v for k, v in stack.items() if k != "analysis"} + for stack in result["detected_stacks"] + ], + } + return json.dumps(clean_result, indent=2) + + +def detect_stack(project_dir: str | Path) -> StackDetectionResult: + """ + Convenience function to detect stack in a project. + + Args: + project_dir: Path to the project directory + + Returns: + StackDetectionResult + """ + detector = StackDetector(Path(project_dir)) + return detector.detect() diff --git a/analyzers/vue_analyzer.py b/analyzers/vue_analyzer.py new file mode 100644 index 00000000..75b3ae41 --- /dev/null +++ b/analyzers/vue_analyzer.py @@ -0,0 +1,319 @@ +""" +Vue.js Analyzer +=============== + +Detects Vue.js and Nuxt.js projects. +Extracts routes from Vue Router and Nuxt file-based routing. +""" + +import json +import re +from pathlib import Path + +from .base_analyzer import ( + AnalysisResult, + BaseAnalyzer, + ComponentInfo, + EndpointInfo, + RouteInfo, +) + + +class VueAnalyzer(BaseAnalyzer): + """Analyzer for Vue.js and Nuxt.js projects.""" + + @property + def stack_name(self) -> str: + return self._detected_stack + + def __init__(self, project_dir: Path): + super().__init__(project_dir) + self._detected_stack = "vue" # Default, may change to "nuxt" + + def can_analyze(self) -> tuple[bool, float]: + """Detect if this is a Vue.js/Nuxt.js project.""" + confidence = 0.0 + + # Check package.json + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + deps = { + **data.get("dependencies", {}), + **data.get("devDependencies", {}), + } + + # Check for Nuxt first (more specific) + if "nuxt" in deps or "nuxt3" in deps: + self._detected_stack = "nuxt" + confidence = 0.95 + return True, confidence + + # Check for Vue + if "vue" in deps: + confidence = 0.85 + + # Check for Vite + if "vite" in deps: + self._detected_stack = "vue-vite" + confidence = 0.9 + + # Check for Vue CLI + if "@vue/cli-service" in deps: + self._detected_stack = "vue-cli" + confidence = 0.9 + + return True, confidence + + except (json.JSONDecodeError, OSError): + pass + + # Check for Nuxt config + if (self.project_dir / "nuxt.config.js").exists() or \ + (self.project_dir / "nuxt.config.ts").exists(): + self._detected_stack = "nuxt" + return True, 0.95 + + # Check for common Vue files + if (self.project_dir / "src" / "App.vue").exists(): + return True, 0.7 + + return False, 0.0 + + def analyze(self) -> AnalysisResult: + """Analyze the Vue.js/Nuxt.js project.""" + routes: list[RouteInfo] = [] + components: list[ComponentInfo] = [] + endpoints: list[EndpointInfo] = [] + config_files: list[str] = [] + dependencies: dict[str, str] = {} + entry_point: str | None = None + + # Load dependencies from package.json + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + dependencies = { + **data.get("dependencies", {}), + **data.get("devDependencies", {}), + } + except (json.JSONDecodeError, OSError): + pass + + # Collect config files + for config_name in [ + "nuxt.config.js", "nuxt.config.ts", + "vite.config.js", "vite.config.ts", + "vue.config.js", "tsconfig.json", + "tailwind.config.js", "tailwind.config.ts", + ]: + if (self.project_dir / config_name).exists(): + config_files.append(config_name) + + # Detect entry point + for entry in ["src/main.ts", "src/main.js", "app.vue", "src/App.vue"]: + if (self.project_dir / entry).exists(): + entry_point = entry + break + + # Extract routes based on stack type + if self._detected_stack == "nuxt": + routes = self._extract_nuxt_routes() + endpoints = self._extract_nuxt_api_routes() + else: + routes = self._extract_vue_router_routes() + + # Extract components + components = self._extract_components() + + return { + "stack_name": self._detected_stack, + "confidence": 0.85, + "routes": routes, + "components": components, + "endpoints": endpoints, + "entry_point": entry_point, + "config_files": config_files, + "dependencies": dependencies, + "metadata": { + "has_typescript": "typescript" in dependencies, + "has_tailwind": "tailwindcss" in dependencies, + "has_vue_router": "vue-router" in dependencies, + "has_pinia": "pinia" in dependencies, + "has_vuex": "vuex" in dependencies, + }, + } + + def _extract_nuxt_routes(self) -> list[RouteInfo]: + """Extract routes from Nuxt file-based routing.""" + routes: list[RouteInfo] = [] + + # Check for pages directory + pages_dirs = [ + self.project_dir / "pages", + self.project_dir / "src" / "pages", + ] + + for pages_dir in pages_dirs: + if pages_dir.exists(): + routes.extend(self._extract_pages_routes(pages_dir)) + + return routes + + def _extract_pages_routes(self, pages_dir: Path) -> list[RouteInfo]: + """Extract routes from Nuxt pages directory.""" + routes: list[RouteInfo] = [] + + for page_file in pages_dir.rglob("*.vue"): + rel_path = page_file.relative_to(pages_dir) + route_path = "/" + str(rel_path.with_suffix("")) + + # Handle index files + route_path = route_path.replace("/index", "") + if not route_path: + route_path = "/" + + # Handle dynamic routes: [id].vue or _id.vue -> :id + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + route_path = re.sub(r"/_([^/]+)", r"/:\1", route_path) + + routes.append({ + "path": route_path, + "method": "GET", + "handler": page_file.stem, + "file": str(page_file.relative_to(self.project_dir)), + }) + + return routes + + def _extract_nuxt_api_routes(self) -> list[EndpointInfo]: + """Extract API routes from Nuxt server directory.""" + endpoints: list[EndpointInfo] = [] + + # Nuxt 3 uses server/api directory + api_dirs = [ + self.project_dir / "server" / "api", + self.project_dir / "server" / "routes", + ] + + for api_dir in api_dirs: + if not api_dir.exists(): + continue + + for api_file in api_dir.rglob("*.ts"): + rel_path = api_file.relative_to(api_dir) + route_path = "/api/" + str(rel_path.with_suffix("")) + + # Handle index files + route_path = route_path.replace("/index", "") + + # Handle dynamic routes + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + + # Try to detect method from filename + method = "ALL" + for m in ["get", "post", "put", "patch", "delete"]: + if api_file.stem.endswith(f".{m}") or api_file.stem == m: + method = m.upper() + route_path = route_path.replace(f".{m}", "") + break + + endpoints.append({ + "path": route_path, + "method": method, + "handler": "handler", + "file": str(api_file.relative_to(self.project_dir)), + "description": f"{method} {route_path}", + }) + + # Also check .js files + for api_file in api_dir.rglob("*.js"): + rel_path = api_file.relative_to(api_dir) + route_path = "/api/" + str(rel_path.with_suffix("")) + route_path = route_path.replace("/index", "") + route_path = re.sub(r"\[([^\]]+)\]", r":\1", route_path) + + endpoints.append({ + "path": route_path, + "method": "ALL", + "handler": "handler", + "file": str(api_file.relative_to(self.project_dir)), + "description": f"API endpoint at {route_path}", + }) + + return endpoints + + def _extract_vue_router_routes(self) -> list[RouteInfo]: + """Extract routes from Vue Router configuration.""" + routes: list[RouteInfo] = [] + + # Look for router configuration files + router_files = ( + self._find_files("**/router/**/*.js") + + self._find_files("**/router/**/*.ts") + + self._find_files("**/router.js") + + self._find_files("**/router.ts") + + self._find_files("**/routes.js") + + self._find_files("**/routes.ts") + ) + + # Pattern for Vue Router routes + # { path: '/about', ... } + route_pattern = re.compile( + r'{\s*path:\s*["\']([^"\']+)["\']', + re.IGNORECASE + ) + + for file in router_files: + content = self._read_file_safe(file) + if content is None: + continue + + for match in route_pattern.finditer(content): + routes.append({ + "path": match.group(1), + "method": "GET", + "handler": "RouterRoute", + "file": str(file.relative_to(self.project_dir)), + }) + + return routes + + def _extract_components(self) -> list[ComponentInfo]: + """Extract Vue components.""" + components: list[ComponentInfo] = [] + + # Find component files + component_files = ( + self._find_files("**/components/**/*.vue") + + self._find_files("**/views/**/*.vue") + ) + + for file in component_files: + # Determine component type + if "views" in file.parts: + comp_type = "view" + elif "layouts" in file.parts: + comp_type = "layout" + else: + comp_type = "component" + + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": comp_type, + }) + + # Find page files (Nuxt) + page_files = self._find_files("**/pages/**/*.vue") + + for file in page_files: + components.append({ + "name": file.stem, + "file": str(file.relative_to(self.project_dir)), + "type": "page", + }) + + return components diff --git a/api/database.py b/api/database.py index f3a0cce0..aa58da4f 100644 --- a/api/database.py +++ b/api/database.py @@ -396,3 +396,210 @@ def get_db() -> Session: yield db finally: db.close() + + +# ============================================================================= +# Atomic Transaction Helpers for Parallel Mode +# ============================================================================= +# These helpers prevent database corruption when multiple processes access the +# same SQLite database concurrently. They use IMMEDIATE transactions which +# acquire write locks at the start (preventing stale reads) and atomic +# UPDATE ... WHERE clauses (preventing check-then-modify races). + + +from contextlib import contextmanager + + +@contextmanager +def atomic_transaction(session_maker, isolation_level: str = "IMMEDIATE"): + """Context manager for atomic SQLite transactions. + + Uses BEGIN IMMEDIATE to acquire a write lock immediately, preventing + stale reads in read-modify-write patterns. This is essential for + preventing race conditions in parallel mode. + + Args: + session_maker: SQLAlchemy sessionmaker + isolation_level: "IMMEDIATE" (default) or "EXCLUSIVE" + - IMMEDIATE: Acquires write lock at transaction start + - EXCLUSIVE: Also blocks other readers (rarely needed) + + Yields: + SQLAlchemy session with automatic commit/rollback + + Example: + with atomic_transaction(session_maker) as session: + # All reads in this block are protected by write lock + feature = session.query(Feature).filter(...).first() + feature.priority = new_priority + # Commit happens automatically on exit + """ + session = session_maker() + try: + # Start transaction with write lock + session.execute(text(f"BEGIN {isolation_level}")) + yield session + session.commit() + except Exception: + session.rollback() + raise + finally: + session.close() + + +def atomic_claim_feature(session_maker, feature_id: int) -> dict: + """Atomically claim a feature for implementation. + + Uses atomic UPDATE ... WHERE to prevent race conditions where two agents + try to claim the same feature simultaneously. + + Args: + session_maker: SQLAlchemy sessionmaker + feature_id: ID of the feature to claim + + Returns: + Dict with: + - success: True if claimed, False if already claimed/passing/not found + - feature: Feature dict if claimed successfully + - error: Error message if not claimed + """ + session = session_maker() + try: + # Atomic claim: only succeeds if feature is not already claimed or passing + result = session.execute(text(""" + UPDATE features + SET in_progress = 1 + WHERE id = :id AND passes = 0 AND in_progress = 0 + """), {"id": feature_id}) + session.commit() + + if result.rowcount == 0: + # Check why the claim failed + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if feature is None: + return {"success": False, "error": f"Feature {feature_id} not found"} + if feature.passes: + return {"success": False, "error": f"Feature {feature_id} already passing"} + if feature.in_progress: + return {"success": False, "error": f"Feature {feature_id} already in progress"} + return {"success": False, "error": "Claim failed for unknown reason"} + + # Fetch the claimed feature + feature = session.query(Feature).filter(Feature.id == feature_id).first() + return {"success": True, "feature": feature.to_dict()} + finally: + session.close() + + +def atomic_mark_passing(session_maker, feature_id: int) -> dict: + """Atomically mark a feature as passing. + + Uses atomic UPDATE with BEGIN IMMEDIATE to ensure consistency and + prevent race conditions in parallel mode. + + Args: + session_maker: SQLAlchemy sessionmaker + feature_id: ID of the feature to mark passing + + Returns: + Dict with success status and feature name + """ + try: + with atomic_transaction(session_maker) as session: + # Get the feature name for the response (protected by write lock) + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if feature is None: + return {"success": False, "error": f"Feature {feature_id} not found"} + + name = feature.name + + # Atomic update + session.execute(text(""" + UPDATE features + SET passes = 1, in_progress = 0 + WHERE id = :id + """), {"id": feature_id}) + + return {"success": True, "feature_id": feature_id, "name": name} + except Exception as e: + return {"success": False, "error": str(e)} + + +def atomic_update_priority_to_end(session_maker, feature_id: int) -> dict: + """Atomically move a feature to the end of the queue. + + Uses BEGIN IMMEDIATE and a subquery to atomically calculate MAX(priority) + 1 + and update in a single statement, preventing race conditions where two features + get the same priority. + + Args: + session_maker: SQLAlchemy sessionmaker + feature_id: ID of the feature to move + + Returns: + Dict with old_priority and new_priority + """ + try: + with atomic_transaction(session_maker) as session: + # Get current state (protected by write lock) + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if feature is None: + return {"success": False, "error": f"Feature {feature_id} not found"} + if feature.passes: + return {"success": False, "error": "Cannot skip a feature that is already passing"} + + old_priority = feature.priority + name = feature.name + + # Atomic update: set priority to max+1 in a single statement + session.execute(text(""" + UPDATE features + SET priority = (SELECT COALESCE(MAX(priority), 0) + 1 FROM features), + in_progress = 0 + WHERE id = :id + """), {"id": feature_id}) + + # Flush to ensure update is visible, then fetch new priority + session.flush() + result = session.execute( + text("SELECT priority FROM features WHERE id = :id"), + {"id": feature_id} + ).fetchone() + new_priority = result[0] if result else old_priority + 1 + + return { + "success": True, + "id": feature_id, + "name": name, + "old_priority": old_priority, + "new_priority": new_priority, + } + except Exception as e: + return {"success": False, "error": str(e)} + + +def atomic_get_next_priority(session_maker) -> int: + """Atomically get the next available priority. + + Uses BEGIN IMMEDIATE to ensure consistent reads under concurrent access. + + Args: + session_maker: SQLAlchemy sessionmaker + + Returns: + Next priority value (max + 1, or 1 if no features exist) + """ + session = session_maker() + try: + # Use BEGIN IMMEDIATE for proper serialization in parallel mode + session.execute(text("BEGIN IMMEDIATE")) + result = session.execute(text(""" + SELECT COALESCE(MAX(priority), 0) + 1 FROM features + """)).fetchone() + session.commit() + return result[0] + except Exception: + session.rollback() + raise + finally: + session.close() diff --git a/auto_documentation.py b/auto_documentation.py new file mode 100644 index 00000000..db92bab6 --- /dev/null +++ b/auto_documentation.py @@ -0,0 +1,715 @@ +""" +Auto Documentation Generator +============================ + +Automatically generates documentation for projects: +- README.md from app_spec.txt +- API documentation from route analysis +- Setup guide from dependencies and scripts +- Component documentation from source files + +Triggers: +- After initialization (optional) +- After all features pass (optional) +- On-demand via API + +Configuration: +- docs.enabled: Enable/disable auto-generation +- docs.generate_on_init: Generate after project init +- docs.generate_on_complete: Generate when all features pass +- docs.output_dir: Output directory (default: "docs") +""" + +import json +import logging +import re +from dataclasses import dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Optional + +logger = logging.getLogger(__name__) + + +@dataclass +class APIEndpoint: + """Represents an API endpoint for documentation.""" + + method: str + path: str + description: str = "" + parameters: list[dict] = field(default_factory=list) + response_type: str = "" + auth_required: bool = False + + +@dataclass +class ComponentDoc: + """Represents a component for documentation.""" + + name: str + file_path: str + description: str = "" + props: list[dict] = field(default_factory=list) + exports: list[str] = field(default_factory=list) + + +@dataclass +class ProjectDocs: + """Complete project documentation.""" + + project_name: str + description: str + tech_stack: dict + setup_steps: list[str] + features: list[dict] + api_endpoints: list[APIEndpoint] + components: list[ComponentDoc] + environment_vars: list[dict] + scripts: dict + generated_at: str = "" + + def __post_init__(self): + if not self.generated_at: + self.generated_at = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") + + +class DocumentationGenerator: + """ + Generates documentation for a project. + + Usage: + generator = DocumentationGenerator(project_dir) + docs = generator.generate() + generator.write_readme(docs) + generator.write_api_docs(docs) + """ + + def __init__(self, project_dir: Path, output_dir: str = "docs"): + self.project_dir = Path(project_dir) + self.output_dir = self.project_dir / output_dir + self.app_spec: Optional[dict] = None + + def generate(self) -> ProjectDocs: + """ + Generate complete project documentation. + + Returns: + ProjectDocs with all documentation data + """ + # Parse app spec + self.app_spec = self._parse_app_spec() + + # Gather information + tech_stack = self._detect_tech_stack() + setup_steps = self._extract_setup_steps() + features = self._extract_features() + api_endpoints = self._extract_api_endpoints() + components = self._extract_components() + env_vars = self._extract_environment_vars() + scripts = self._extract_scripts() + + return ProjectDocs( + project_name=self.app_spec.get("name", self.project_dir.name) if self.app_spec else self.project_dir.name, + description=self.app_spec.get("description", "") if self.app_spec else "", + tech_stack=tech_stack, + setup_steps=setup_steps, + features=features, + api_endpoints=api_endpoints, + components=components, + environment_vars=env_vars, + scripts=scripts, + ) + + def _parse_app_spec(self) -> Optional[dict]: + """Parse app_spec.txt XML file.""" + spec_path = self.project_dir / "prompts" / "app_spec.txt" + if not spec_path.exists(): + return None + + try: + content = spec_path.read_text() + + # Extract key elements from XML + result = {} + + # App name + name_match = re.search(r"]*>([^<]+)", content) + if name_match: + result["name"] = name_match.group(1).strip() + + # Description + desc_match = re.search(r"]*>(.*?)", content, re.DOTALL) + if desc_match: + result["description"] = desc_match.group(1).strip() + + # Tech stack + stack_match = re.search(r"]*>(.*?)", content, re.DOTALL) + if stack_match: + result["tech_stack_raw"] = stack_match.group(1).strip() + + # Features + features_match = re.search(r"]*>(.*?)", content, re.DOTALL) + if features_match: + result["features_raw"] = features_match.group(1).strip() + + return result + + except Exception as e: + logger.warning(f"Error parsing app_spec.txt: {e}") + return None + + def _detect_tech_stack(self) -> dict: + """Detect tech stack from project files.""" + stack = { + "frontend": [], + "backend": [], + "database": [], + "tools": [], + } + + # Check package.json + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + deps = {**data.get("dependencies", {}), **data.get("devDependencies", {})} + + if "react" in deps: + stack["frontend"].append("React") + if "next" in deps: + stack["frontend"].append("Next.js") + if "vue" in deps: + stack["frontend"].append("Vue.js") + if "express" in deps: + stack["backend"].append("Express") + if "fastify" in deps: + stack["backend"].append("Fastify") + if "@nestjs/core" in deps: + stack["backend"].append("NestJS") + if "typescript" in deps: + stack["tools"].append("TypeScript") + if "tailwindcss" in deps: + stack["tools"].append("Tailwind CSS") + if "prisma" in deps: + stack["database"].append("Prisma") + except Exception: + pass + + # Check Python + requirements = self.project_dir / "requirements.txt" + pyproject = self.project_dir / "pyproject.toml" + + if requirements.exists() or pyproject.exists(): + content = "" + if requirements.exists(): + content = requirements.read_text() + if pyproject.exists(): + content += pyproject.read_text() + + if "fastapi" in content.lower(): + stack["backend"].append("FastAPI") + if "django" in content.lower(): + stack["backend"].append("Django") + if "flask" in content.lower(): + stack["backend"].append("Flask") + if "sqlalchemy" in content.lower(): + stack["database"].append("SQLAlchemy") + if "postgresql" in content.lower() or "psycopg" in content.lower(): + stack["database"].append("PostgreSQL") + + return stack + + def _extract_setup_steps(self) -> list[str]: + """Extract setup steps from init.sh and package.json.""" + steps = [] + + # Prerequisites + package_json = self.project_dir / "package.json" + requirements = self.project_dir / "requirements.txt" + + if package_json.exists(): + steps.append("Ensure Node.js is installed (v18+ recommended)") + if requirements.exists(): + steps.append("Ensure Python 3.10+ is installed") + + # Installation + if package_json.exists(): + steps.append("Run `npm install` to install dependencies") + if requirements.exists(): + steps.append("Create virtual environment: `python -m venv venv`") + steps.append("Activate venv: `source venv/bin/activate` (Unix) or `venv\\Scripts\\activate` (Windows)") + steps.append("Install dependencies: `pip install -r requirements.txt`") + + # Check for init.sh + init_sh = self.project_dir / "init.sh" + if init_sh.exists(): + steps.append("Run initialization script: `./init.sh`") + + # Check for .env.example + env_example = self.project_dir / ".env.example" + if env_example.exists(): + steps.append("Copy `.env.example` to `.env` and configure environment variables") + + # Development server + if package_json.exists(): + steps.append("Start development server: `npm run dev`") + elif (self.project_dir / "main.py").exists(): + steps.append("Start server: `python main.py` or `uvicorn main:app --reload`") + + return steps + + def _extract_features(self) -> list[dict]: + """Extract features from database or app_spec.""" + features = [] + + # Try to read from features.db + db_path = self.project_dir / "features.db" + if db_path.exists(): + try: + from api.database import Feature, get_session + + session = get_session(db_path) + db_features = session.query(Feature).order_by(Feature.priority).all() + + for f in db_features: + features.append( + { + "category": f.category, + "name": f.name, + "description": f.description, + "status": "completed" if f.passes else "pending", + } + ) + session.close() + except Exception as e: + logger.warning(f"Error reading features.db: {e}") + + # If no features from DB, try app_spec + if not features and self.app_spec and self.app_spec.get("features_raw"): + # Parse feature items from raw text + raw = self.app_spec["features_raw"] + for line in raw.split("\n"): + line = line.strip() + if line.startswith("-") or line.startswith("*"): + features.append( + { + "category": "Feature", + "name": line.lstrip("-* "), + "description": "", + "status": "pending", + } + ) + + return features + + def _extract_api_endpoints(self) -> list[APIEndpoint]: + """Extract API endpoints from source files.""" + endpoints = [] + + # Check for Express routes (JS and TS files) + from itertools import chain + js_ts_routes = chain( + self.project_dir.glob("**/routes/**/*.js"), + self.project_dir.glob("**/routes/**/*.ts"), + ) + for route_file in js_ts_routes: + try: + content = route_file.read_text() + # Match router.get/post/put/delete + matches = re.findall( + r'router\.(get|post|put|delete|patch)\s*\(\s*[\'"]([^\'"]+)[\'"]', + content, + re.IGNORECASE, + ) + for method, path in matches: + endpoints.append( + APIEndpoint( + method=method.upper(), + path=path, + description=f"Endpoint from {route_file.name}", + ) + ) + except Exception: + pass + + # Check for FastAPI routes + for py_file in self.project_dir.glob("**/*.py"): + if "node_modules" in str(py_file) or "venv" in str(py_file): + continue + try: + content = py_file.read_text() + # Match @app.get/post/etc or @router.get/post/etc + matches = re.findall( + r'@(?:app|router)\.(get|post|put|delete|patch)\s*\(\s*[\'"]([^\'"]+)[\'"]', + content, + re.IGNORECASE, + ) + for method, path in matches: + endpoints.append( + APIEndpoint( + method=method.upper(), + path=path, + description=f"Endpoint from {py_file.name}", + ) + ) + except Exception: + pass + + return endpoints + + def _extract_components(self) -> list[ComponentDoc]: + """Extract component documentation from source files.""" + components = [] + + # React/Vue components + for ext in ["tsx", "jsx", "vue"]: + for comp_file in self.project_dir.glob(f"**/components/**/*.{ext}"): + if "node_modules" in str(comp_file): + continue + try: + content = comp_file.read_text() + name = comp_file.stem + + # Try to extract description from JSDoc + description = "" + jsdoc_match = re.search(r"/\*\*\s*(.*?)\s*\*/", content, re.DOTALL) + if jsdoc_match: + description = jsdoc_match.group(1).strip() + # Clean up JSDoc syntax + description = re.sub(r"\s*\*\s*", " ", description) + description = re.sub(r"@\w+.*", "", description).strip() + + # Extract props from TypeScript interface + props = [] + props_match = re.search(r"interface\s+\w*Props\s*{([^}]+)}", content) + if props_match: + props_content = props_match.group(1) + for line in props_content.split("\n"): + line = line.strip() + if ":" in line and not line.startswith("//"): + prop_match = re.match(r"(\w+)\??:\s*(.+?)[;,]?$", line) + if prop_match: + props.append( + { + "name": prop_match.group(1), + "type": prop_match.group(2), + } + ) + + components.append( + ComponentDoc( + name=name, + file_path=str(comp_file.relative_to(self.project_dir)), + description=description, + props=props, + ) + ) + except Exception: + pass + + return components + + def _extract_environment_vars(self) -> list[dict]: + """Extract environment variables from .env.example or .env.""" + env_vars = [] + + for env_file in [".env.example", ".env.sample", ".env"]: + env_path = self.project_dir / env_file + if env_path.exists(): + try: + for line in env_path.read_text().split("\n"): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + key, value = line.split("=", 1) + # Mask sensitive values + if any( + s in key.lower() for s in ["secret", "password", "key", "token"] + ): + value = "***" + elif env_file == ".env": + value = "***" # Mask all values from actual .env + + env_vars.append( + { + "name": key.strip(), + "example": value.strip(), + "required": not value.strip() or value == "***", + } + ) + break # Only process first found env file + except Exception: + pass + + return env_vars + + def _extract_scripts(self) -> dict: + """Extract npm scripts from package.json.""" + scripts = {} + + package_json = self.project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + scripts = data.get("scripts", {}) + except Exception: + pass + + return scripts + + def write_readme(self, docs: ProjectDocs) -> Path: + """ + Write README.md file. + + Args: + docs: ProjectDocs data + + Returns: + Path to written file + """ + readme_path = self.project_dir / "README.md" + + lines = [] + lines.append(f"# {docs.project_name}\n") + + if docs.description: + lines.append(f"{docs.description}\n") + + # Tech Stack + if any(docs.tech_stack.values()): + lines.append("## Tech Stack\n") + for category, items in docs.tech_stack.items(): + if items: + lines.append(f"**{category.title()}:** {', '.join(items)}\n") + lines.append("") + + # Features + if docs.features: + lines.append("## Features\n") + # Group by category + categories = {} + for f in docs.features: + cat = f.get("category", "General") + if cat not in categories: + categories[cat] = [] + categories[cat].append(f) + + for cat, features in categories.items(): + lines.append(f"### {cat}\n") + for f in features: + status = "[x]" if f.get("status") == "completed" else "[ ]" + lines.append(f"- {status} {f['name']}") + lines.append("") + + # Getting Started + if docs.setup_steps: + lines.append("## Getting Started\n") + lines.append("### Prerequisites\n") + for step in docs.setup_steps[:2]: # First few are usually prerequisites + lines.append(f"- {step}") + lines.append("") + lines.append("### Installation\n") + for i, step in enumerate(docs.setup_steps[2:], 1): + lines.append(f"{i}. {step}") + lines.append("") + + # Environment Variables + if docs.environment_vars: + lines.append("## Environment Variables\n") + lines.append("| Variable | Required | Example |") + lines.append("|----------|----------|---------|") + for var in docs.environment_vars: + required = "Yes" if var.get("required") else "No" + lines.append(f"| `{var['name']}` | {required} | `{var['example']}` |") + lines.append("") + + # Available Scripts + if docs.scripts: + lines.append("## Available Scripts\n") + for name, command in docs.scripts.items(): + lines.append(f"- `npm run {name}` - {command}") + lines.append("") + + # API Endpoints + if docs.api_endpoints: + lines.append("## API Endpoints\n") + lines.append("| Method | Path | Description |") + lines.append("|--------|------|-------------|") + for ep in docs.api_endpoints[:20]: # Limit to 20 + lines.append(f"| {ep.method} | `{ep.path}` | {ep.description} |") + if len(docs.api_endpoints) > 20: + lines.append(f"\n*...and {len(docs.api_endpoints) - 20} more endpoints*") + lines.append("") + + # Components + if docs.components: + lines.append("## Components\n") + for comp in docs.components[:15]: # Limit to 15 + lines.append(f"### {comp.name}\n") + if comp.description: + lines.append(f"{comp.description}\n") + lines.append(f"**File:** `{comp.file_path}`\n") + if comp.props: + lines.append("**Props:**") + for prop in comp.props: + lines.append(f"- `{prop['name']}`: {prop['type']}") + lines.append("") + + # Footer + lines.append("---\n") + lines.append(f"*Generated on {docs.generated_at[:10]} by Autocoder*\n") + + readme_path.write_text("\n".join(lines)) + return readme_path + + def write_api_docs(self, docs: ProjectDocs) -> Optional[Path]: + """ + Write API documentation file. + + Args: + docs: ProjectDocs data + + Returns: + Path to written file or None if no API endpoints + """ + if not docs.api_endpoints: + return None + + self.output_dir.mkdir(parents=True, exist_ok=True) + api_docs_path = self.output_dir / "API.md" + + lines = [] + lines.append(f"# {docs.project_name} API Documentation\n") + + # Group endpoints by base path + grouped = {} + for ep in docs.api_endpoints: + # Handle root path "/" and paths like "/api/..." + parts = ep.path.split("/") + base = parts[1] if len(parts) > 1 and parts[1] else "root" + if base not in grouped: + grouped[base] = [] + grouped[base].append(ep) + + for base, endpoints in sorted(grouped.items()): + lines.append(f"## {base.title()}\n") + for ep in endpoints: + lines.append(f"### {ep.method} `{ep.path}`\n") + if ep.description: + lines.append(f"{ep.description}\n") + if ep.parameters: + lines.append("**Parameters:**") + for param in ep.parameters: + lines.append(f"- `{param['name']}` ({param.get('type', 'any')})") + lines.append("") + if ep.response_type: + lines.append(f"**Response:** `{ep.response_type}`\n") + lines.append("") + + lines.append("---\n") + lines.append(f"*Generated on {docs.generated_at[:10]} by Autocoder*\n") + + api_docs_path.write_text("\n".join(lines)) + return api_docs_path + + def write_setup_guide(self, docs: ProjectDocs) -> Path: + """ + Write detailed setup guide. + + Args: + docs: ProjectDocs data + + Returns: + Path to written file + """ + self.output_dir.mkdir(parents=True, exist_ok=True) + setup_path = self.output_dir / "SETUP.md" + + lines = [] + lines.append(f"# {docs.project_name} Setup Guide\n") + + # Prerequisites + lines.append("## Prerequisites\n") + if docs.tech_stack.get("frontend"): + lines.append("- Node.js 18 or later") + lines.append("- npm, yarn, or pnpm") + if docs.tech_stack.get("backend") and any( + "Fast" in b or "Django" in b or "Flask" in b for b in docs.tech_stack.get("backend", []) + ): + lines.append("- Python 3.10 or later") + lines.append("- pip or pipenv") + lines.append("") + + # Installation + lines.append("## Installation\n") + for i, step in enumerate(docs.setup_steps, 1): + lines.append(f"### Step {i}: {step.split(':')[0] if ':' in step else 'Setup'}\n") + lines.append(f"{step}\n") + # Add code block for command steps + if "`" in step: + cmd = re.search(r"`([^`]+)`", step) + if cmd: + lines.append(f"```bash\n{cmd.group(1)}\n```\n") + + # Environment Configuration + if docs.environment_vars: + lines.append("## Environment Configuration\n") + lines.append("Create a `.env` file in the project root:\n") + lines.append("```env") + for var in docs.environment_vars: + lines.append(f"{var['name']}={var['example']}") + lines.append("```\n") + + # Running the Application + lines.append("## Running the Application\n") + if docs.scripts: + if "dev" in docs.scripts: + lines.append("### Development\n") + lines.append("```bash\nnpm run dev\n```\n") + if "build" in docs.scripts: + lines.append("### Production Build\n") + lines.append("```bash\nnpm run build\n```\n") + if "start" in docs.scripts: + lines.append("### Start Production Server\n") + lines.append("```bash\nnpm start\n```\n") + + lines.append("---\n") + lines.append(f"*Generated on {docs.generated_at[:10]} by Autocoder*\n") + + setup_path.write_text("\n".join(lines)) + return setup_path + + def generate_all(self) -> dict: + """ + Generate all documentation files. + + Returns: + Dict with paths to generated files + """ + docs = self.generate() + + results = { + "readme": str(self.write_readme(docs)), + "setup": str(self.write_setup_guide(docs)), + } + + api_path = self.write_api_docs(docs) + if api_path: + results["api"] = str(api_path) + + return results + + +def generate_documentation(project_dir: Path, output_dir: str = "docs") -> dict: + """ + Generate all documentation for a project. + + Args: + project_dir: Project directory + output_dir: Output directory for docs + + Returns: + Dict with paths to generated files + """ + generator = DocumentationGenerator(project_dir, output_dir) + return generator.generate_all() diff --git a/autonomous_agent_demo.py b/autonomous_agent_demo.py index 16702f5e..166e80b1 100644 --- a/autonomous_agent_demo.py +++ b/autonomous_agent_demo.py @@ -46,6 +46,7 @@ from agent import run_autonomous_agent from registry import DEFAULT_MODEL, get_project_path +from structured_logging import get_logger def parse_args() -> argparse.Namespace: @@ -178,6 +179,9 @@ def main() -> None: project_dir_input = args.project_dir project_dir = Path(project_dir_input) + # Logger will be initialized after project_dir is resolved + logger = None + if project_dir.is_absolute(): # Absolute path provided - use directly if not project_dir.exists(): @@ -193,6 +197,17 @@ def main() -> None: print("Use an absolute path or register the project first.") return + # Initialize logger now that project_dir is resolved + logger = get_logger(project_dir, agent_id="entry-point", console_output=False) + logger.info( + "Script started", + input_path=project_dir_input, + resolved_path=str(project_dir), + agent_type=args.agent_type, + concurrency=args.concurrency, + yolo_mode=args.yolo, + ) + try: if args.agent_type: # Subprocess mode - spawned by orchestrator for a specific role @@ -228,8 +243,12 @@ def main() -> None: except KeyboardInterrupt: print("\n\nInterrupted by user") print("To resume, run the same command again") + if logger: + logger.info("Interrupted by user") except Exception as e: print(f"\nFatal error: {e}") + if logger: + logger.error("Fatal error", error_type=type(e).__name__, message=str(e)[:200]) raise diff --git a/client.py b/client.py index 7ea04a5e..36a5b937 100644 --- a/client.py +++ b/client.py @@ -16,6 +16,7 @@ from dotenv import load_dotenv from security import bash_security_hook +from structured_logging import get_logger # Load environment variables from .env file if present load_dotenv() @@ -94,7 +95,7 @@ def get_playwright_browser() -> str: "mcp__features__feature_create_bulk", "mcp__features__feature_create", "mcp__features__feature_clear_in_progress", - "mcp__features__feature_release_testing", # Release testing claim + "mcp__features__feature_verify_quality", # Run quality checks (lint, type-check) # Dependency management "mcp__features__feature_add_dependency", "mcp__features__feature_remove_dependency", @@ -179,6 +180,9 @@ def create_client( Note: Authentication is handled by start.bat/start.sh before this runs. The Claude SDK auto-detects credentials from the Claude CLI configuration """ + # Initialize logger for client configuration events + logger = get_logger(project_dir, agent_id="client", console_output=False) + # Build allowed tools list based on mode # In YOLO mode, exclude Playwright tools for faster prototyping allowed_tools = [*BUILTIN_TOOLS, *FEATURE_MCP_TOOLS] @@ -225,6 +229,7 @@ def create_client( with open(settings_file, "w") as f: json.dump(security_settings, f, indent=2) + logger.info("Settings file written", file_path=str(settings_file)) print(f"Created security settings at {settings_file}") print(" - Sandbox enabled (OS-level bash isolation)") print(f" - Filesystem restricted to: {project_dir.resolve()}") @@ -300,6 +305,7 @@ def create_client( if sdk_env: print(f" - API overrides: {', '.join(sdk_env.keys())}") + logger.info("API overrides configured", is_ollama=is_ollama, overrides=list(sdk_env.keys())) if is_ollama: print(" - Ollama Mode: Using local models") elif "ANTHROPIC_BASE_URL" in sdk_env: @@ -352,6 +358,16 @@ async def pre_compact_hook( # } return SyncHookJSONOutput() + # Log client creation + logger.info( + "Client created", + model=model, + yolo_mode=yolo_mode, + agent_id=agent_id, + is_alternative_api=is_alternative_api, + max_turns=1000, + ) + return ClaudeSDKClient( options=ClaudeAgentOptions( model=model, diff --git a/design_tokens.py b/design_tokens.py new file mode 100644 index 00000000..fb207122 --- /dev/null +++ b/design_tokens.py @@ -0,0 +1,593 @@ +""" +Design Tokens Management +======================== + +Manages design tokens for consistent styling across projects. + +Features: +- Parse design tokens from app_spec.txt or JSON config +- Generate CSS custom properties +- Generate Tailwind CSS configuration +- Generate SCSS variables +- Validate color contrast ratios +- Support for light/dark themes + +Configuration: +- design_tokens section in app_spec.txt +- .autocoder/design-tokens.json for custom tokens + +Token Categories: +- Colors (primary, secondary, accent, neutral, semantic) +- Spacing (scale, gutters, margins) +- Typography (fonts, sizes, weights, line-heights) +- Borders (radii, widths) +- Shadows +- Animations (durations, easings) +""" + +import colorsys +import json +import logging +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Optional + +logger = logging.getLogger(__name__) + + +@dataclass +class ColorToken: + """A color token with variants.""" + + name: str + value: str # Hex color + variants: dict = field(default_factory=dict) # 50-950 shades + + def to_hsl(self) -> tuple[float, float, float]: + """Convert hex to HSL.""" + hex_color = self.value.lstrip("#") + r, g, b = tuple(int(hex_color[i : i + 2], 16) / 255 for i in (0, 2, 4)) + hue, lightness, sat = colorsys.rgb_to_hls(r, g, b) + return (hue * 360, sat * 100, lightness * 100) + + def generate_shades(self) -> dict: + """Generate 50-950 shades from base color.""" + hue, sat, lightness = self.to_hsl() + + shades = { + "50": self._hsl_to_hex(hue, max(10, sat * 0.3), 95), + "100": self._hsl_to_hex(hue, max(15, sat * 0.5), 90), + "200": self._hsl_to_hex(hue, max(20, sat * 0.6), 80), + "300": self._hsl_to_hex(hue, max(25, sat * 0.7), 70), + "400": self._hsl_to_hex(hue, max(30, sat * 0.85), 60), + "500": self.value, # Base color + "600": self._hsl_to_hex(hue, min(100, sat * 1.1), lightness * 0.85), + "700": self._hsl_to_hex(hue, min(100, sat * 1.15), lightness * 0.7), + "800": self._hsl_to_hex(hue, min(100, sat * 1.2), lightness * 0.55), + "900": self._hsl_to_hex(hue, min(100, sat * 1.25), lightness * 0.4), + "950": self._hsl_to_hex(hue, min(100, sat * 1.3), lightness * 0.25), + } + return shades + + def _hsl_to_hex(self, hue: float, sat: float, lightness: float) -> str: + """Convert HSL to hex.""" + r, g, b = colorsys.hls_to_rgb(hue / 360, lightness / 100, sat / 100) + return f"#{int(r * 255):02x}{int(g * 255):02x}{int(b * 255):02x}" + + +@dataclass +class DesignTokens: + """Complete design token system.""" + + colors: dict = field(default_factory=dict) + spacing: list = field(default_factory=lambda: [4, 8, 12, 16, 24, 32, 48, 64, 96]) + typography: dict = field(default_factory=dict) + borders: dict = field(default_factory=dict) + shadows: dict = field(default_factory=dict) + animations: dict = field(default_factory=dict) + + @classmethod + def default(cls) -> "DesignTokens": + """Create default design tokens.""" + return cls( + colors={ + "primary": "#3B82F6", # Blue + "secondary": "#6366F1", # Indigo + "accent": "#F59E0B", # Amber + "success": "#10B981", # Emerald + "warning": "#F59E0B", # Amber + "error": "#EF4444", # Red + "info": "#3B82F6", # Blue + "neutral": "#6B7280", # Gray + }, + spacing=[4, 8, 12, 16, 24, 32, 48, 64, 96], + typography={ + "font_family": { + "sans": "Inter, system-ui, sans-serif", + "mono": "JetBrains Mono, monospace", + }, + "font_size": { + "xs": "0.75rem", + "sm": "0.875rem", + "base": "1rem", + "lg": "1.125rem", + "xl": "1.25rem", + "2xl": "1.5rem", + "3xl": "1.875rem", + "4xl": "2.25rem", + }, + "font_weight": { + "normal": "400", + "medium": "500", + "semibold": "600", + "bold": "700", + }, + "line_height": { + "tight": "1.25", + "normal": "1.5", + "relaxed": "1.75", + }, + }, + borders={ + "radius": { + "none": "0", + "sm": "0.125rem", + "md": "0.375rem", + "lg": "0.5rem", + "xl": "0.75rem", + "2xl": "1rem", + "full": "9999px", + }, + "width": { + "0": "0", + "1": "1px", + "2": "2px", + "4": "4px", + }, + }, + shadows={ + "sm": "0 1px 2px 0 rgb(0 0 0 / 0.05)", + "md": "0 4px 6px -1px rgb(0 0 0 / 0.1), 0 2px 4px -2px rgb(0 0 0 / 0.1)", + "lg": "0 10px 15px -3px rgb(0 0 0 / 0.1), 0 4px 6px -4px rgb(0 0 0 / 0.1)", + "xl": "0 20px 25px -5px rgb(0 0 0 / 0.1), 0 8px 10px -6px rgb(0 0 0 / 0.1)", + }, + animations={ + "duration": { + "fast": "150ms", + "normal": "300ms", + "slow": "500ms", + }, + "easing": { + "linear": "linear", + "ease-in": "cubic-bezier(0.4, 0, 1, 1)", + "ease-out": "cubic-bezier(0, 0, 0.2, 1)", + "ease-in-out": "cubic-bezier(0.4, 0, 0.2, 1)", + }, + }, + ) + + +class DesignTokensManager: + """ + Manages design tokens for a project. + + Usage: + manager = DesignTokensManager(project_dir) + tokens = manager.load() + manager.generate_css(tokens) + manager.generate_tailwind_config(tokens) + """ + + def __init__(self, project_dir: Path): + self.project_dir = Path(project_dir) + self.config_path = self.project_dir / ".autocoder" / "design-tokens.json" + + def load(self) -> DesignTokens: + """ + Load design tokens from config file or app_spec.txt. + + Returns: + DesignTokens instance + """ + # Try to load from config file + if self.config_path.exists(): + return self._load_from_config() + + # Try to parse from app_spec.txt + app_spec = self.project_dir / "prompts" / "app_spec.txt" + if app_spec.exists(): + tokens = self._parse_from_app_spec(app_spec) + if tokens: + return tokens + + # Return defaults + return DesignTokens.default() + + def _load_from_config(self) -> DesignTokens: + """Load tokens from JSON config.""" + try: + data = json.loads(self.config_path.read_text()) + return DesignTokens( + colors=data.get("colors", {}), + spacing=data.get("spacing", [4, 8, 12, 16, 24, 32, 48, 64, 96]), + typography=data.get("typography", {}), + borders=data.get("borders", {}), + shadows=data.get("shadows", {}), + animations=data.get("animations", {}), + ) + except Exception as e: + logger.warning(f"Error loading design tokens config: {e}") + return DesignTokens.default() + + def _parse_from_app_spec(self, app_spec_path: Path) -> Optional[DesignTokens]: + """Parse design tokens from app_spec.txt.""" + try: + content = app_spec_path.read_text() + + # Find design_tokens section + match = re.search(r"]*>(.*?)", content, re.DOTALL) + if not match: + return None + + tokens_content = match.group(1) + tokens = DesignTokens.default() + + # Parse colors + colors_match = re.search(r"]*>(.*?)", tokens_content, re.DOTALL) + if colors_match: + for color_match in re.finditer(r"<(\w+)>([^<]+)", colors_match.group(1)): + tokens.colors[color_match.group(1)] = color_match.group(2).strip() + + # Parse spacing + spacing_match = re.search(r"]*>(.*?)", tokens_content, re.DOTALL) + if spacing_match: + scale_match = re.search(r"\s*\[([^\]]+)\]", spacing_match.group(1)) + if scale_match: + tokens.spacing = [int(x.strip()) for x in scale_match.group(1).split(",")] + + # Parse typography + typo_match = re.search(r"]*>(.*?)", tokens_content, re.DOTALL) + if typo_match: + font_match = re.search(r"([^<]+)", typo_match.group(1)) + if font_match: + tokens.typography["font_family"] = {"sans": font_match.group(1).strip()} + + return tokens + + except Exception as e: + logger.warning(f"Error parsing app_spec.txt for design tokens: {e}") + return None + + def save(self, tokens: DesignTokens) -> Path: + """ + Save design tokens to config file. + + Args: + tokens: DesignTokens to save + + Returns: + Path to saved file + """ + self.config_path.parent.mkdir(parents=True, exist_ok=True) + + data = { + "colors": tokens.colors, + "spacing": tokens.spacing, + "typography": tokens.typography, + "borders": tokens.borders, + "shadows": tokens.shadows, + "animations": tokens.animations, + } + + self.config_path.write_text(json.dumps(data, indent=2)) + return self.config_path + + def generate_css(self, tokens: DesignTokens, output_path: Optional[Path] = None) -> str: + """ + Generate CSS custom properties from design tokens. + + Args: + tokens: DesignTokens to convert + output_path: Optional path to write CSS file + + Returns: + CSS content + """ + lines = [ + "/* Design Tokens - Auto-generated by Autocoder */", + "/* Do not edit directly - modify .autocoder/design-tokens.json instead */", + "", + ":root {", + ] + + # Colors with shades + lines.append(" /* Colors */") + for name, value in tokens.colors.items(): + color_token = ColorToken(name=name, value=value) + shades = color_token.generate_shades() + + lines.append(f" --color-{name}: {value};") + for shade, shade_value in shades.items(): + lines.append(f" --color-{name}-{shade}: {shade_value};") + + # Spacing + lines.append("") + lines.append(" /* Spacing */") + for i, space in enumerate(tokens.spacing): + lines.append(f" --spacing-{i}: {space}px;") + + # Typography + lines.append("") + lines.append(" /* Typography */") + if "font_family" in tokens.typography: + for name, value in tokens.typography["font_family"].items(): + lines.append(f" --font-{name}: {value};") + + if "font_size" in tokens.typography: + for name, value in tokens.typography["font_size"].items(): + lines.append(f" --text-{name}: {value};") + + if "font_weight" in tokens.typography: + for name, value in tokens.typography["font_weight"].items(): + lines.append(f" --font-weight-{name}: {value};") + + if "line_height" in tokens.typography: + for name, value in tokens.typography["line_height"].items(): + lines.append(f" --leading-{name}: {value};") + + # Borders + lines.append("") + lines.append(" /* Borders */") + if "radius" in tokens.borders: + for name, value in tokens.borders["radius"].items(): + lines.append(f" --radius-{name}: {value};") + + if "width" in tokens.borders: + for name, value in tokens.borders["width"].items(): + lines.append(f" --border-{name}: {value};") + + # Shadows + lines.append("") + lines.append(" /* Shadows */") + for name, value in tokens.shadows.items(): + lines.append(f" --shadow-{name}: {value};") + + # Animations + lines.append("") + lines.append(" /* Animations */") + if "duration" in tokens.animations: + for name, value in tokens.animations["duration"].items(): + lines.append(f" --duration-{name}: {value};") + + if "easing" in tokens.animations: + for name, value in tokens.animations["easing"].items(): + lines.append(f" --ease-{name}: {value};") + + lines.append("}") + + css_content = "\n".join(lines) + + if output_path: + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(css_content) + + return css_content + + def generate_tailwind_config(self, tokens: DesignTokens, output_path: Optional[Path] = None) -> str: + """ + Generate Tailwind CSS configuration from design tokens. + + Args: + tokens: DesignTokens to convert + output_path: Optional path to write config file + + Returns: + JavaScript config content + """ + # Build color config with shades + colors = {} + for name, value in tokens.colors.items(): + color_token = ColorToken(name=name, value=value) + shades = color_token.generate_shades() + colors[name] = { + "DEFAULT": value, + **shades, + } + + # Build spacing config + spacing = {} + for i, space in enumerate(tokens.spacing): + spacing[str(i)] = f"{space}px" + spacing[str(space)] = f"{space}px" + + # Build the config + config = { + "theme": { + "extend": { + "colors": colors, + "spacing": spacing, + "fontFamily": tokens.typography.get("font_family", {}), + "fontSize": tokens.typography.get("font_size", {}), + "fontWeight": tokens.typography.get("font_weight", {}), + "lineHeight": tokens.typography.get("line_height", {}), + "borderRadius": tokens.borders.get("radius", {}), + "borderWidth": tokens.borders.get("width", {}), + "boxShadow": tokens.shadows, + "transitionDuration": tokens.animations.get("duration", {}), + "transitionTimingFunction": tokens.animations.get("easing", {}), + } + } + } + + # Format as JavaScript + config_json = json.dumps(config, indent=2) + js_content = f"""/** @type {{import('tailwindcss').Config}} */ +// Design Tokens - Auto-generated by Autocoder +// Modify .autocoder/design-tokens.json to update + +module.exports = {config_json} +""" + + if output_path: + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(js_content) + + return js_content + + def generate_scss(self, tokens: DesignTokens, output_path: Optional[Path] = None) -> str: + """ + Generate SCSS variables from design tokens. + + Args: + tokens: DesignTokens to convert + output_path: Optional path to write SCSS file + + Returns: + SCSS content + """ + lines = [ + "// Design Tokens - Auto-generated by Autocoder", + "// Do not edit directly - modify .autocoder/design-tokens.json instead", + "", + "// Colors", + ] + + for name, value in tokens.colors.items(): + color_token = ColorToken(name=name, value=value) + shades = color_token.generate_shades() + + lines.append(f"$color-{name}: {value};") + for shade, shade_value in shades.items(): + lines.append(f"$color-{name}-{shade}: {shade_value};") + + lines.append("") + lines.append("// Spacing") + for i, space in enumerate(tokens.spacing): + lines.append(f"$spacing-{i}: {space}px;") + + lines.append("") + lines.append("// Typography") + if "font_family" in tokens.typography: + for name, value in tokens.typography["font_family"].items(): + lines.append(f"$font-{name}: {value};") + + if "font_size" in tokens.typography: + for name, value in tokens.typography["font_size"].items(): + lines.append(f"$text-{name}: {value};") + + lines.append("") + lines.append("// Borders") + if "radius" in tokens.borders: + for name, value in tokens.borders["radius"].items(): + lines.append(f"$radius-{name}: {value};") + + lines.append("") + lines.append("// Shadows") + for name, value in tokens.shadows.items(): + lines.append(f"$shadow-{name}: {value};") + + scss_content = "\n".join(lines) + + if output_path: + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(scss_content) + + return scss_content + + def validate_contrast(self, tokens: DesignTokens) -> list[dict]: + """ + Validate color contrast ratios for accessibility. + + Args: + tokens: DesignTokens to validate + + Returns: + List of contrast issues + """ + issues = [] + + # Check primary colors against white/black backgrounds + for name, value in tokens.colors.items(): + color_token = ColorToken(name=name, value=value) + _hue, _sat, lightness = color_token.to_hsl() + + # Simple contrast check based on lightness + if lightness > 50: + # Light color - should contrast with white + if lightness > 85: + issues.append( + { + "color": name, + "value": value, + "issue": "Color may not have sufficient contrast with white background", + "suggestion": "Use darker shade for text on white", + } + ) + else: + # Dark color - should contrast with black + if lightness < 15: + issues.append( + { + "color": name, + "value": value, + "issue": "Color may not have sufficient contrast with dark background", + "suggestion": "Use lighter shade for text on dark", + } + ) + + return issues + + def generate_all(self, output_dir: Optional[Path] = None) -> dict: + """ + Generate all token files. + + Args: + output_dir: Output directory (default: project root styles/) + + Returns: + Dict with paths to generated files + """ + tokens = self.load() + output = output_dir or self.project_dir / "src" / "styles" + + # Generate files and store paths (not content) + css_path = output / "tokens.css" + scss_path = output / "_tokens.scss" + + self.generate_css(tokens, css_path) + self.generate_scss(tokens, scss_path) + + results = { + "css": str(css_path), + "scss": str(scss_path), + } + + # Check for Tailwind + if (self.project_dir / "tailwind.config.js").exists() or ( + self.project_dir / "tailwind.config.ts" + ).exists(): + tailwind_path = output / "tailwind.tokens.js" + self.generate_tailwind_config(tokens, tailwind_path) + results["tailwind"] = str(tailwind_path) + + # Validate and report + issues = self.validate_contrast(tokens) + if issues: + results["contrast_issues"] = issues + + return results + + +def generate_design_tokens(project_dir: Path) -> dict: + """ + Generate all design token files for a project. + + Args: + project_dir: Project directory + + Returns: + Dict with paths to generated files + """ + manager = DesignTokensManager(project_dir) + return manager.generate_all() diff --git a/git_workflow.py b/git_workflow.py new file mode 100644 index 00000000..7262987d --- /dev/null +++ b/git_workflow.py @@ -0,0 +1,526 @@ +""" +Git Workflow Module +=================== + +Professional git workflow with feature branches for Autocoder. + +Workflow Modes: +- feature_branches: Create branch per feature, merge on completion +- trunk: All changes on main branch (default) +- none: No git operations + +Branch naming: feature/{feature_id}-{slugified-name} +Example: feature/42-user-can-login +""" + +import logging +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Literal, Optional + +logger = logging.getLogger(__name__) + +# Type alias for workflow modes +WorkflowMode = Literal["feature_branches", "trunk", "none"] + + +@dataclass +class BranchInfo: + """Information about a git branch.""" + + name: str + feature_id: Optional[int] = None + is_feature_branch: bool = False + is_current: bool = False + + +@dataclass +class WorkflowResult: + """Result of a workflow operation.""" + + success: bool + message: str + branch_name: Optional[str] = None + previous_branch: Optional[str] = None + + +def slugify(text: str) -> str: + """ + Convert text to URL-friendly slug. + + Example: "User can login" -> "user-can-login" + """ + # Convert to lowercase + text = text.lower() + # Replace spaces and underscores with hyphens + text = re.sub(r"[\s_]+", "-", text) + # Remove non-alphanumeric characters (except hyphens) + text = re.sub(r"[^a-z0-9-]", "", text) + # Remove consecutive hyphens + text = re.sub(r"-+", "-", text) + # Trim hyphens from ends + text = text.strip("-") + # Limit length + return text[:50] + + +def get_branch_name(feature_id: int, feature_name: str, prefix: str = "feature/") -> str: + """ + Generate branch name for a feature. + + Args: + feature_id: Feature ID + feature_name: Feature name + prefix: Branch prefix (default: "feature/") + + Returns: + Branch name like "feature/42-user-can-login" + """ + slug = slugify(feature_name) + return f"{prefix}{feature_id}-{slug}" + + +class GitWorkflow: + """ + Git workflow manager for feature branches. + + Usage: + workflow = GitWorkflow(project_dir, mode="feature_branches") + + # Start working on a feature + result = workflow.start_feature(42, "User can login") + # ... implement feature ... + + # Complete feature (merge to main) + result = workflow.complete_feature(42) + + # Or abort feature + result = workflow.abort_feature(42) + """ + + def __init__( + self, + project_dir: Path, + mode: WorkflowMode = "trunk", + branch_prefix: str = "feature/", + main_branch: str = "main", + auto_merge: bool = False, + ): + self.project_dir = Path(project_dir) + self.mode = mode + self.branch_prefix = branch_prefix + self.main_branch = main_branch + self.auto_merge = auto_merge + + def _run_git(self, *args, check: bool = True) -> subprocess.CompletedProcess: + """Run a git command in the project directory.""" + cmd = ["git"] + list(args) + return subprocess.run( + cmd, + cwd=self.project_dir, + capture_output=True, + text=True, + check=check, + ) + + def _is_git_repo(self) -> bool: + """Check if directory is a git repository.""" + try: + self._run_git("rev-parse", "--git-dir") + return True + except subprocess.CalledProcessError: + return False + + def _get_current_branch(self) -> Optional[str]: + """Get name of current branch.""" + try: + result = self._run_git("rev-parse", "--abbrev-ref", "HEAD") + return result.stdout.strip() + except subprocess.CalledProcessError: + return None + + def _branch_exists(self, branch_name: str) -> bool: + """Check if a branch exists.""" + result = self._run_git("branch", "--list", branch_name, check=False) + return bool(result.stdout.strip()) + + def _has_uncommitted_changes(self) -> bool: + """Check for uncommitted changes.""" + result = self._run_git("status", "--porcelain", check=False) + return bool(result.stdout.strip()) + + def get_feature_branch(self, feature_id: int) -> Optional[str]: + """ + Find branch for a feature ID. + + Returns branch name if found, None otherwise. + """ + result = self._run_git("branch", "--list", f"{self.branch_prefix}{feature_id}-*", check=False) + branches = [b.strip().lstrip("* ") for b in result.stdout.strip().split("\n") if b.strip()] + return branches[0] if branches else None + + def start_feature(self, feature_id: int, feature_name: str) -> WorkflowResult: + """ + Start working on a feature (create and checkout branch). + + In trunk mode, this is a no-op. + In feature_branches mode, creates branch and checks it out. + + Args: + feature_id: Feature ID + feature_name: Feature name for branch naming + + Returns: + WorkflowResult with success status and branch info + """ + if self.mode == "none": + return WorkflowResult( + success=True, + message="Git workflow disabled", + ) + + if self.mode == "trunk": + return WorkflowResult( + success=True, + message="Using trunk-based development", + branch_name=self.main_branch, + ) + + # feature_branches mode + if not self._is_git_repo(): + return WorkflowResult( + success=False, + message="Not a git repository", + ) + + # Check for existing branch + existing_branch = self.get_feature_branch(feature_id) + if existing_branch: + # Switch to existing branch + try: + self._run_git("checkout", existing_branch) + return WorkflowResult( + success=True, + message=f"Switched to existing branch: {existing_branch}", + branch_name=existing_branch, + ) + except subprocess.CalledProcessError as e: + return WorkflowResult( + success=False, + message=f"Failed to checkout branch: {e.stderr}", + ) + + # Create new branch + branch_name = get_branch_name(feature_id, feature_name, self.branch_prefix) + current_branch = self._get_current_branch() + + try: + # Stash uncommitted changes if any + had_changes = self._has_uncommitted_changes() + if had_changes: + self._run_git("stash", "push", "-m", f"Auto-stash before feature/{feature_id}") + + # Create and checkout new branch from main + self._run_git("checkout", self.main_branch) + self._run_git("checkout", "-b", branch_name) + + # Apply stashed changes if any + if had_changes: + self._run_git("stash", "pop", check=False) + + logger.info(f"Created feature branch: {branch_name}") + return WorkflowResult( + success=True, + message=f"Created branch: {branch_name}", + branch_name=branch_name, + previous_branch=current_branch, + ) + + except subprocess.CalledProcessError as e: + return WorkflowResult( + success=False, + message=f"Failed to create branch: {e.stderr}", + ) + + def commit_feature_progress( + self, + feature_id: int, + message: str, + add_all: bool = True, + ) -> WorkflowResult: + """ + Commit current changes for a feature. + + Args: + feature_id: Feature ID + message: Commit message + add_all: Whether to add all changes + + Returns: + WorkflowResult with success status + """ + if self.mode == "none": + return WorkflowResult( + success=True, + message="Git workflow disabled", + ) + + if not self._is_git_repo(): + return WorkflowResult( + success=False, + message="Not a git repository", + ) + + try: + if add_all: + self._run_git("add", "-A") + + # Check if there are staged changes + result = self._run_git("diff", "--cached", "--quiet", check=False) + if result.returncode == 0: + return WorkflowResult( + success=True, + message="No changes to commit", + ) + + # Commit + full_message = f"feat(feature-{feature_id}): {message}" + self._run_git("commit", "-m", full_message) + + return WorkflowResult( + success=True, + message=f"Committed: {message}", + ) + + except subprocess.CalledProcessError as e: + return WorkflowResult( + success=False, + message=f"Commit failed: {e.stderr}", + ) + + def complete_feature(self, feature_id: int) -> WorkflowResult: + """ + Complete a feature (merge to main if auto_merge enabled). + + Args: + feature_id: Feature ID + + Returns: + WorkflowResult with success status + """ + if self.mode != "feature_branches": + return WorkflowResult( + success=True, + message="Feature branches not enabled", + ) + + branch_name = self.get_feature_branch(feature_id) + if not branch_name: + return WorkflowResult( + success=False, + message=f"No branch found for feature {feature_id}", + ) + + current_branch = self._get_current_branch() + + try: + # Commit any remaining changes + if self._has_uncommitted_changes(): + self._run_git("add", "-A") + self._run_git("commit", "-m", f"feat(feature-{feature_id}): final changes") + + if not self.auto_merge: + return WorkflowResult( + success=True, + message=f"Feature complete on branch {branch_name}. Manual merge required.", + branch_name=branch_name, + ) + + # Auto-merge enabled + self._run_git("checkout", self.main_branch) + self._run_git("merge", "--no-ff", branch_name, "-m", f"Merge feature {feature_id}") + + # Optionally delete feature branch + # self._run_git("branch", "-d", branch_name) + + logger.info(f"Merged feature branch {branch_name} to {self.main_branch}") + return WorkflowResult( + success=True, + message=f"Merged {branch_name} to {self.main_branch}", + branch_name=self.main_branch, + previous_branch=branch_name, + ) + + except subprocess.CalledProcessError as e: + # Restore original branch on failure + if current_branch: + self._run_git("checkout", current_branch, check=False) + return WorkflowResult( + success=False, + message=f"Merge failed: {e.stderr}", + ) + + def abort_feature(self, feature_id: int, delete_branch: bool = False) -> WorkflowResult: + """ + Abort a feature (discard changes, optionally delete branch). + + Args: + feature_id: Feature ID + delete_branch: Whether to delete the feature branch + + Returns: + WorkflowResult with success status + """ + if self.mode != "feature_branches": + return WorkflowResult( + success=True, + message="Feature branches not enabled", + ) + + branch_name = self.get_feature_branch(feature_id) + if not branch_name: + return WorkflowResult( + success=False, + message=f"No branch found for feature {feature_id}", + ) + + try: + # Discard uncommitted changes + self._run_git("checkout", "--", ".", check=False) + self._run_git("clean", "-fd", check=False) + + # Switch back to main + self._run_git("checkout", self.main_branch) + + if delete_branch: + self._run_git("branch", "-D", branch_name) + return WorkflowResult( + success=True, + message=f"Aborted and deleted branch {branch_name}", + branch_name=self.main_branch, + ) + + return WorkflowResult( + success=True, + message=f"Aborted feature, branch {branch_name} preserved", + branch_name=self.main_branch, + ) + + except subprocess.CalledProcessError as e: + return WorkflowResult( + success=False, + message=f"Abort failed: {e.stderr}", + ) + + def list_feature_branches(self) -> list[BranchInfo]: + """ + List all feature branches. + + Returns: + List of BranchInfo objects + """ + if not self._is_git_repo(): + return [] + + result = self._run_git("branch", "--list", f"{self.branch_prefix}*", check=False) + + branches = [] + for line in result.stdout.strip().split("\n"): + if not line.strip(): + continue + is_current = line.startswith("*") + name = line.strip().lstrip("* ") + + # Extract feature ID from branch name + feature_id = None + match = re.search(rf"{re.escape(self.branch_prefix)}(\d+)-", name) + if match: + feature_id = int(match.group(1)) + + branches.append( + BranchInfo( + name=name, + feature_id=feature_id, + is_feature_branch=True, + is_current=is_current, + ) + ) + + return branches + + def get_status(self) -> dict: + """ + Get current git workflow status. + + Returns: + Dict with current branch, mode, uncommitted changes, etc. + """ + if not self._is_git_repo(): + return { + "is_git_repo": False, + "mode": self.mode, + } + + current = self._get_current_branch() + feature_branches = self.list_feature_branches() + + # Check if current branch is a feature branch + current_feature_id = None + if current and current.startswith(self.branch_prefix): + match = re.search(rf"{re.escape(self.branch_prefix)}(\d+)-", current) + if match: + current_feature_id = int(match.group(1)) + + return { + "is_git_repo": True, + "mode": self.mode, + "current_branch": current, + "main_branch": self.main_branch, + "is_on_feature_branch": current_feature_id is not None, + "current_feature_id": current_feature_id, + "has_uncommitted_changes": self._has_uncommitted_changes(), + "feature_branches": [b.name for b in feature_branches], + "feature_branch_count": len(feature_branches), + } + + +def get_workflow(project_dir: Path) -> GitWorkflow: + """ + Get git workflow manager for a project. + + Reads configuration from .autocoder/config.json. + + Args: + project_dir: Project directory + + Returns: + GitWorkflow instance configured for the project + """ + # Try to load config + mode: WorkflowMode = "trunk" + branch_prefix = "feature/" + main_branch = "main" + auto_merge = False + + try: + from server.services.autocoder_config import load_config + + config = load_config(project_dir) + git_config = config.get("git_workflow", {}) + + mode = git_config.get("mode", "trunk") + branch_prefix = git_config.get("branch_prefix", "feature/") + main_branch = git_config.get("main_branch", "main") + auto_merge = git_config.get("auto_merge", False) + except Exception: + pass + + return GitWorkflow( + project_dir, + mode=mode, + branch_prefix=branch_prefix, + main_branch=main_branch, + auto_merge=auto_merge, + ) diff --git a/integrations/__init__.py b/integrations/__init__.py new file mode 100644 index 00000000..df9ad1ec --- /dev/null +++ b/integrations/__init__.py @@ -0,0 +1,13 @@ +""" +Integrations Package +==================== + +External integrations for Autocoder including CI/CD, deployment, etc. +""" + +from .ci import generate_ci_config, generate_github_workflow + +__all__ = [ + "generate_ci_config", + "generate_github_workflow", +] diff --git a/integrations/ci/__init__.py b/integrations/ci/__init__.py new file mode 100644 index 00000000..48f9e200 --- /dev/null +++ b/integrations/ci/__init__.py @@ -0,0 +1,66 @@ +""" +CI/CD Integration Module +======================== + +Generate CI/CD configuration based on detected tech stack. + +Supported providers: +- GitHub Actions +- GitLab CI (planned) + +Features: +- Auto-detect tech stack and generate appropriate workflows +- Lint, type-check, test, build, deploy stages +- Environment management (staging, production) +""" + +from .github_actions import ( + GitHubWorkflow, + WorkflowTrigger, + generate_all_workflows, + generate_github_workflow, +) + +__all__ = [ + "generate_github_workflow", + "generate_all_workflows", + "GitHubWorkflow", + "WorkflowTrigger", +] + + +def generate_ci_config(project_dir, provider: str = "github") -> dict: + """ + Generate CI configuration based on detected tech stack. + + Args: + project_dir: Project directory + provider: CI provider ("github" or "gitlab") + + Returns: + Dict with generated configuration and file paths + """ + from pathlib import Path + + project_dir = Path(project_dir) + + if provider == "github": + workflows = generate_all_workflows(project_dir) + return { + "provider": "github", + "workflows": workflows, + "output_dir": str(project_dir / ".github" / "workflows"), + } + + elif provider == "gitlab": + # GitLab CI support planned + return { + "provider": "gitlab", + "error": "GitLab CI not yet implemented", + } + + else: + return { + "provider": provider, + "error": f"Unknown provider: {provider}", + } diff --git a/integrations/ci/github_actions.py b/integrations/ci/github_actions.py new file mode 100644 index 00000000..0a57cb07 --- /dev/null +++ b/integrations/ci/github_actions.py @@ -0,0 +1,609 @@ +""" +GitHub Actions Workflow Generator +================================= + +Generate GitHub Actions workflows based on detected tech stack. + +Workflow types: +- CI: Lint, type-check, test on push/PR +- Deploy: Build and deploy on merge to main +- Security: Dependency audit and code scanning +""" + +import json +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path +from typing import Any, Literal, Optional + +import yaml + + +class WorkflowTrigger(str, Enum): + """Workflow trigger types.""" + + PUSH = "push" + PULL_REQUEST = "pull_request" + WORKFLOW_DISPATCH = "workflow_dispatch" + SCHEDULE = "schedule" + + +@dataclass +class WorkflowJob: + """A job in a GitHub Actions workflow.""" + + name: str + runs_on: str = "ubuntu-latest" + steps: list[dict] = field(default_factory=list) + needs: list[str] = field(default_factory=list) + if_condition: Optional[str] = None + env: dict[str, str] = field(default_factory=dict) + + def to_dict(self) -> dict: + """Convert to workflow YAML format.""" + result = { + "name": self.name, + "runs-on": self.runs_on, + "steps": self.steps, + } + if self.needs: + result["needs"] = self.needs + if self.if_condition: + result["if"] = self.if_condition + if self.env: + result["env"] = self.env + return result + + +@dataclass +class GitHubWorkflow: + """A GitHub Actions workflow.""" + + name: str + filename: str + on: dict[str, Any] + jobs: dict[str, WorkflowJob] + env: dict[str, str] = field(default_factory=dict) + permissions: dict[str, str] = field(default_factory=dict) + + def to_yaml(self) -> str: + """Convert to YAML string.""" + workflow = { + "name": self.name, + "on": self.on, + "jobs": {name: job.to_dict() for name, job in self.jobs.items()}, + } + if self.env: + workflow["env"] = self.env + if self.permissions: + workflow["permissions"] = self.permissions + + return yaml.dump(workflow, default_flow_style=False, sort_keys=False) + + def save(self, project_dir: Path) -> Path: + """Save workflow to .github/workflows directory.""" + workflows_dir = project_dir / ".github" / "workflows" + workflows_dir.mkdir(parents=True, exist_ok=True) + + output_path = workflows_dir / self.filename + with open(output_path, "w") as f: + f.write(self.to_yaml()) + + return output_path + + +def _detect_stack(project_dir: Path) -> dict: + """Detect tech stack from project files.""" + stack = { + "has_node": False, + "has_python": False, + "has_typescript": False, + "has_react": False, + "has_nextjs": False, + "has_vue": False, + "has_fastapi": False, + "has_django": False, + "node_version": "20", + "python_version": "3.11", + "package_manager": "npm", + } + + # Check for Node.js + package_json = project_dir / "package.json" + if package_json.exists(): + stack["has_node"] = True + try: + with open(package_json) as f: + pkg = json.load(f) + deps = {**pkg.get("dependencies", {}), **pkg.get("devDependencies", {})} + + if "typescript" in deps: + stack["has_typescript"] = True + if "react" in deps: + stack["has_react"] = True + if "next" in deps: + stack["has_nextjs"] = True + if "vue" in deps: + stack["has_vue"] = True + + # Detect package manager + if (project_dir / "pnpm-lock.yaml").exists(): + stack["package_manager"] = "pnpm" + elif (project_dir / "yarn.lock").exists(): + stack["package_manager"] = "yarn" + elif (project_dir / "bun.lockb").exists(): + stack["package_manager"] = "bun" + + # Node version from engines + engines = pkg.get("engines", {}) + if "node" in engines: + version = engines["node"].strip(">=^~") + if version and version[0].isdigit(): + stack["node_version"] = version.split(".")[0] + except (json.JSONDecodeError, KeyError): + pass + + # Check for Python + if (project_dir / "requirements.txt").exists() or (project_dir / "pyproject.toml").exists(): + stack["has_python"] = True + + # Check for FastAPI + requirements_path = project_dir / "requirements.txt" + if requirements_path.exists(): + content = requirements_path.read_text().lower() + if "fastapi" in content: + stack["has_fastapi"] = True + if "django" in content: + stack["has_django"] = True + + # Python version from pyproject.toml + pyproject = project_dir / "pyproject.toml" + if pyproject.exists(): + content = pyproject.read_text() + if "python_requires" in content or "requires-python" in content: + import re + match = re.search(r'["\']>=?3\.(\d+)', content) + if match: + stack["python_version"] = f"3.{match.group(1)}" + + return stack + + +def _checkout_step() -> dict: + """Standard checkout step.""" + return { + "name": "Checkout code", + "uses": "actions/checkout@v4", + } + + +def _setup_node_step(version: str, cache: str = "npm") -> dict: + """Setup Node.js step.""" + return { + "name": "Setup Node.js", + "uses": "actions/setup-node@v4", + "with": { + "node-version": version, + "cache": cache, + }, + } + + +def _setup_python_step(version: str) -> dict: + """Setup Python step.""" + return { + "name": "Setup Python", + "uses": "actions/setup-python@v5", + "with": { + "python-version": version, + "cache": "pip", + }, + } + + +def _install_deps_step(package_manager: str = "npm") -> dict: + """Install dependencies step.""" + commands = { + "npm": "npm ci", + "yarn": "yarn install --frozen-lockfile", + "pnpm": "pnpm install --frozen-lockfile", + "bun": "bun install --frozen-lockfile", + } + return { + "name": "Install dependencies", + "run": commands.get(package_manager, "npm ci"), + } + + +def _python_install_step() -> dict: + """Python install dependencies step.""" + return { + "name": "Install dependencies", + "run": "pip install -r requirements.txt", + } + + +def generate_ci_workflow(project_dir: Path) -> GitHubWorkflow: + """ + Generate CI workflow for lint, type-check, and tests. + + Triggers on push to feature branches and PRs to main. + """ + stack = _detect_stack(project_dir) + + jobs = {} + + # Node.js jobs + if stack["has_node"]: + lint_steps = [ + _checkout_step(), + _setup_node_step(stack["node_version"], stack["package_manager"]), + _install_deps_step(stack["package_manager"]), + { + "name": "Run linter", + "run": f"{stack['package_manager']} run lint" if stack["package_manager"] != "npm" else "npm run lint", + }, + ] + + jobs["lint"] = WorkflowJob( + name="Lint", + steps=lint_steps, + ) + + if stack["has_typescript"]: + typecheck_steps = [ + _checkout_step(), + _setup_node_step(stack["node_version"], stack["package_manager"]), + _install_deps_step(stack["package_manager"]), + { + "name": "Type check", + "run": "npx tsc --noEmit", + }, + ] + + jobs["typecheck"] = WorkflowJob( + name="Type Check", + steps=typecheck_steps, + ) + + test_steps = [ + _checkout_step(), + _setup_node_step(stack["node_version"], stack["package_manager"]), + _install_deps_step(stack["package_manager"]), + { + "name": "Run tests", + "run": f"{stack['package_manager']} test" if stack["package_manager"] != "npm" else "npm test", + }, + ] + + jobs["test"] = WorkflowJob( + name="Test", + steps=test_steps, + needs=["lint"] + (["typecheck"] if stack["has_typescript"] else []), + ) + + build_steps = [ + _checkout_step(), + _setup_node_step(stack["node_version"], stack["package_manager"]), + _install_deps_step(stack["package_manager"]), + { + "name": "Build", + "run": f"{stack['package_manager']} run build" if stack["package_manager"] != "npm" else "npm run build", + }, + ] + + jobs["build"] = WorkflowJob( + name="Build", + steps=build_steps, + needs=["test"], + ) + + # Python jobs + if stack["has_python"]: + python_lint_steps = [ + _checkout_step(), + _setup_python_step(stack["python_version"]), + _python_install_step(), + { + "name": "Run ruff", + "run": "pip install ruff && ruff check .", + }, + ] + + jobs["python-lint"] = WorkflowJob( + name="Python Lint", + steps=python_lint_steps, + ) + + python_test_steps = [ + _checkout_step(), + _setup_python_step(stack["python_version"]), + _python_install_step(), + { + "name": "Run tests", + "run": "pip install pytest && pytest", + }, + ] + + jobs["python-test"] = WorkflowJob( + name="Python Test", + steps=python_test_steps, + needs=["python-lint"], + ) + + return GitHubWorkflow( + name="CI", + filename="ci.yml", + on={ + "push": { + "branches": ["main", "master", "feature/*"], + }, + "pull_request": { + "branches": ["main", "master"], + }, + }, + jobs=jobs, + ) + + +def generate_security_workflow(project_dir: Path) -> GitHubWorkflow: + """ + Generate security scanning workflow. + + Runs dependency audit and code scanning. + """ + stack = _detect_stack(project_dir) + + jobs = {} + + if stack["has_node"]: + audit_steps = [ + _checkout_step(), + _setup_node_step(stack["node_version"], stack["package_manager"]), + { + "name": "Run npm audit", + "run": "npm audit --audit-level=moderate", + "continue-on-error": True, + }, + ] + + jobs["npm-audit"] = WorkflowJob( + name="NPM Audit", + steps=audit_steps, + ) + + if stack["has_python"]: + pip_audit_steps = [ + _checkout_step(), + _setup_python_step(stack["python_version"]), + { + "name": "Run pip-audit", + "run": "pip install pip-audit && pip-audit -r requirements.txt", + "continue-on-error": True, + }, + ] + + jobs["pip-audit"] = WorkflowJob( + name="Pip Audit", + steps=pip_audit_steps, + ) + + # CodeQL analysis + codeql_steps = [ + _checkout_step(), + { + "name": "Initialize CodeQL", + "uses": "github/codeql-action/init@v3", + "with": { + "languages": ", ".join( + filter(None, [ + "javascript" if stack["has_node"] else None, + "python" if stack["has_python"] else None, + ]) + ), + }, + }, + { + "name": "Autobuild", + "uses": "github/codeql-action/autobuild@v3", + }, + { + "name": "Perform CodeQL Analysis", + "uses": "github/codeql-action/analyze@v3", + }, + ] + + jobs["codeql"] = WorkflowJob( + name="CodeQL Analysis", + steps=codeql_steps, + ) + + return GitHubWorkflow( + name="Security", + filename="security.yml", + on={ + "push": { + "branches": ["main", "master"], + }, + "pull_request": { + "branches": ["main", "master"], + }, + "schedule": [ + {"cron": "0 0 * * 0"}, # Weekly on Sunday + ], + }, + jobs=jobs, + permissions={ + "security-events": "write", + "actions": "read", + "contents": "read", + }, + ) + + +def generate_deploy_workflow(project_dir: Path) -> GitHubWorkflow: + """ + Generate deployment workflow. + + Builds and deploys on merge to main. + """ + stack = _detect_stack(project_dir) + + jobs = {} + + # Build job + build_steps = [_checkout_step()] + + if stack["has_node"]: + build_steps.extend([ + _setup_node_step(stack["node_version"], stack["package_manager"]), + _install_deps_step(stack["package_manager"]), + { + "name": "Build", + "run": f"{stack['package_manager']} run build" if stack["package_manager"] != "npm" else "npm run build", + }, + { + "name": "Upload build artifacts", + "uses": "actions/upload-artifact@v4", + "with": { + "name": "build", + "path": "dist/", + "retention-days": 7, + }, + }, + ]) + + if stack["has_python"]: + build_steps.extend([ + _setup_python_step(stack["python_version"]), + _python_install_step(), + { + "name": "Build package", + "run": "pip install build && python -m build", + }, + ]) + + jobs["build"] = WorkflowJob( + name="Build", + steps=build_steps, + ) + + # Deploy staging job (placeholder) + deploy_staging_steps = [ + _checkout_step(), + { + "name": "Download build artifacts", + "uses": "actions/download-artifact@v4", + "with": { + "name": "build", + "path": "dist/", + }, + }, + { + "name": "Deploy to staging", + "run": "echo 'Add your staging deployment commands here'", + "env": { + "DEPLOY_ENV": "staging", + }, + }, + ] + + jobs["deploy-staging"] = WorkflowJob( + name="Deploy to Staging", + steps=deploy_staging_steps, + needs=["build"], + env={"DEPLOY_ENV": "staging"}, + ) + + # Deploy production job (manual trigger) + deploy_prod_steps = [ + _checkout_step(), + { + "name": "Download build artifacts", + "uses": "actions/download-artifact@v4", + "with": { + "name": "build", + "path": "dist/", + }, + }, + { + "name": "Deploy to production", + "run": "echo 'Add your production deployment commands here'", + "env": { + "DEPLOY_ENV": "production", + }, + }, + ] + + jobs["deploy-production"] = WorkflowJob( + name="Deploy to Production", + steps=deploy_prod_steps, + needs=["deploy-staging"], + if_condition="github.event_name == 'workflow_dispatch'", + env={"DEPLOY_ENV": "production"}, + ) + + return GitHubWorkflow( + name="Deploy", + filename="deploy.yml", + on={ + "push": { + "branches": ["main", "master"], + }, + "workflow_dispatch": {}, + }, + jobs=jobs, + ) + + +def generate_github_workflow( + project_dir: Path, + workflow_type: Literal["ci", "security", "deploy"] = "ci", + save: bool = True, +) -> GitHubWorkflow: + """ + Generate a GitHub Actions workflow. + + Args: + project_dir: Project directory + workflow_type: Type of workflow (ci, security, deploy) + save: Whether to save the workflow file + + Returns: + GitHubWorkflow instance + """ + generators = { + "ci": generate_ci_workflow, + "security": generate_security_workflow, + "deploy": generate_deploy_workflow, + } + + generator = generators.get(workflow_type) + if not generator: + raise ValueError(f"Unknown workflow type: {workflow_type}") + + workflow = generator(Path(project_dir)) + + if save: + workflow.save(Path(project_dir)) + + return workflow + + +def generate_all_workflows(project_dir: Path, save: bool = True) -> dict[str, GitHubWorkflow]: + """ + Generate all workflow types for a project. + + Args: + project_dir: Project directory + save: Whether to save workflow files + + Returns: + Dict mapping workflow type to GitHubWorkflow + """ + workflows = {} + for workflow_type in ["ci", "security", "deploy"]: + workflows[workflow_type] = generate_github_workflow( + project_dir, workflow_type, save + ) + return workflows diff --git a/mcp_server/feature_mcp.py b/mcp_server/feature_mcp.py index a394f1e9..0a3f0e74 100755 --- a/mcp_server/feature_mcp.py +++ b/mcp_server/feature_mcp.py @@ -30,24 +30,25 @@ import json import os import sys -import threading from contextlib import asynccontextmanager from pathlib import Path from typing import Annotated from mcp.server.fastmcp import FastMCP from pydantic import BaseModel, Field +from sqlalchemy import text # Add parent directory to path so we can import from api module sys.path.insert(0, str(Path(__file__).parent.parent)) -from api.database import Feature, create_database +from api.database import Feature, atomic_transaction, create_database from api.dependency_resolver import ( MAX_DEPENDENCIES_PER_FEATURE, compute_scheduling_scores, would_create_circular_dependency, ) from api.migration import migrate_json_to_sqlite +from quality_gates import load_quality_config, verify_quality # Configuration from environment PROJECT_DIR = Path(os.environ.get("PROJECT_DIR", ".")).resolve() @@ -96,8 +97,9 @@ class BulkCreateInput(BaseModel): _session_maker = None _engine = None -# Lock for priority assignment to prevent race conditions -_priority_lock = threading.Lock() +# NOTE: The old threading.Lock() was removed because it only worked per-process, +# not cross-process. In parallel mode, multiple MCP servers run in separate +# processes, so the lock was useless. We now use atomic SQL operations instead. @asynccontextmanager @@ -226,6 +228,42 @@ def feature_get_summary( session.close() +@mcp.tool() +def feature_verify_quality() -> str: + """Run quality checks (lint, type-check) on the project. + + Automatically detects and runs available linters and type checkers: + - Linters: ESLint, Biome (JS/TS), ruff, flake8 (Python) + - Type checkers: TypeScript (tsc), Python (mypy) + - Custom scripts: .autocoder/quality-checks.sh + + Use this tool before marking a feature as passing to ensure code quality. + In strict mode (default), feature_mark_passing will block if quality checks fail. + + Returns: + JSON with: passed (bool), checks (dict), summary (str) + """ + config = load_quality_config(PROJECT_DIR) + + if not config.get("enabled", True): + return json.dumps({ + "passed": True, + "checks": {}, + "summary": "Quality gates disabled" + }) + + checks_config = config.get("checks", {}) + result = verify_quality( + PROJECT_DIR, + do_lint=checks_config.get("lint", True), + do_type_check=checks_config.get("type_check", True), + do_custom=True, + custom_script_path=checks_config.get("custom_script"), + ) + + return json.dumps(result) + + @mcp.tool() def feature_mark_passing( feature_id: Annotated[int, Field(description="The ID of the feature to mark as passing", ge=1)] @@ -235,24 +273,64 @@ def feature_mark_passing( Updates the feature's passes field to true and clears the in_progress flag. Use this after you have implemented the feature and verified it works correctly. + Uses atomic SQL UPDATE for parallel safety. + + IMPORTANT: In strict mode (default), this tool will run quality checks + (lint, type-check) and BLOCK if they fail. Run feature_verify_quality first + to see what checks will be performed. + Args: feature_id: The ID of the feature to mark as passing Returns: - JSON with success confirmation: {success, feature_id, name} + JSON with success confirmation: {success, feature_id, name, quality_result} + If strict mode is enabled and quality checks fail, returns an error. """ + # Run quality checks BEFORE opening DB session to avoid holding locks + config = load_quality_config(PROJECT_DIR) + quality_result = None + + if config.get("enabled", True): + checks_config = config.get("checks", {}) + quality_result = verify_quality( + PROJECT_DIR, + do_lint=checks_config.get("lint", True), + do_type_check=checks_config.get("type_check", True), + do_custom=True, + custom_script_path=checks_config.get("custom_script"), + ) + + # In strict mode, block if quality checks failed + if config.get("strict_mode", True) and not quality_result["passed"]: + return json.dumps({ + "error": "Quality checks failed - cannot mark feature as passing", + "quality_result": quality_result, + "hint": "Fix the issues and try again, or disable strict_mode in .autocoder/config.json" + }) + + # Now open DB session for the atomic update session = get_session() try: + # First get the feature name for the response feature = session.query(Feature).filter(Feature.id == feature_id).first() - if feature is None: return json.dumps({"error": f"Feature with ID {feature_id} not found"}) - feature.passes = True - feature.in_progress = False + name = feature.name + + # Atomic update - prevents race conditions in parallel mode + session.execute(text(""" + UPDATE features + SET passes = 1, in_progress = 0 + WHERE id = :id + """), {"id": feature_id}) session.commit() - return json.dumps({"success": True, "feature_id": feature_id, "name": feature.name}) + result = {"success": True, "feature_id": feature_id, "name": name} + if quality_result: + result["quality_result"] = quality_result + + return json.dumps(result) except Exception as e: session.rollback() return json.dumps({"error": f"Failed to mark feature passing: {str(e)}"}) @@ -270,6 +348,8 @@ def feature_mark_failing( Use this when a testing agent discovers that a previously-passing feature no longer works correctly (regression detected). + Uses atomic SQL UPDATE for parallel safety. + After marking as failing, you should: 1. Investigate the root cause 2. Fix the regression @@ -284,14 +364,20 @@ def feature_mark_failing( """ session = get_session() try: + # Check if feature exists first feature = session.query(Feature).filter(Feature.id == feature_id).first() - if feature is None: return json.dumps({"error": f"Feature with ID {feature_id} not found"}) - feature.passes = False - feature.in_progress = False + # Atomic update - prevents race conditions in parallel mode + session.execute(text(""" + UPDATE features + SET passes = 0, in_progress = 0 + WHERE id = :id + """), {"id": feature_id}) session.commit() + + # Refresh to get updated state session.refresh(feature) return json.dumps({ @@ -320,6 +406,8 @@ def feature_skip( worked on after all other pending features. Also clears the in_progress flag so the feature returns to "pending" status. + Uses atomic SQL UPDATE with subquery for parallel safety. + Args: feature_id: The ID of the feature to skip @@ -337,25 +425,28 @@ def feature_skip( return json.dumps({"error": "Cannot skip a feature that is already passing"}) old_priority = feature.priority + name = feature.name + + # Atomic update: set priority to max+1 in a single statement + # This prevents race conditions where two features get the same priority + session.execute(text(""" + UPDATE features + SET priority = (SELECT COALESCE(MAX(priority), 0) + 1 FROM features), + in_progress = 0 + WHERE id = :id + """), {"id": feature_id}) + session.commit() - # Use lock to prevent race condition in priority assignment - with _priority_lock: - # Get max priority and set this feature to max + 1 - max_priority_result = session.query(Feature.priority).order_by(Feature.priority.desc()).first() - new_priority = (max_priority_result[0] + 1) if max_priority_result else 1 - - feature.priority = new_priority - feature.in_progress = False - session.commit() - + # Refresh to get new priority session.refresh(feature) + new_priority = feature.priority return json.dumps({ - "id": feature.id, - "name": feature.name, + "id": feature_id, + "name": name, "old_priority": old_priority, "new_priority": new_priority, - "message": f"Feature '{feature.name}' moved to end of queue" + "message": f"Feature '{name}' moved to end of queue" }) except Exception as e: session.rollback() @@ -373,6 +464,9 @@ def feature_mark_in_progress( This prevents other agent sessions from working on the same feature. Call this after getting your assigned feature details with feature_get_by_id. + Uses atomic UPDATE WHERE for parallel safety - prevents two agents from + claiming the same feature simultaneously. + Args: feature_id: The ID of the feature to mark as in-progress @@ -381,21 +475,27 @@ def feature_mark_in_progress( """ session = get_session() try: - feature = session.query(Feature).filter(Feature.id == feature_id).first() - - if feature is None: - return json.dumps({"error": f"Feature with ID {feature_id} not found"}) - - if feature.passes: - return json.dumps({"error": f"Feature with ID {feature_id} is already passing"}) - - if feature.in_progress: - return json.dumps({"error": f"Feature with ID {feature_id} is already in-progress"}) - - feature.in_progress = True + # Atomic claim: only succeeds if feature is not already claimed or passing + result = session.execute(text(""" + UPDATE features + SET in_progress = 1 + WHERE id = :id AND passes = 0 AND in_progress = 0 + """), {"id": feature_id}) session.commit() - session.refresh(feature) + if result.rowcount == 0: + # Check why the claim failed + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if feature is None: + return json.dumps({"error": f"Feature with ID {feature_id} not found"}) + if feature.passes: + return json.dumps({"error": f"Feature with ID {feature_id} is already passing"}) + if feature.in_progress: + return json.dumps({"error": f"Feature with ID {feature_id} is already in-progress"}) + return json.dumps({"error": "Failed to mark feature in-progress for unknown reason"}) + + # Fetch the claimed feature + feature = session.query(Feature).filter(Feature.id == feature_id).first() return json.dumps(feature.to_dict()) except Exception as e: session.rollback() @@ -413,6 +513,8 @@ def feature_claim_and_get( Combines feature_mark_in_progress + feature_get_by_id into a single operation. If already in-progress, still returns the feature details (idempotent). + Uses atomic UPDATE WHERE for parallel safety. + Args: feature_id: The ID of the feature to claim and retrieve @@ -421,24 +523,35 @@ def feature_claim_and_get( """ session = get_session() try: + # First check if feature exists and get initial state feature = session.query(Feature).filter(Feature.id == feature_id).first() - if feature is None: return json.dumps({"error": f"Feature with ID {feature_id} not found"}) if feature.passes: return json.dumps({"error": f"Feature with ID {feature_id} is already passing"}) - # Idempotent: if already in-progress, just return details - already_claimed = feature.in_progress - if not already_claimed: - feature.in_progress = True - session.commit() + # Try atomic claim: only succeeds if not already claimed + result = session.execute(text(""" + UPDATE features + SET in_progress = 1 + WHERE id = :id AND passes = 0 AND in_progress = 0 + """), {"id": feature_id}) + session.commit() + + # Determine if we claimed it or it was already claimed + already_claimed = result.rowcount == 0 + if already_claimed: + # Verify it's in_progress (not some other failure condition) session.refresh(feature) + if not feature.in_progress: + return json.dumps({"error": f"Failed to claim feature {feature_id} for unknown reason"}) - result = feature.to_dict() - result["already_claimed"] = already_claimed - return json.dumps(result) + # Refresh to get current state + session.refresh(feature) + result_dict = feature.to_dict() + result_dict["already_claimed"] = already_claimed + return json.dumps(result_dict) except Exception as e: session.rollback() return json.dumps({"error": f"Failed to claim feature: {str(e)}"}) @@ -455,6 +568,8 @@ def feature_clear_in_progress( Use this when abandoning a feature or manually unsticking a stuck feature. The feature will return to the pending queue. + Uses atomic SQL UPDATE for parallel safety. + Args: feature_id: The ID of the feature to clear in-progress status @@ -463,15 +578,20 @@ def feature_clear_in_progress( """ session = get_session() try: + # Check if feature exists feature = session.query(Feature).filter(Feature.id == feature_id).first() - if feature is None: return json.dumps({"error": f"Feature with ID {feature_id} not found"}) - feature.in_progress = False + # Atomic update - idempotent, safe in parallel mode + session.execute(text(""" + UPDATE features + SET in_progress = 0 + WHERE id = :id + """), {"id": feature_id}) session.commit() - session.refresh(feature) + session.refresh(feature) return json.dumps(feature.to_dict()) except Exception as e: session.rollback() @@ -492,6 +612,8 @@ def feature_create_bulk( This is typically used by the initializer agent to set up the initial feature list from the app specification. + Uses EXCLUSIVE transaction to prevent priority collisions in parallel mode. + Args: features: List of features to create, each with: - category (str): Feature category @@ -506,13 +628,14 @@ def feature_create_bulk( Returns: JSON with: created (int) - number of features created, with_dependencies (int) """ - session = get_session() try: - # Use lock to prevent race condition in priority assignment - with _priority_lock: - # Get the starting priority - max_priority_result = session.query(Feature.priority).order_by(Feature.priority.desc()).first() - start_priority = (max_priority_result[0] + 1) if max_priority_result else 1 + # Use EXCLUSIVE transaction for bulk inserts to prevent conflicts + with atomic_transaction(_session_maker, "EXCLUSIVE") as session: + # Get the starting priority atomically within the transaction + result = session.execute(text(""" + SELECT COALESCE(MAX(priority), 0) FROM features + """)).fetchone() + start_priority = (result[0] or 0) + 1 # First pass: validate all features and their index-based dependencies for i, feature_data in enumerate(features): @@ -546,11 +669,11 @@ def feature_create_bulk( "error": f"Feature at index {i} cannot depend on feature at index {idx} (forward reference not allowed)" }) - # Second pass: create all features + # Second pass: create all features with reserved priorities created_features: list[Feature] = [] for i, feature_data in enumerate(features): db_feature = Feature( - priority=start_priority + i, + priority=start_priority + i, # Guaranteed unique within EXCLUSIVE transaction category=feature_data["category"], name=feature_data["name"], description=feature_data["description"], @@ -574,17 +697,13 @@ def feature_create_bulk( created_features[i].dependencies = sorted(dep_ids) deps_count += 1 - session.commit() - - return json.dumps({ - "created": len(created_features), - "with_dependencies": deps_count - }) + # Commit happens automatically on context manager exit + return json.dumps({ + "created": len(created_features), + "with_dependencies": deps_count + }) except Exception as e: - session.rollback() return json.dumps({"error": str(e)}) - finally: - session.close() @mcp.tool() @@ -599,6 +718,8 @@ def feature_create( Use this when the user asks to add a new feature, capability, or test case. The feature will be added with the next available priority number. + Uses IMMEDIATE transaction for parallel safety. + Args: category: Feature category for grouping (e.g., 'Authentication', 'API', 'UI') name: Descriptive name for the feature @@ -608,13 +729,14 @@ def feature_create( Returns: JSON with the created feature details including its ID """ - session = get_session() try: - # Use lock to prevent race condition in priority assignment - with _priority_lock: - # Get the next priority - max_priority_result = session.query(Feature.priority).order_by(Feature.priority.desc()).first() - next_priority = (max_priority_result[0] + 1) if max_priority_result else 1 + # Use IMMEDIATE transaction to prevent priority collisions + with atomic_transaction(_session_maker, "IMMEDIATE") as session: + # Get the next priority atomically within the transaction + result = session.execute(text(""" + SELECT COALESCE(MAX(priority), 0) + 1 FROM features + """)).fetchone() + next_priority = result[0] db_feature = Feature( priority=next_priority, @@ -626,20 +748,18 @@ def feature_create( in_progress=False, ) session.add(db_feature) - session.commit() + session.flush() # Get the ID - session.refresh(db_feature) + feature_dict = db_feature.to_dict() + # Commit happens automatically on context manager exit return json.dumps({ "success": True, "message": f"Created feature: {name}", - "feature": db_feature.to_dict() + "feature": feature_dict }) except Exception as e: - session.rollback() return json.dumps({"error": str(e)}) - finally: - session.close() @mcp.tool() @@ -652,6 +772,8 @@ def feature_add_dependency( The dependency_id feature must be completed before feature_id can be started. Validates: self-reference, existence, circular dependencies, max limit. + Uses IMMEDIATE transaction to prevent stale reads during cycle detection. + Args: feature_id: The ID of the feature that will depend on another feature dependency_id: The ID of the feature that must be completed first @@ -659,52 +781,49 @@ def feature_add_dependency( Returns: JSON with success status and updated dependencies list, or error message """ - session = get_session() try: - # Security: Self-reference check + # Security: Self-reference check (can do before transaction) if feature_id == dependency_id: return json.dumps({"error": "A feature cannot depend on itself"}) - feature = session.query(Feature).filter(Feature.id == feature_id).first() - dependency = session.query(Feature).filter(Feature.id == dependency_id).first() - - if not feature: - return json.dumps({"error": f"Feature {feature_id} not found"}) - if not dependency: - return json.dumps({"error": f"Dependency feature {dependency_id} not found"}) - - current_deps = feature.dependencies or [] - - # Security: Max dependencies limit - if len(current_deps) >= MAX_DEPENDENCIES_PER_FEATURE: - return json.dumps({"error": f"Maximum {MAX_DEPENDENCIES_PER_FEATURE} dependencies allowed per feature"}) - - # Check if already exists - if dependency_id in current_deps: - return json.dumps({"error": "Dependency already exists"}) - - # Security: Circular dependency check - # would_create_circular_dependency(features, source_id, target_id) - # source_id = feature gaining the dependency, target_id = feature being depended upon - all_features = [f.to_dict() for f in session.query(Feature).all()] - if would_create_circular_dependency(all_features, feature_id, dependency_id): - return json.dumps({"error": "Cannot add: would create circular dependency"}) - - # Add dependency - current_deps.append(dependency_id) - feature.dependencies = sorted(current_deps) - session.commit() - - return json.dumps({ - "success": True, - "feature_id": feature_id, - "dependencies": feature.dependencies - }) + # Use IMMEDIATE transaction for consistent cycle detection + with atomic_transaction(_session_maker, "IMMEDIATE") as session: + feature = session.query(Feature).filter(Feature.id == feature_id).first() + dependency = session.query(Feature).filter(Feature.id == dependency_id).first() + + if not feature: + return json.dumps({"error": f"Feature {feature_id} not found"}) + if not dependency: + return json.dumps({"error": f"Dependency feature {dependency_id} not found"}) + + current_deps = feature.dependencies or [] + + # Security: Max dependencies limit + if len(current_deps) >= MAX_DEPENDENCIES_PER_FEATURE: + return json.dumps({"error": f"Maximum {MAX_DEPENDENCIES_PER_FEATURE} dependencies allowed per feature"}) + + # Check if already exists + if dependency_id in current_deps: + return json.dumps({"error": "Dependency already exists"}) + + # Security: Circular dependency check + # Within IMMEDIATE transaction, snapshot is protected by write lock + all_features = [f.to_dict() for f in session.query(Feature).all()] + if would_create_circular_dependency(all_features, feature_id, dependency_id): + return json.dumps({"error": "Cannot add: would create circular dependency"}) + + # Add dependency atomically + new_deps = sorted(current_deps + [dependency_id]) + feature.dependencies = new_deps + # Commit happens automatically on context manager exit + + return json.dumps({ + "success": True, + "feature_id": feature_id, + "dependencies": new_deps + }) except Exception as e: - session.rollback() return json.dumps({"error": f"Failed to add dependency: {str(e)}"}) - finally: - session.close() @mcp.tool() @@ -714,6 +833,8 @@ def feature_remove_dependency( ) -> str: """Remove a dependency from a feature. + Uses IMMEDIATE transaction for parallel safety. + Args: feature_id: The ID of the feature to remove a dependency from dependency_id: The ID of the dependency to remove @@ -721,30 +842,29 @@ def feature_remove_dependency( Returns: JSON with success status and updated dependencies list, or error message """ - session = get_session() try: - feature = session.query(Feature).filter(Feature.id == feature_id).first() - if not feature: - return json.dumps({"error": f"Feature {feature_id} not found"}) - - current_deps = feature.dependencies or [] - if dependency_id not in current_deps: - return json.dumps({"error": "Dependency does not exist"}) - - current_deps.remove(dependency_id) - feature.dependencies = current_deps if current_deps else None - session.commit() - - return json.dumps({ - "success": True, - "feature_id": feature_id, - "dependencies": feature.dependencies or [] - }) + # Use IMMEDIATE transaction for consistent read-modify-write + with atomic_transaction(_session_maker, "IMMEDIATE") as session: + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if not feature: + return json.dumps({"error": f"Feature {feature_id} not found"}) + + current_deps = feature.dependencies or [] + if dependency_id not in current_deps: + return json.dumps({"error": "Dependency does not exist"}) + + # Remove dependency atomically + new_deps = [d for d in current_deps if d != dependency_id] + feature.dependencies = new_deps if new_deps else None + # Commit happens automatically on context manager exit + + return json.dumps({ + "success": True, + "feature_id": feature_id, + "dependencies": new_deps + }) except Exception as e: - session.rollback() return json.dumps({"error": f"Failed to remove dependency: {str(e)}"}) - finally: - session.close() @mcp.tool() @@ -890,6 +1010,8 @@ def feature_set_dependencies( Validates: self-reference, existence of all dependencies, circular dependencies, max limit. + Uses IMMEDIATE transaction to prevent stale reads during cycle detection. + Args: feature_id: The ID of the feature to set dependencies for dependency_ids: List of feature IDs that must be completed first @@ -897,9 +1019,8 @@ def feature_set_dependencies( Returns: JSON with success status and updated dependencies list, or error message """ - session = get_session() try: - # Security: Self-reference check + # Security: Self-reference check (can do before transaction) if feature_id in dependency_ids: return json.dumps({"error": "A feature cannot depend on itself"}) @@ -911,45 +1032,45 @@ def feature_set_dependencies( if len(dependency_ids) != len(set(dependency_ids)): return json.dumps({"error": "Duplicate dependencies not allowed"}) - feature = session.query(Feature).filter(Feature.id == feature_id).first() - if not feature: - return json.dumps({"error": f"Feature {feature_id} not found"}) - - # Validate all dependencies exist - all_feature_ids = {f.id for f in session.query(Feature).all()} - missing = [d for d in dependency_ids if d not in all_feature_ids] - if missing: - return json.dumps({"error": f"Dependencies not found: {missing}"}) - - # Check for circular dependencies - all_features = [f.to_dict() for f in session.query(Feature).all()] - # Temporarily update the feature's dependencies for cycle check - test_features = [] - for f in all_features: - if f["id"] == feature_id: - test_features.append({**f, "dependencies": dependency_ids}) - else: - test_features.append(f) - - for dep_id in dependency_ids: - # source_id = feature_id (gaining dep), target_id = dep_id (being depended upon) - if would_create_circular_dependency(test_features, feature_id, dep_id): - return json.dumps({"error": f"Cannot add dependency {dep_id}: would create circular dependency"}) - - # Set dependencies - feature.dependencies = sorted(dependency_ids) if dependency_ids else None - session.commit() - - return json.dumps({ - "success": True, - "feature_id": feature_id, - "dependencies": feature.dependencies or [] - }) + # Use IMMEDIATE transaction for consistent cycle detection + with atomic_transaction(_session_maker, "IMMEDIATE") as session: + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if not feature: + return json.dumps({"error": f"Feature {feature_id} not found"}) + + # Validate all dependencies exist + all_feature_ids = {f.id for f in session.query(Feature).all()} + missing = [d for d in dependency_ids if d not in all_feature_ids] + if missing: + return json.dumps({"error": f"Dependencies not found: {missing}"}) + + # Check for circular dependencies + # Within IMMEDIATE transaction, snapshot is protected by write lock + all_features = [f.to_dict() for f in session.query(Feature).all()] + # Temporarily update the feature's dependencies for cycle check + test_features = [] + for f in all_features: + if f["id"] == feature_id: + test_features.append({**f, "dependencies": dependency_ids}) + else: + test_features.append(f) + + for dep_id in dependency_ids: + if would_create_circular_dependency(test_features, feature_id, dep_id): + return json.dumps({"error": f"Cannot add dependency {dep_id}: would create circular dependency"}) + + # Set dependencies atomically + sorted_deps = sorted(dependency_ids) if dependency_ids else None + feature.dependencies = sorted_deps + # Commit happens automatically on context manager exit + + return json.dumps({ + "success": True, + "feature_id": feature_id, + "dependencies": sorted_deps or [] + }) except Exception as e: - session.rollback() return json.dumps({"error": f"Failed to set dependencies: {str(e)}"}) - finally: - session.close() if __name__ == "__main__": diff --git a/parallel_orchestrator.py b/parallel_orchestrator.py index 486b9635..e1adfe09 100644 --- a/parallel_orchestrator.py +++ b/parallel_orchestrator.py @@ -19,7 +19,9 @@ """ import asyncio +import atexit import os +import signal import subprocess import sys import threading @@ -27,10 +29,13 @@ from pathlib import Path from typing import Callable, Literal +from sqlalchemy import text + from api.database import Feature, create_database from api.dependency_resolver import are_dependencies_satisfied, compute_scheduling_scores from progress import has_features from server.utils.process_utils import kill_process_tree +from structured_logging import get_logger # Root directory of autocoder (where this script and autonomous_agent_demo.py live) AUTOCODER_ROOT = Path(__file__).parent.resolve() @@ -192,6 +197,16 @@ def __init__( # Database session for this orchestrator self._engine, self._session_maker = create_database(project_dir) + # Structured logger for persistent logs (saved to {project_dir}/.autocoder/logs.db) + # Uses console_output=False since orchestrator already has its own print statements + self._logger = get_logger(project_dir, agent_id="orchestrator", console_output=False) + self._logger.info( + "Orchestrator initialized", + max_concurrency=self.max_concurrency, + yolo_mode=yolo_mode, + testing_agent_ratio=testing_agent_ratio, + ) + def get_session(self): """Get a new database session.""" return self._session_maker() @@ -454,22 +469,45 @@ def start_feature(self, feature_id: int, resume: bool = False) -> tuple[bool, st # Mark as in_progress in database (or verify it's resumable) session = self.get_session() try: - feature = session.query(Feature).filter(Feature.id == feature_id).first() - if not feature: - return False, "Feature not found" - if feature.passes: - return False, "Feature already complete" - if resume: - # Resuming: feature should already be in_progress + # Resuming: verify feature is already in_progress + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if not feature: + return False, "Feature not found" if not feature.in_progress: return False, "Feature not in progress, cannot resume" + if feature.passes: + return False, "Feature already complete" else: - # Starting fresh: feature should not be in_progress - if feature.in_progress: - return False, "Feature already in progress" - feature.in_progress = True + # Starting fresh: atomic claim using UPDATE-WHERE pattern (same as testing agent) + # This prevents race conditions where multiple agents try to claim the same feature + from sqlalchemy import text + result = session.execute( + text(""" + UPDATE features + SET in_progress = 1 + WHERE id = :feature_id + AND passes = 0 + AND in_progress = 0 + """), + {"feature_id": feature_id} + ) session.commit() + + if result.rowcount == 0: + # Atomic claim failed - another agent claimed it or feature is already complete + # Query to determine exact reason for better error messages + feature = session.query(Feature).filter(Feature.id == feature_id).first() + if not feature: + return False, "Feature not found" + if feature.passes: + return False, "Feature already complete" + if feature.in_progress: + return False, "Feature already claimed by another agent" + # Edge case: feature exists but conditions changed between check and update + return False, "Feature state changed, unable to claim" + + debug_log.log("CLAIM", f"Successfully claimed feature #{feature_id} atomically") finally: session.close() @@ -514,6 +552,7 @@ def _spawn_coding_agent(self, feature_id: int) -> tuple[bool, str]: ) except Exception as e: # Reset in_progress on failure + self._logger.error("Spawn coding agent failed", feature_id=feature_id, error=str(e)[:200]) session = self.get_session() try: feature = session.query(Feature).filter(Feature.id == feature_id).first() @@ -539,6 +578,7 @@ def _spawn_coding_agent(self, feature_id: int) -> tuple[bool, str]: self.on_status(feature_id, "running") print(f"Started coding agent for feature #{feature_id}", flush=True) + self._logger.info("Spawned coding agent", feature_id=feature_id, pid=proc.pid) return True, f"Started feature {feature_id}" def _spawn_testing_agent(self) -> tuple[bool, str]: @@ -597,6 +637,7 @@ def _spawn_testing_agent(self) -> tuple[bool, str]: ) except Exception as e: debug_log.log("TESTING", f"FAILED to spawn testing agent: {e}") + self._logger.error("Spawn testing agent failed", feature_id=feature_id, error=str(e)[:200]) return False, f"Failed to start testing agent: {e}" # Register process with feature ID (same pattern as coding agents) @@ -788,9 +829,11 @@ def _on_agent_complete( return # Coding agent completion + agent_status = "success" if return_code == 0 else "failed" debug_log.log("COMPLETE", f"Coding agent for feature #{feature_id} finished", return_code=return_code, - status="success" if return_code == 0 else "failed") + status=agent_status) + self._logger.info("Coding agent completed", feature_id=feature_id, status=agent_status, return_code=return_code) with self._lock: self.running_coding_agents.pop(feature_id, None) @@ -826,6 +869,7 @@ def _on_agent_complete( print(f"Feature #{feature_id} has failed {failure_count} times, will not retry", flush=True) debug_log.log("COMPLETE", f"Feature #{feature_id} exceeded max retries", failure_count=failure_count) + self._logger.warning("Feature exceeded max retries", feature_id=feature_id, failure_count=failure_count) status = "completed" if return_code == 0 else "failed" if self.on_status: @@ -1043,6 +1087,12 @@ async def run_loop(self): continue # Priority 2: Start new ready features + # CRITICAL: Dispose engine to force fresh database reads + # Coding agents run as subprocesses and commit changes (passes=True, in_progress=False). + # SQLAlchemy connection pool may cache stale connections. Disposing ensures we see + # all subprocess commits when checking dependencies. + debug_log.log("DB", "Disposing engine before get_ready_features()") + self._engine.dispose() ready = self.get_ready_features() if not ready: # Wait for running features to complete @@ -1081,27 +1131,40 @@ async def run_loop(self): slots_available=slots, features_to_start=[f['id'] for f in features_to_start]) + # Atomic UPDATE-WHERE in start_feature() provides database-level protection + # against race conditions. Continue to next feature if claiming fails. + claimed_count = 0 for i, feature in enumerate(features_to_start): - print(f"[DEBUG] Starting feature {i+1}/{len(features_to_start)}: #{feature['id']} - {feature['name']}", flush=True) + print(f"[DEBUG] Attempting to claim feature #{feature['id']} ({i+1}/{len(features_to_start)})...", flush=True) success, msg = self.start_feature(feature["id"]) + if not success: - print(f"[DEBUG] Failed to start feature #{feature['id']}: {msg}", flush=True) - debug_log.log("SPAWN", f"FAILED to start feature #{feature['id']}", - feature_name=feature['name'], - error=msg) - else: - print(f"[DEBUG] Successfully started feature #{feature['id']}", flush=True) - with self._lock: - running_count = len(self.running_coding_agents) - print(f"[DEBUG] Running coding agents after start: {running_count}", flush=True) - debug_log.log("SPAWN", f"Successfully started feature #{feature['id']}", + print(f"[DEBUG] Failed to claim feature #{feature['id']}: {msg}", flush=True) + debug_log.log("SPAWN", f"Failed to claim feature #{feature['id']}", feature_name=feature['name'], - running_coding_agents=running_count) + error=msg, + retry_strategy="continue_to_next") + # Continue to next feature instead of stopping + # This handles race conditions where another agent claimed it first + continue + + claimed_count += 1 + print(f"[DEBUG] Successfully claimed feature #{feature['id']}", flush=True) + with self._lock: + running_count = len(self.running_coding_agents) + print(f"[DEBUG] Running coding agents after start: {running_count}", flush=True) + debug_log.log("SPAWN", f"Successfully claimed feature #{feature['id']}", + feature_name=feature['name'], + running_coding_agents=running_count) + + if claimed_count < len(features_to_start): + print(f"[DEBUG] Claimed {claimed_count}/{len(features_to_start)} features (some already claimed)", flush=True) await asyncio.sleep(2) # Brief pause between starts except Exception as e: print(f"Orchestrator error: {e}", flush=True) + self._logger.error("Orchestrator loop error", error_type=type(e).__name__, message=str(e)[:200]) await self._wait_for_agent_completion() # Wait for remaining agents to complete @@ -1131,6 +1194,31 @@ def get_status(self) -> dict: "yolo_mode": self.yolo_mode, } + def cleanup(self) -> None: + """Clean up database resources. + + CRITICAL: Must be called when orchestrator exits to prevent database corruption. + - Forces WAL checkpoint to flush pending writes to main database file + - Disposes engine to close all connections + + This prevents stale cache issues when the orchestrator restarts. + """ + if self._engine is not None: + try: + debug_log.log("CLEANUP", "Forcing WAL checkpoint before dispose") + with self._engine.connect() as conn: + conn.execute(text("PRAGMA wal_checkpoint(FULL)")) + conn.commit() + debug_log.log("CLEANUP", "WAL checkpoint completed, disposing engine") + except Exception as e: + debug_log.log("CLEANUP", f"WAL checkpoint failed (non-fatal): {e}") + + try: + self._engine.dispose() + debug_log.log("CLEANUP", "Engine disposed successfully") + except Exception as e: + debug_log.log("CLEANUP", f"Engine dispose failed: {e}") + async def run_parallel_orchestrator( project_dir: Path, @@ -1157,11 +1245,60 @@ async def run_parallel_orchestrator( testing_agent_ratio=testing_agent_ratio, ) + # Clear any stuck features from previous interrupted sessions + # This is the RIGHT place to clear - BEFORE spawning any agents + # Agents will NO LONGER clear features on their individual startups (see agent.py fix) + try: + session = orchestrator.get_session() + cleared_count = 0 + + # Get all features marked in_progress + from api.database import Feature + stuck_features = session.query(Feature).filter( + Feature.in_progress == True + ).all() + + for feature in stuck_features: + feature.in_progress = False + cleared_count += 1 + + session.commit() + session.close() + + if cleared_count > 0: + print(f"[ORCHESTRATOR] Cleared {cleared_count} stuck features from previous session", flush=True) + + except Exception as e: + print(f"[ORCHESTRATOR] Warning: Failed to clear stuck features: {e}", flush=True) + + # Set up cleanup to run on exit (handles normal exit, exceptions, signals) + def cleanup_handler(): + debug_log.log("CLEANUP", "Cleanup handler invoked") + orchestrator.cleanup() + + atexit.register(cleanup_handler) + + # Set up signal handlers for graceful shutdown (SIGTERM, SIGINT) + def signal_handler(signum, frame): + debug_log.log("SIGNAL", f"Received signal {signum}, initiating cleanup") + print(f"\nReceived signal {signum}. Stopping agents...", flush=True) + orchestrator.stop_all() + orchestrator.cleanup() + sys.exit(0) + + # Register signal handlers (SIGTERM for process termination, SIGINT for Ctrl+C) + signal.signal(signal.SIGTERM, signal_handler) + signal.signal(signal.SIGINT, signal_handler) + try: await orchestrator.run_loop() except KeyboardInterrupt: print("\n\nInterrupted by user. Stopping agents...", flush=True) orchestrator.stop_all() + finally: + # CRITICAL: Always clean up database resources on exit + # This forces WAL checkpoint and disposes connections + orchestrator.cleanup() def main(): diff --git a/progress.py b/progress.py index 0821c90a..85f5d61c 100644 --- a/progress.py +++ b/progress.py @@ -10,12 +10,34 @@ import os import sqlite3 import urllib.request +from contextlib import closing from datetime import datetime, timezone from pathlib import Path WEBHOOK_URL = os.environ.get("PROGRESS_N8N_WEBHOOK_URL") PROGRESS_CACHE_FILE = ".progress_cache" +# SQLite connection settings for parallel mode safety +SQLITE_TIMEOUT = 30 # seconds to wait for locks +SQLITE_BUSY_TIMEOUT_MS = 30000 # milliseconds for PRAGMA busy_timeout + + +def _get_connection(db_file: Path) -> sqlite3.Connection: + """Get a SQLite connection with proper timeout settings. + + Uses timeout=30s and PRAGMA busy_timeout=30000 for safe operation + in parallel mode where multiple processes access the same database. + + Args: + db_file: Path to the SQLite database file + + Returns: + sqlite3.Connection with proper timeout settings + """ + conn = sqlite3.connect(db_file, timeout=SQLITE_TIMEOUT) + conn.execute(f"PRAGMA busy_timeout={SQLITE_BUSY_TIMEOUT_MS}") + return conn + def has_features(project_dir: Path) -> bool: """ @@ -31,8 +53,6 @@ def has_features(project_dir: Path) -> bool: Returns False if no features exist (initializer needs to run). """ - import sqlite3 - # Check legacy JSON file first json_file = project_dir / "feature_list.json" if json_file.exists(): @@ -44,12 +64,11 @@ def has_features(project_dir: Path) -> bool: return False try: - conn = sqlite3.connect(db_file) - cursor = conn.cursor() - cursor.execute("SELECT COUNT(*) FROM features") - count = cursor.fetchone()[0] - conn.close() - return count > 0 + with closing(_get_connection(db_file)) as conn: + cursor = conn.cursor() + cursor.execute("SELECT COUNT(*) FROM features") + count = cursor.fetchone()[0] + return count > 0 except Exception: # Database exists but can't be read or has no features table return False @@ -59,6 +78,8 @@ def count_passing_tests(project_dir: Path) -> tuple[int, int, int]: """ Count passing, in_progress, and total tests via direct database access. + Uses connection with proper timeout settings for parallel mode safety. + Args: project_dir: Directory containing the project @@ -70,36 +91,35 @@ def count_passing_tests(project_dir: Path) -> tuple[int, int, int]: return 0, 0, 0 try: - conn = sqlite3.connect(db_file) - cursor = conn.cursor() - # Single aggregate query instead of 3 separate COUNT queries - # Handle case where in_progress column doesn't exist yet (legacy DBs) - try: - cursor.execute(""" - SELECT - COUNT(*) as total, - SUM(CASE WHEN passes = 1 THEN 1 ELSE 0 END) as passing, - SUM(CASE WHEN in_progress = 1 THEN 1 ELSE 0 END) as in_progress - FROM features - """) - row = cursor.fetchone() - total = row[0] or 0 - passing = row[1] or 0 - in_progress = row[2] or 0 - except sqlite3.OperationalError: - # Fallback for databases without in_progress column - cursor.execute(""" - SELECT - COUNT(*) as total, - SUM(CASE WHEN passes = 1 THEN 1 ELSE 0 END) as passing - FROM features - """) - row = cursor.fetchone() - total = row[0] or 0 - passing = row[1] or 0 - in_progress = 0 - conn.close() - return passing, in_progress, total + with closing(_get_connection(db_file)) as conn: + cursor = conn.cursor() + # Single aggregate query instead of 3 separate COUNT queries + # Handle case where in_progress column doesn't exist yet (legacy DBs) + try: + cursor.execute(""" + SELECT + COUNT(*) as total, + SUM(CASE WHEN passes = 1 THEN 1 ELSE 0 END) as passing, + SUM(CASE WHEN in_progress = 1 THEN 1 ELSE 0 END) as in_progress + FROM features + """) + row = cursor.fetchone() + total = row[0] or 0 + passing = row[1] or 0 + in_progress = row[2] or 0 + except sqlite3.OperationalError: + # Fallback for databases without in_progress column + cursor.execute(""" + SELECT + COUNT(*) as total, + SUM(CASE WHEN passes = 1 THEN 1 ELSE 0 END) as passing + FROM features + """) + row = cursor.fetchone() + total = row[0] or 0 + passing = row[1] or 0 + in_progress = 0 + return passing, in_progress, total except Exception as e: print(f"[Database error in count_passing_tests: {e}]") return 0, 0, 0 @@ -109,6 +129,8 @@ def get_all_passing_features(project_dir: Path) -> list[dict]: """ Get all passing features for webhook notifications. + Uses connection with proper timeout settings for parallel mode safety. + Args: project_dir: Directory containing the project @@ -120,17 +142,16 @@ def get_all_passing_features(project_dir: Path) -> list[dict]: return [] try: - conn = sqlite3.connect(db_file) - cursor = conn.cursor() - cursor.execute( - "SELECT id, category, name FROM features WHERE passes = 1 ORDER BY priority ASC" - ) - features = [ - {"id": row[0], "category": row[1], "name": row[2]} - for row in cursor.fetchall() - ] - conn.close() - return features + with closing(_get_connection(db_file)) as conn: + cursor = conn.cursor() + cursor.execute( + "SELECT id, category, name FROM features WHERE passes = 1 ORDER BY priority ASC" + ) + features = [ + {"id": row[0], "category": row[1], "name": row[2]} + for row in cursor.fetchall() + ] + return features except Exception: return [] @@ -214,6 +235,47 @@ def send_progress_webhook(passing: int, total: int, project_dir: Path) -> None: ) +def clear_stuck_features(project_dir: Path) -> int: + """ + Clear all in_progress flags from features at agent startup. + + When an agent is stopped mid-work (e.g., user interrupt, crash), + features can be left with in_progress=True and become orphaned. + This function clears those flags so features return to the pending queue. + + Args: + project_dir: Directory containing the project + + Returns: + Number of features that were unstuck + """ + db_file = project_dir / "features.db" + if not db_file.exists(): + return 0 + + try: + with closing(_get_connection(db_file)) as conn: + cursor = conn.cursor() + + # Count how many will be cleared + cursor.execute("SELECT COUNT(*) FROM features WHERE in_progress = 1") + count = cursor.fetchone()[0] + + if count > 0: + # Clear all in_progress flags + cursor.execute("UPDATE features SET in_progress = 0 WHERE in_progress = 1") + conn.commit() + print(f"[Auto-recovery] Cleared {count} stuck feature(s) from previous session") + + return count + except sqlite3.OperationalError: + # Table doesn't exist or doesn't have in_progress column + return 0 + except Exception as e: + print(f"[Warning] Could not clear stuck features: {e}") + return 0 + + def print_session_header(session_num: int, is_initializer: bool) -> None: """Print a formatted header for the session.""" session_type = "INITIALIZER" if is_initializer else "CODING AGENT" diff --git a/quality_gates.py b/quality_gates.py new file mode 100644 index 00000000..f16df843 --- /dev/null +++ b/quality_gates.py @@ -0,0 +1,398 @@ +""" +Quality Gates Module +==================== + +Provides quality checking functionality for the Autocoder system. +Runs lint, type-check, and custom scripts before allowing features +to be marked as passing. + +Supports: +- ESLint/Biome for JavaScript/TypeScript +- ruff/flake8 for Python +- Custom scripts via .autocoder/quality-checks.sh +""" + +import json +import shutil +import subprocess +from datetime import datetime, timezone +from pathlib import Path +from typing import TypedDict + + +class QualityCheckResult(TypedDict): + """Result of a single quality check.""" + name: str + passed: bool + output: str + duration_ms: int + + +class QualityGateResult(TypedDict): + """Result of all quality checks combined.""" + passed: bool + timestamp: str + checks: dict[str, QualityCheckResult] + summary: str + + +def _run_command(cmd: list[str], cwd: Path, timeout: int = 60) -> tuple[int, str, int]: + """ + Run a command and return (exit_code, output, duration_ms). + + Args: + cmd: Command and arguments as a list + cwd: Working directory + timeout: Timeout in seconds + + Returns: + (exit_code, combined_output, duration_ms) + """ + import time + start = time.time() + + try: + result = subprocess.run( + cmd, + cwd=cwd, + capture_output=True, + text=True, + timeout=timeout, + ) + duration_ms = int((time.time() - start) * 1000) + output = result.stdout + result.stderr + return result.returncode, output.strip(), duration_ms + except subprocess.TimeoutExpired: + duration_ms = int((time.time() - start) * 1000) + return 124, f"Command timed out after {timeout}s", duration_ms + except FileNotFoundError: + return 127, f"Command not found: {cmd[0]}", 0 + except Exception as e: + return 1, str(e), 0 + + +def _detect_js_linter(project_dir: Path) -> tuple[str, list[str]] | None: + """ + Detect the JavaScript/TypeScript linter to use. + + Returns: + (name, command) tuple, or None if no linter detected + """ + # Check for ESLint + if (project_dir / "node_modules/.bin/eslint").exists(): + return ("eslint", ["node_modules/.bin/eslint", ".", "--max-warnings=0"]) + + # Check for Biome + if (project_dir / "node_modules/.bin/biome").exists(): + return ("biome", ["node_modules/.bin/biome", "lint", "."]) + + # Check for package.json lint script + package_json = project_dir / "package.json" + if package_json.exists(): + try: + data = json.loads(package_json.read_text()) + scripts = data.get("scripts", {}) + if "lint" in scripts: + return ("npm_lint", ["npm", "run", "lint"]) + except (json.JSONDecodeError, OSError): + pass + + return None + + +def _detect_python_linter(project_dir: Path) -> tuple[str, list[str]] | None: + """ + Detect the Python linter to use. + + Returns: + (name, command) tuple, or None if no linter detected + """ + # Check for ruff + if shutil.which("ruff"): + return ("ruff", ["ruff", "check", "."]) + + # Check for flake8 + if shutil.which("flake8"): + return ("flake8", ["flake8", "."]) + + # Check in virtual environment + venv_ruff = project_dir / "venv/bin/ruff" + if venv_ruff.exists(): + return ("ruff", [str(venv_ruff), "check", "."]) + + venv_flake8 = project_dir / "venv/bin/flake8" + if venv_flake8.exists(): + return ("flake8", [str(venv_flake8), "."]) + + return None + + +def _detect_type_checker(project_dir: Path) -> tuple[str, list[str]] | None: + """ + Detect the type checker to use. + + Returns: + (name, command) tuple, or None if no type checker detected + """ + # TypeScript + if (project_dir / "tsconfig.json").exists(): + if (project_dir / "node_modules/.bin/tsc").exists(): + return ("tsc", ["node_modules/.bin/tsc", "--noEmit"]) + if shutil.which("npx"): + # Use --no-install to fail fast if tsc is not locally installed + # rather than prompting/auto-downloading + return ("tsc", ["npx", "--no-install", "tsc", "--noEmit"]) + + # Python (mypy) + if (project_dir / "pyproject.toml").exists() or (project_dir / "setup.py").exists(): + if shutil.which("mypy"): + return ("mypy", ["mypy", "."]) + venv_mypy = project_dir / "venv/bin/mypy" + if venv_mypy.exists(): + return ("mypy", [str(venv_mypy), "."]) + + return None + + +def run_lint_check(project_dir: Path) -> QualityCheckResult: + """ + Run lint check on the project. + + Automatically detects the appropriate linter based on project type. + + Args: + project_dir: Path to the project directory + + Returns: + QualityCheckResult with lint results + """ + # Try JS/TS linter first + linter = _detect_js_linter(project_dir) + if linter is None: + # Try Python linter + linter = _detect_python_linter(project_dir) + + if linter is None: + return { + "name": "lint", + "passed": True, + "output": "No linter detected, skipping lint check", + "duration_ms": 0, + } + + name, cmd = linter + exit_code, output, duration_ms = _run_command(cmd, project_dir) + + # Truncate output if too long + if len(output) > 5000: + output = output[:5000] + "\n... (truncated)" + + return { + "name": f"lint ({name})", + "passed": exit_code == 0, + "output": output if output else "No issues found", + "duration_ms": duration_ms, + } + + +def run_type_check(project_dir: Path) -> QualityCheckResult: + """ + Run type check on the project. + + Automatically detects the appropriate type checker based on project type. + + Args: + project_dir: Path to the project directory + + Returns: + QualityCheckResult with type check results + """ + checker = _detect_type_checker(project_dir) + + if checker is None: + return { + "name": "type_check", + "passed": True, + "output": "No type checker detected, skipping type check", + "duration_ms": 0, + } + + name, cmd = checker + exit_code, output, duration_ms = _run_command(cmd, project_dir, timeout=120) + + # Truncate output if too long + if len(output) > 5000: + output = output[:5000] + "\n... (truncated)" + + return { + "name": f"type_check ({name})", + "passed": exit_code == 0, + "output": output if output else "No type errors found", + "duration_ms": duration_ms, + } + + +def run_custom_script( + project_dir: Path, + script_path: str | None = None, + explicit_config: bool = False, +) -> QualityCheckResult | None: + """ + Run a custom quality check script. + + Args: + project_dir: Path to the project directory + script_path: Path to the script (relative to project), defaults to .autocoder/quality-checks.sh + explicit_config: If True, user explicitly configured this script, so missing = error + + Returns: + QualityCheckResult, or None if default script doesn't exist + """ + user_configured = script_path is not None or explicit_config + + if script_path is None: + script_path = ".autocoder/quality-checks.sh" + + script_full_path = project_dir / script_path + + if not script_full_path.exists(): + if user_configured: + # User explicitly configured a script that doesn't exist - return error + return { + "name": "custom_script", + "passed": False, + "output": f"Configured script not found: {script_path}", + "duration_ms": 0, + } + # Default script doesn't exist - that's OK, skip silently + return None + + # Make sure it's executable + try: + script_full_path.chmod(0o755) + except OSError: + pass + + exit_code, output, duration_ms = _run_command( + ["bash", str(script_full_path)], + project_dir, + timeout=300, # 5 minutes for custom scripts + ) + + # Truncate output if too long + if len(output) > 10000: + output = output[:10000] + "\n... (truncated)" + + return { + "name": "custom_script", + "passed": exit_code == 0, + "output": output if output else "Script completed successfully", + "duration_ms": duration_ms, + } + + +def verify_quality( + project_dir: Path, + do_lint: bool = True, + do_type_check: bool = True, + do_custom: bool = True, + custom_script_path: str | None = None, +) -> QualityGateResult: + """ + Run all configured quality checks. + + Args: + project_dir: Path to the project directory + do_lint: Whether to run lint check + do_type_check: Whether to run type check + do_custom: Whether to run custom script + custom_script_path: Path to custom script (optional) + + Returns: + QualityGateResult with all check results + """ + checks: dict[str, QualityCheckResult] = {} + all_passed = True + + if do_lint: + lint_result = run_lint_check(project_dir) + checks["lint"] = lint_result + if not lint_result["passed"]: + all_passed = False + + if do_type_check: + type_result = run_type_check(project_dir) + checks["type_check"] = type_result + if not type_result["passed"]: + all_passed = False + + if do_custom: + custom_result = run_custom_script( + project_dir, + custom_script_path, + explicit_config=custom_script_path is not None, + ) + if custom_result is not None: + checks["custom_script"] = custom_result + if not custom_result["passed"]: + all_passed = False + + # Build summary + passed_count = sum(1 for c in checks.values() if c["passed"]) + total_count = len(checks) + failed_names = [name for name, c in checks.items() if not c["passed"]] + + if all_passed: + summary = f"All {total_count} quality checks passed" + else: + summary = f"{passed_count}/{total_count} checks passed. Failed: {', '.join(failed_names)}" + + return { + "passed": all_passed, + "timestamp": datetime.now(timezone.utc).isoformat(), + "checks": checks, + "summary": summary, + } + + +def load_quality_config(project_dir: Path) -> dict: + """ + Load quality gates configuration from .autocoder/config.json. + + Args: + project_dir: Path to the project directory + + Returns: + Quality gates config dict with defaults applied + """ + defaults = { + "enabled": True, + "strict_mode": True, + "checks": { + "lint": True, + "type_check": True, + "unit_tests": False, + "custom_script": None, + }, + } + + config_path = project_dir / ".autocoder" / "config.json" + if not config_path.exists(): + return defaults + + try: + data = json.loads(config_path.read_text()) + quality_config = data.get("quality_gates", {}) + + # Merge with defaults + result = defaults.copy() + for key in ["enabled", "strict_mode"]: + if key in quality_config: + result[key] = quality_config[key] + + if "checks" in quality_config: + result["checks"] = {**defaults["checks"], **quality_config["checks"]} + + return result + except (json.JSONDecodeError, OSError): + return defaults diff --git a/rate_limit_utils.py b/rate_limit_utils.py new file mode 100644 index 00000000..6d817f30 --- /dev/null +++ b/rate_limit_utils.py @@ -0,0 +1,69 @@ +""" +Rate Limit Utilities +==================== + +Shared utilities for detecting and handling API rate limits. +Used by both agent.py (production) and test_agent.py (tests). +""" + +import re +from typing import Optional + +# Rate limit detection patterns (used in both exception messages and response text) +RATE_LIMIT_PATTERNS = [ + "limit reached", + "rate limit", + "rate_limit", + "too many requests", + "quota exceeded", + "please wait", + "try again later", + "429", + "overloaded", +] + + +def parse_retry_after(error_message: str) -> Optional[int]: + """ + Extract retry-after seconds from various error message formats. + + Handles common formats: + - "Retry-After: 60" + - "retry after 60 seconds" + - "try again in 5 seconds" + - "30 seconds remaining" + + Args: + error_message: The error message to parse + + Returns: + Seconds to wait, or None if not parseable. + """ + patterns = [ + r"retry.?after[:\s]+(\d+)\s*(?:seconds?)?", + r"try again in\s+(\d+)\s*(?:seconds?|s\b)", + r"(\d+)\s*seconds?\s*(?:remaining|left|until)", + ] + + for pattern in patterns: + match = re.search(pattern, error_message, re.IGNORECASE) + if match: + return int(match.group(1)) + + return None + + +def is_rate_limit_error(error_message: str) -> bool: + """ + Detect if an error message indicates a rate limit. + + Checks against common rate limit patterns from various API providers. + + Args: + error_message: The error message to check + + Returns: + True if the message indicates a rate limit, False otherwise. + """ + error_lower = error_message.lower() + return any(pattern in error_lower for pattern in RATE_LIMIT_PATTERNS) diff --git a/review_agent.py b/review_agent.py new file mode 100644 index 00000000..12d36f94 --- /dev/null +++ b/review_agent.py @@ -0,0 +1,560 @@ +""" +Review Agent Module +=================== + +Automatic code review agent that analyzes completed features. + +Features: +- Analyzes recent commits after N features complete +- Detects common issues: + - Dead code (unused variables, functions) + - Inconsistent naming + - Missing error handling + - Code duplication + - Security issues +- Creates new features for found issues +- Generates review reports + +Configuration: +- review.enabled: Enable/disable review agent +- review.trigger_after_features: Run review after N features (default: 5) +- review.checks: Which checks to run +""" + +import ast +import json +import logging +import os +import re +import subprocess +from dataclasses import dataclass, field +from datetime import datetime, timezone +from enum import Enum +from pathlib import Path +from typing import Optional + +logger = logging.getLogger(__name__) + + +class IssueSeverity(str, Enum): + """Severity levels for review issues.""" + + ERROR = "error" + WARNING = "warning" + INFO = "info" + STYLE = "style" + + +class IssueCategory(str, Enum): + """Categories of review issues.""" + + DEAD_CODE = "dead_code" + NAMING = "naming" + ERROR_HANDLING = "error_handling" + DUPLICATION = "duplication" + SECURITY = "security" + PERFORMANCE = "performance" + COMPLEXITY = "complexity" + DOCUMENTATION = "documentation" + STYLE = "style" + + +@dataclass +class ReviewIssue: + """A code review issue.""" + + category: IssueCategory + severity: IssueSeverity + title: str + description: str + file_path: str + line_number: Optional[int] = None + code_snippet: Optional[str] = None + suggestion: Optional[str] = None + + def to_dict(self) -> dict: + """Convert to dictionary.""" + result = { + "category": self.category.value, + "severity": self.severity.value, + "title": self.title, + "description": self.description, + "file_path": self.file_path, + } + if self.line_number: + result["line_number"] = self.line_number + if self.code_snippet: + result["code_snippet"] = self.code_snippet + if self.suggestion: + result["suggestion"] = self.suggestion + return result + + def to_feature(self) -> dict: + """Convert to a feature for tracking.""" + return { + "category": "Code Review", + "name": self.title, + "description": self.description, + "steps": [ + f"Review issue in {self.file_path}" + (f":{self.line_number}" if self.line_number else ""), + self.suggestion or "Fix the identified issue", + "Verify the fix works correctly", + ], + } + + +@dataclass +class ReviewReport: + """Complete review report.""" + + project_dir: str + review_time: str + commits_reviewed: list[str] = field(default_factory=list) + files_reviewed: list[str] = field(default_factory=list) + issues: list[ReviewIssue] = field(default_factory=list) + summary: dict = field(default_factory=dict) + + def to_dict(self) -> dict: + """Convert to dictionary.""" + return { + "project_dir": self.project_dir, + "review_time": self.review_time, + "commits_reviewed": self.commits_reviewed, + "files_reviewed": self.files_reviewed, + "issues": [i.to_dict() for i in self.issues], + "summary": { + "total_issues": len(self.issues), + "by_severity": { + s.value: len([i for i in self.issues if i.severity == s]) + for s in IssueSeverity + }, + "by_category": { + c.value: len([i for i in self.issues if i.category == c]) + for c in IssueCategory + }, + }, + } + + +class ReviewAgent: + """ + Code review agent for automatic quality checks. + + Usage: + agent = ReviewAgent(project_dir) + report = agent.review() + features = agent.get_issues_as_features() + """ + + def __init__( + self, + project_dir: Path, + check_dead_code: bool = True, + check_naming: bool = True, + check_error_handling: bool = True, + check_security: bool = True, + check_complexity: bool = True, + ): + self.project_dir = Path(project_dir) + self.check_dead_code = check_dead_code + self.check_naming = check_naming + self.check_error_handling = check_error_handling + self.check_security = check_security + self.check_complexity = check_complexity + self.issues: list[ReviewIssue] = [] + + def review( + self, + commits: Optional[list[str]] = None, + files: Optional[list[str]] = None, + ) -> ReviewReport: + """ + Run code review. + + Args: + commits: Specific commits to review (default: recent commits) + files: Specific files to review (default: changed files) + + Returns: + ReviewReport with all findings + """ + self.issues = [] + + # Get files to review + if files: + files_to_review = [self.project_dir / f for f in files] + elif commits: + files_to_review = self._get_changed_files(commits) + else: + # Review all source files + files_to_review = list(self._iter_source_files()) + + # Run checks + for file_path in files_to_review: + if not file_path.exists(): + continue + + try: + content = file_path.read_text(errors="ignore") + + if file_path.suffix == ".py": + self._review_python_file(file_path, content) + elif file_path.suffix in {".js", ".ts", ".jsx", ".tsx"}: + self._review_javascript_file(file_path, content) + except Exception as e: + logger.warning(f"Error reviewing {file_path}: {e}") + + # Generate report + return ReviewReport( + project_dir=str(self.project_dir), + review_time=datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + commits_reviewed=commits or [], + files_reviewed=[str(f.relative_to(self.project_dir)) for f in files_to_review if f.exists()], + issues=self.issues, + ) + + def _iter_source_files(self): + """Iterate over source files in project.""" + extensions = {".py", ".js", ".ts", ".jsx", ".tsx"} + skip_dirs = {"node_modules", "venv", ".venv", "__pycache__", ".git", "dist", "build"} + + for root, dirs, files in os.walk(self.project_dir): + dirs[:] = [d for d in dirs if d not in skip_dirs] + for file in files: + if Path(file).suffix in extensions: + yield Path(root) / file + + def _get_changed_files(self, commits: list[str]) -> list[Path]: + """Get files changed in specified commits.""" + files = set() + for commit in commits: + try: + result = subprocess.run( + ["git", "diff-tree", "--no-commit-id", "--name-only", "-r", commit], + cwd=self.project_dir, + capture_output=True, + text=True, + ) + for line in result.stdout.strip().split("\n"): + if line: + files.add(self.project_dir / line) + except Exception: + pass + return list(files) + + def _review_python_file(self, file_path: Path, content: str) -> None: + """Review a Python file.""" + relative_path = str(file_path.relative_to(self.project_dir)) + + # Parse AST + try: + tree = ast.parse(content) + except SyntaxError: + return + + # Check for dead code (unused imports) + if self.check_dead_code: + self._check_python_unused_imports(tree, content, relative_path) + + # Check naming conventions + if self.check_naming: + self._check_python_naming(tree, relative_path) + + # Check error handling + if self.check_error_handling: + self._check_python_error_handling(tree, content, relative_path) + + # Check complexity + if self.check_complexity: + self._check_python_complexity(tree, relative_path) + + # Check security patterns + if self.check_security: + self._check_security_patterns(content, relative_path) + + def _check_python_unused_imports(self, tree: ast.AST, content: str, file_path: str) -> None: + """Check for unused imports in Python.""" + imports = [] + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + name = alias.asname or alias.name.split(".")[0] + imports.append((name, node.lineno)) + elif isinstance(node, ast.ImportFrom): + for alias in node.names: + if alias.name != "*": + name = alias.asname or alias.name + imports.append((name, node.lineno)) + + # Simple check: see if import name appears in rest of file + for name, lineno in imports: + # Count occurrences (excluding import lines) + pattern = rf"\b{re.escape(name)}\b" + matches = list(re.finditer(pattern, content)) + # If only appears once (the import), likely unused + if len(matches) <= 1: + self.issues.append( + ReviewIssue( + category=IssueCategory.DEAD_CODE, + severity=IssueSeverity.WARNING, + title=f"Possibly unused import: {name}", + description=f"Import '{name}' may be unused in this file", + file_path=file_path, + line_number=lineno, + suggestion="Remove unused import if not needed", + ) + ) + + def _check_python_naming(self, tree: ast.AST, file_path: str) -> None: + """Check Python naming conventions.""" + for node in ast.walk(tree): + # Check class names (should be PascalCase) + if isinstance(node, ast.ClassDef): + if not re.match(r"^[A-Z][a-zA-Z0-9]*$", node.name): + self.issues.append( + ReviewIssue( + category=IssueCategory.NAMING, + severity=IssueSeverity.STYLE, + title=f"Class name not PascalCase: {node.name}", + description=f"Class '{node.name}' should use PascalCase naming", + file_path=file_path, + line_number=node.lineno, + suggestion="Rename to follow PascalCase convention", + ) + ) + + # Check function names (should be snake_case) + elif isinstance(node, ast.FunctionDef): + if not node.name.startswith("_") and not re.match(r"^[a-z_][a-z0-9_]*$", node.name): + if not re.match(r"^__\w+__$", node.name): # Skip dunder methods + self.issues.append( + ReviewIssue( + category=IssueCategory.NAMING, + severity=IssueSeverity.STYLE, + title=f"Function name not snake_case: {node.name}", + description=f"Function '{node.name}' should use snake_case naming", + file_path=file_path, + line_number=node.lineno, + suggestion="Rename to follow snake_case convention", + ) + ) + + def _check_python_error_handling(self, tree: ast.AST, content: str, file_path: str) -> None: + """Check error handling in Python.""" + for node in ast.walk(tree): + # Check for bare except clauses + if isinstance(node, ast.ExceptHandler): + if node.type is None: + self.issues.append( + ReviewIssue( + category=IssueCategory.ERROR_HANDLING, + severity=IssueSeverity.WARNING, + title="Bare except clause", + description="Bare 'except:' catches all exceptions including KeyboardInterrupt", + file_path=file_path, + line_number=node.lineno, + suggestion="Use 'except Exception:' or catch specific exceptions", + ) + ) + + # Check for pass in except + if isinstance(node, ast.ExceptHandler): + if len(node.body) == 1 and isinstance(node.body[0], ast.Pass): + self.issues.append( + ReviewIssue( + category=IssueCategory.ERROR_HANDLING, + severity=IssueSeverity.WARNING, + title="Empty except handler", + description="Exception is caught but silently ignored", + file_path=file_path, + line_number=node.lineno, + suggestion="Add logging or proper error handling", + ) + ) + + def _check_python_complexity(self, tree: ast.AST, file_path: str) -> None: + """Check code complexity in Python.""" + for node in ast.walk(tree): + if isinstance(node, ast.FunctionDef): + # Count lines in function + if hasattr(node, "end_lineno") and node.end_lineno: + lines = node.end_lineno - node.lineno + if lines > 50: + self.issues.append( + ReviewIssue( + category=IssueCategory.COMPLEXITY, + severity=IssueSeverity.INFO, + title=f"Long function: {node.name} ({lines} lines)", + description=f"Function '{node.name}' is {lines} lines long", + file_path=file_path, + line_number=node.lineno, + suggestion="Consider breaking into smaller functions", + ) + ) + + # Count parameters + num_args = len(node.args.args) + len(node.args.posonlyargs) + len(node.args.kwonlyargs) + if num_args > 7: + self.issues.append( + ReviewIssue( + category=IssueCategory.COMPLEXITY, + severity=IssueSeverity.INFO, + title=f"Too many parameters: {node.name} ({num_args})", + description=f"Function '{node.name}' has {num_args} parameters", + file_path=file_path, + line_number=node.lineno, + suggestion="Consider using a config object or dataclass", + ) + ) + + def _check_security_patterns(self, content: str, file_path: str) -> None: + """Check for common security issues.""" + lines = content.split("\n") + + patterns = [ + (r"eval\s*\(", "Use of eval()", "Avoid eval() - it can execute arbitrary code"), + (r"exec\s*\(", "Use of exec()", "Avoid exec() - it can execute arbitrary code"), + (r"shell\s*=\s*True", "subprocess with shell=True", "Avoid shell=True to prevent injection"), + (r"pickle\.load", "Use of pickle.load", "Pickle can execute arbitrary code"), + ] + + for i, line in enumerate(lines, 1): + for pattern, title, suggestion in patterns: + if re.search(pattern, line): + self.issues.append( + ReviewIssue( + category=IssueCategory.SECURITY, + severity=IssueSeverity.WARNING, + title=title, + description="Potential security issue detected", + file_path=file_path, + line_number=i, + code_snippet=line.strip()[:80], + suggestion=suggestion, + ) + ) + + def _review_javascript_file(self, file_path: Path, content: str) -> None: + """Review a JavaScript/TypeScript file.""" + relative_path = str(file_path.relative_to(self.project_dir)) + lines = content.split("\n") + + # Check for console.log statements + for i, line in enumerate(lines, 1): + if re.search(r"console\.(log|debug|info)\s*\(", line): + # Skip if in comment + if not line.strip().startswith("//"): + self.issues.append( + ReviewIssue( + category=IssueCategory.DEAD_CODE, + severity=IssueSeverity.INFO, + title="console.log statement", + description="Debug logging should be removed in production", + file_path=relative_path, + line_number=i, + code_snippet=line.strip()[:80], + suggestion="Remove or use proper logging", + ) + ) + + # Check for TODO/FIXME comments + for i, line in enumerate(lines, 1): + if re.search(r"(TODO|FIXME|XXX|HACK):", line, re.IGNORECASE): + self.issues.append( + ReviewIssue( + category=IssueCategory.DOCUMENTATION, + severity=IssueSeverity.INFO, + title="TODO/FIXME comment found", + description="Outstanding work marked in code", + file_path=relative_path, + line_number=i, + code_snippet=line.strip()[:80], + suggestion="Address the TODO or create a tracking issue", + ) + ) + + # Check for security patterns + if self.check_security: + self._check_js_security_patterns(content, relative_path) + + def _check_js_security_patterns(self, content: str, file_path: str) -> None: + """Check JavaScript security patterns.""" + lines = content.split("\n") + + patterns = [ + (r"eval\s*\(", "Use of eval()", "Avoid eval() - use JSON.parse() or Function()"), + (r"innerHTML\s*=", "Direct innerHTML assignment", "Use textContent or sanitize HTML"), + (r"dangerouslySetInnerHTML", "dangerouslySetInnerHTML usage", "Ensure content is sanitized"), + ] + + for i, line in enumerate(lines, 1): + for pattern, title, suggestion in patterns: + if re.search(pattern, line): + self.issues.append( + ReviewIssue( + category=IssueCategory.SECURITY, + severity=IssueSeverity.WARNING, + title=title, + description="Potential security issue detected", + file_path=file_path, + line_number=i, + code_snippet=line.strip()[:80], + suggestion=suggestion, + ) + ) + + def get_issues_as_features(self) -> list[dict]: + """ + Convert significant issues to features for tracking. + + Only creates features for errors and warnings, not info/style. + """ + features = [] + seen = set() + + for issue in self.issues: + if issue.severity in {IssueSeverity.ERROR, IssueSeverity.WARNING}: + # Deduplicate by title + if issue.title not in seen: + seen.add(issue.title) + features.append(issue.to_feature()) + + return features + + def save_report(self, report: ReviewReport) -> Path: + """Save review report to file.""" + reports_dir = self.project_dir / ".autocoder" / "review-reports" + reports_dir.mkdir(parents=True, exist_ok=True) + + timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") + report_path = reports_dir / f"review_{timestamp}.json" + + with open(report_path, "w") as f: + json.dump(report.to_dict(), f, indent=2) + + return report_path + + +def run_review( + project_dir: Path, + commits: Optional[list[str]] = None, + save_report: bool = True, +) -> ReviewReport: + """ + Run code review on a project. + + Args: + project_dir: Project directory + commits: Specific commits to review + save_report: Whether to save the report + + Returns: + ReviewReport with findings + """ + agent = ReviewAgent(project_dir) + report = agent.review(commits=commits) + + if save_report: + agent.save_report(report) + + return report diff --git a/security.py b/security.py index 44507a4a..af19a1e3 100644 --- a/security.py +++ b/security.py @@ -6,6 +6,7 @@ Uses an allowlist approach - only explicitly permitted commands can run. """ +import logging import os import re import shlex @@ -14,6 +15,8 @@ import yaml +logger = logging.getLogger(__name__) + # Regex pattern for valid pkill process names (no regex metacharacters allowed) # Matches alphanumeric names with dots, underscores, and hyphens VALID_PROCESS_NAME_PATTERN = re.compile(r"^[A-Za-z0-9._-]+$") @@ -106,7 +109,7 @@ "az", # Container and orchestration "kubectl", - "docker-compose", + # Note: docker-compose removed - commonly needed for local dev environments } @@ -140,6 +143,45 @@ def split_command_segments(command_string: str) -> list[str]: return result +def _extract_primary_command(segment: str) -> str | None: + """ + Fallback command extraction when shlex fails. + + Extracts the first word that looks like a command, handling cases + like complex docker exec commands with nested quotes. + + Args: + segment: The command segment to parse + + Returns: + The primary command name, or None if extraction fails + """ + # Remove leading whitespace + segment = segment.lstrip() + + if not segment: + return None + + # Skip env var assignments at start (VAR=value cmd) + words = segment.split() + while words and "=" in words[0] and not words[0].startswith("="): + words = words[1:] + + if not words: + return None + + # Extract first token (the command) + first_word = words[0] + + # Match valid command characters (alphanumeric, dots, underscores, hyphens, slashes) + match = re.match(r"^([a-zA-Z0-9_./-]+)", first_word) + if match: + cmd = match.group(1) + return os.path.basename(cmd) + + return None + + def extract_commands(command_string: str) -> list[str]: """ Extract command names from a shell command string. @@ -156,7 +198,7 @@ def extract_commands(command_string: str) -> list[str]: commands = [] # shlex doesn't treat ; as a separator, so we need to pre-process - import re + # (re is already imported at module level) # Split on semicolons that aren't inside quotes (simple heuristic) # This handles common cases like "echo hello; ls" @@ -171,8 +213,11 @@ def extract_commands(command_string: str) -> list[str]: tokens = shlex.split(segment) except ValueError: # Malformed command (unclosed quotes, etc.) - # Return empty to trigger block (fail-safe) - return [] + # Try fallback extraction instead of blocking entirely + fallback_cmd = _extract_primary_command(segment) + if fallback_cmd: + commands.append(fallback_cmd) + continue if not tokens: continue @@ -444,58 +489,74 @@ def load_org_config() -> Optional[dict]: config = yaml.safe_load(f) if not config: + logger.warning(f"Org config at {config_path} is empty") return None # Validate structure if not isinstance(config, dict): + logger.warning(f"Org config at {config_path} must be a YAML dictionary") return None if "version" not in config: + logger.warning(f"Org config at {config_path} missing required 'version' field") return None # Validate allowed_commands if present if "allowed_commands" in config: allowed = config["allowed_commands"] if not isinstance(allowed, list): + logger.warning(f"Org config at {config_path}: 'allowed_commands' must be a list") return None - for cmd in allowed: + for i, cmd in enumerate(allowed): if not isinstance(cmd, dict): + logger.warning(f"Org config at {config_path}: allowed_commands[{i}] must be a dict") return None if "name" not in cmd: + logger.warning(f"Org config at {config_path}: allowed_commands[{i}] missing 'name'") return None # Validate that name is a non-empty string if not isinstance(cmd["name"], str) or cmd["name"].strip() == "": + logger.warning(f"Org config at {config_path}: allowed_commands[{i}] has invalid 'name'") return None # Validate blocked_commands if present if "blocked_commands" in config: blocked = config["blocked_commands"] if not isinstance(blocked, list): + logger.warning(f"Org config at {config_path}: 'blocked_commands' must be a list") return None - for cmd in blocked: + for i, cmd in enumerate(blocked): if not isinstance(cmd, str): + logger.warning(f"Org config at {config_path}: blocked_commands[{i}] must be a string") return None # Validate pkill_processes if present if "pkill_processes" in config: processes = config["pkill_processes"] if not isinstance(processes, list): + logger.warning(f"Org config at {config_path}: 'pkill_processes' must be a list") return None # Normalize and validate each process name against safe pattern normalized = [] - for proc in processes: + for i, proc in enumerate(processes): if not isinstance(proc, str): + logger.warning(f"Org config at {config_path}: pkill_processes[{i}] must be a string") return None proc = proc.strip() # Block empty strings and regex metacharacters if not proc or not VALID_PROCESS_NAME_PATTERN.fullmatch(proc): + logger.warning(f"Org config at {config_path}: pkill_processes[{i}] has invalid value '{proc}'") return None normalized.append(proc) config["pkill_processes"] = normalized return config - except (yaml.YAMLError, IOError, OSError): + except yaml.YAMLError as e: + logger.warning(f"Failed to parse org config at {config_path}: {e}") + return None + except (IOError, OSError) as e: + logger.warning(f"Failed to read org config at {config_path}: {e}") return None @@ -509,7 +570,7 @@ def load_project_commands(project_dir: Path) -> Optional[dict]: Returns: Dict with parsed YAML config, or None if file doesn't exist or is invalid """ - config_path = project_dir / ".autocoder" / "allowed_commands.yaml" + config_path = project_dir.resolve() / ".autocoder" / "allowed_commands.yaml" if not config_path.exists(): return None @@ -519,53 +580,68 @@ def load_project_commands(project_dir: Path) -> Optional[dict]: config = yaml.safe_load(f) if not config: + logger.warning(f"Project config at {config_path} is empty") return None # Validate structure if not isinstance(config, dict): + logger.warning(f"Project config at {config_path} must be a YAML dictionary") return None if "version" not in config: + logger.warning(f"Project config at {config_path} missing required 'version' field") return None commands = config.get("commands", []) if not isinstance(commands, list): + logger.warning(f"Project config at {config_path}: 'commands' must be a list") return None # Enforce 100 command limit if len(commands) > 100: + logger.warning(f"Project config at {config_path} exceeds 100 command limit ({len(commands)} commands)") return None # Validate each command entry - for cmd in commands: + for i, cmd in enumerate(commands): if not isinstance(cmd, dict): + logger.warning(f"Project config at {config_path}: commands[{i}] must be a dict") return None if "name" not in cmd: + logger.warning(f"Project config at {config_path}: commands[{i}] missing 'name'") return None - # Validate name is a string - if not isinstance(cmd["name"], str): + # Validate name is a non-empty string + if not isinstance(cmd["name"], str) or cmd["name"].strip() == "": + logger.warning(f"Project config at {config_path}: commands[{i}] has invalid 'name'") return None # Validate pkill_processes if present if "pkill_processes" in config: processes = config["pkill_processes"] if not isinstance(processes, list): + logger.warning(f"Project config at {config_path}: 'pkill_processes' must be a list") return None # Normalize and validate each process name against safe pattern normalized = [] - for proc in processes: + for i, proc in enumerate(processes): if not isinstance(proc, str): + logger.warning(f"Project config at {config_path}: pkill_processes[{i}] must be a string") return None proc = proc.strip() # Block empty strings and regex metacharacters if not proc or not VALID_PROCESS_NAME_PATTERN.fullmatch(proc): + logger.warning(f"Project config at {config_path}: pkill_processes[{i}] has invalid value '{proc}'") return None normalized.append(proc) config["pkill_processes"] = normalized return config - except (yaml.YAMLError, IOError, OSError): + except yaml.YAMLError as e: + logger.warning(f"Failed to parse project config at {config_path}: {e}") + return None + except (IOError, OSError) as e: + logger.warning(f"Failed to read project config at {config_path}: {e}") return None diff --git a/security_scanner.py b/security_scanner.py new file mode 100644 index 00000000..cf8b8da5 --- /dev/null +++ b/security_scanner.py @@ -0,0 +1,696 @@ +""" +Security Scanner Module +======================= + +Detect vulnerabilities in generated code and dependencies. + +Features: +- Dependency scanning (npm audit, pip-audit/safety) +- Secret detection (API keys, passwords, tokens) +- Code vulnerability patterns (SQL injection, XSS, command injection) +- OWASP Top 10 pattern matching + +Integration: +- Can be run standalone or as part of quality gates +- Results stored in project's .autocoder/security-reports/ +""" + +import json +import os +import re +import shutil +import subprocess +from dataclasses import dataclass, field +from datetime import datetime, timezone +from enum import Enum +from pathlib import Path +from typing import Optional + + +class Severity(str, Enum): + """Vulnerability severity levels.""" + + CRITICAL = "critical" + HIGH = "high" + MEDIUM = "medium" + LOW = "low" + INFO = "info" + + +class VulnerabilityType(str, Enum): + """Types of vulnerabilities detected.""" + + DEPENDENCY = "dependency" + SECRET = "secret" + SQL_INJECTION = "sql_injection" + XSS = "xss" + COMMAND_INJECTION = "command_injection" + PATH_TRAVERSAL = "path_traversal" + INSECURE_CRYPTO = "insecure_crypto" + HARDCODED_CREDENTIAL = "hardcoded_credential" + SENSITIVE_DATA_EXPOSURE = "sensitive_data_exposure" + OTHER = "other" + + +@dataclass +class Vulnerability: + """A detected vulnerability.""" + + type: VulnerabilityType + severity: Severity + title: str + description: str + file_path: Optional[str] = None + line_number: Optional[int] = None + code_snippet: Optional[str] = None + recommendation: Optional[str] = None + cwe_id: Optional[str] = None + package_name: Optional[str] = None + package_version: Optional[str] = None + + def to_dict(self) -> dict: + """Convert to dictionary.""" + result = { + "type": self.type.value, + "severity": self.severity.value, + "title": self.title, + "description": self.description, + } + if self.file_path: + result["file_path"] = self.file_path + if self.line_number: + result["line_number"] = self.line_number + if self.code_snippet: + result["code_snippet"] = self.code_snippet + if self.recommendation: + result["recommendation"] = self.recommendation + if self.cwe_id: + result["cwe_id"] = self.cwe_id + if self.package_name: + result["package_name"] = self.package_name + if self.package_version: + result["package_version"] = self.package_version + return result + + +@dataclass +class ScanResult: + """Result of a security scan.""" + + project_dir: str + scan_time: str + vulnerabilities: list[Vulnerability] = field(default_factory=list) + summary: dict = field(default_factory=dict) + scans_run: list[str] = field(default_factory=list) + + def to_dict(self) -> dict: + """Convert to dictionary.""" + return { + "project_dir": self.project_dir, + "scan_time": self.scan_time, + "vulnerabilities": [v.to_dict() for v in self.vulnerabilities], + "summary": self.summary, + "scans_run": self.scans_run, + "total_issues": len(self.vulnerabilities), + "by_severity": { + "critical": len([v for v in self.vulnerabilities if v.severity == Severity.CRITICAL]), + "high": len([v for v in self.vulnerabilities if v.severity == Severity.HIGH]), + "medium": len([v for v in self.vulnerabilities if v.severity == Severity.MEDIUM]), + "low": len([v for v in self.vulnerabilities if v.severity == Severity.LOW]), + "info": len([v for v in self.vulnerabilities if v.severity == Severity.INFO]), + }, + } + + +# ============================================================================ +# Secret Patterns +# ============================================================================ + +SECRET_PATTERNS = [ + # API Keys + ( + r'(?i)(api[_-]?key|apikey)\s*[=:]\s*["\']?([a-zA-Z0-9_\-]{20,})["\']?', + "API Key Detected", + Severity.HIGH, + "CWE-798", + ), + # AWS Keys + ( + r'(?i)(AKIA[0-9A-Z]{16})', + "AWS Access Key ID", + Severity.CRITICAL, + "CWE-798", + ), + ( + r'(?i)aws[_-]?secret[_-]?access[_-]?key\s*[=:]\s*["\']?([a-zA-Z0-9/+=]{40})["\']?', + "AWS Secret Access Key", + Severity.CRITICAL, + "CWE-798", + ), + # Private Keys + ( + r'-----BEGIN (RSA |EC |DSA )?PRIVATE KEY-----', + "Private Key Detected", + Severity.CRITICAL, + "CWE-321", + ), + # Passwords + ( + r'(?i)(password|passwd|pwd)\s*[=:]\s*["\']([^"\']{8,})["\']', + "Hardcoded Password", + Severity.HIGH, + "CWE-798", + ), + # Generic Secrets + ( + r'(?i)(secret|token|auth)[_-]?(key|token)?\s*[=:]\s*["\']?([a-zA-Z0-9_\-]{20,})["\']?', + "Secret/Token Detected", + Severity.HIGH, + "CWE-798", + ), + # Database Connection Strings + ( + r'(?i)(mongodb|postgres|mysql|redis)://[^"\'\s]+:[^"\'\s]+@', + "Database Connection String with Credentials", + Severity.HIGH, + "CWE-798", + ), + # JWT Tokens + ( + r'eyJ[a-zA-Z0-9_-]*\.eyJ[a-zA-Z0-9_-]*\.[a-zA-Z0-9_-]*', + "JWT Token Detected", + Severity.MEDIUM, + "CWE-200", + ), + # GitHub Tokens + ( + r'gh[pousr]_[A-Za-z0-9_]{36,}', + "GitHub Token Detected", + Severity.CRITICAL, + "CWE-798", + ), + # Slack Tokens + ( + r'xox[baprs]-[0-9]{10,13}-[0-9]{10,13}[a-zA-Z0-9-]*', + "Slack Token Detected", + Severity.HIGH, + "CWE-798", + ), +] + +# ============================================================================ +# Code Vulnerability Patterns +# ============================================================================ + +CODE_PATTERNS = [ + # SQL Injection + ( + r'(?i)execute\s*\(\s*["\'].*\%.*["\'].*%', + "Potential SQL Injection (string formatting)", + VulnerabilityType.SQL_INJECTION, + Severity.HIGH, + "CWE-89", + "Use parameterized queries instead of string formatting", + ), + ( + r'(?i)(cursor\.execute|db\.execute|connection\.execute)\s*\(\s*f["\']', + "Potential SQL Injection (f-string)", + VulnerabilityType.SQL_INJECTION, + Severity.HIGH, + "CWE-89", + "Use parameterized queries instead of f-strings", + ), + ( + r'(?i)query\s*=\s*["\']SELECT.*\+', + "Potential SQL Injection (string concatenation)", + VulnerabilityType.SQL_INJECTION, + Severity.HIGH, + "CWE-89", + "Use parameterized queries instead of string concatenation", + ), + # XSS + ( + r'(?i)innerHTML\s*=\s*[^"\']*\+', + "Potential XSS (innerHTML with concatenation)", + VulnerabilityType.XSS, + Severity.HIGH, + "CWE-79", + "Use textContent or sanitize HTML before setting innerHTML", + ), + ( + r'(?i)document\.write\s*\(', + "Potential XSS (document.write)", + VulnerabilityType.XSS, + Severity.MEDIUM, + "CWE-79", + "Avoid document.write, use DOM manipulation instead", + ), + ( + r'(?i)dangerouslySetInnerHTML', + "React dangerouslySetInnerHTML usage", + VulnerabilityType.XSS, + Severity.MEDIUM, + "CWE-79", + "Ensure content is properly sanitized before using dangerouslySetInnerHTML", + ), + # Command Injection + ( + r'(?i)(subprocess\.call|subprocess\.run|os\.system|os\.popen)\s*\([^)]*\+', + "Potential Command Injection (string concatenation)", + VulnerabilityType.COMMAND_INJECTION, + Severity.CRITICAL, + "CWE-78", + "Use subprocess with list arguments and avoid shell=True", + ), + ( + r'(?i)shell\s*=\s*True', + "Subprocess with shell=True", + VulnerabilityType.COMMAND_INJECTION, + Severity.MEDIUM, + "CWE-78", + "Avoid shell=True, use list arguments instead", + ), + ( + r'(?i)exec\s*\(\s*[^"\']*\+', + "Potential Code Injection (exec with concatenation)", + VulnerabilityType.COMMAND_INJECTION, + Severity.CRITICAL, + "CWE-94", + "Avoid using exec with user-controlled input", + ), + ( + r'(?i)eval\s*\(\s*[^"\']*\+', + "Potential Code Injection (eval with concatenation)", + VulnerabilityType.COMMAND_INJECTION, + Severity.CRITICAL, + "CWE-94", + "Avoid using eval with user-controlled input", + ), + # Path Traversal + ( + r'(?i)(open|read|write)\s*\([^)]*\+[^)]*\)', + "Potential Path Traversal (file operation with concatenation)", + VulnerabilityType.PATH_TRAVERSAL, + Severity.MEDIUM, + "CWE-22", + "Validate and sanitize file paths before use", + ), + # Insecure Crypto + ( + r'(?i)(md5|sha1)\s*\(', + "Weak Cryptographic Hash (MD5/SHA1)", + VulnerabilityType.INSECURE_CRYPTO, + Severity.LOW, + "CWE-328", + "Use SHA-256 or stronger for security-sensitive operations", + ), + ( + r'(?i)random\.random\s*\(', + "Insecure Random Number Generator", + VulnerabilityType.INSECURE_CRYPTO, + Severity.LOW, + "CWE-330", + "Use secrets module for security-sensitive random values", + ), + # Sensitive Data + ( + r'(?i)console\.(log|info|debug)\s*\([^)]*password', + "Password logged to console", + VulnerabilityType.SENSITIVE_DATA_EXPOSURE, + Severity.MEDIUM, + "CWE-532", + "Remove sensitive data from log statements", + ), + ( + r'(?i)print\s*\([^)]*password', + "Password printed to output", + VulnerabilityType.SENSITIVE_DATA_EXPOSURE, + Severity.MEDIUM, + "CWE-532", + "Remove sensitive data from print statements", + ), +] + + +class SecurityScanner: + """ + Security scanner for detecting vulnerabilities in code and dependencies. + + Usage: + scanner = SecurityScanner(project_dir) + result = scanner.scan() + print(f"Found {len(result.vulnerabilities)} issues") + """ + + def __init__(self, project_dir: Path): + self.project_dir = Path(project_dir) + + def scan( + self, + scan_dependencies: bool = True, + scan_secrets: bool = True, + scan_code: bool = True, + save_report: bool = True, + ) -> ScanResult: + """ + Run security scan on the project. + + Args: + scan_dependencies: Run npm audit / pip-audit + scan_secrets: Scan for hardcoded secrets + scan_code: Scan for code vulnerabilities + save_report: Save report to .autocoder/security-reports/ + + Returns: + ScanResult with all findings + """ + result = ScanResult( + project_dir=str(self.project_dir), + scan_time=datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + ) + + if scan_dependencies: + self._scan_dependencies(result) + + if scan_secrets: + self._scan_secrets(result) + + if scan_code: + self._scan_code_patterns(result) + + # Generate summary + result.summary = { + "total_issues": len(result.vulnerabilities), + "critical": len([v for v in result.vulnerabilities if v.severity == Severity.CRITICAL]), + "high": len([v for v in result.vulnerabilities if v.severity == Severity.HIGH]), + "medium": len([v for v in result.vulnerabilities if v.severity == Severity.MEDIUM]), + "low": len([v for v in result.vulnerabilities if v.severity == Severity.LOW]), + "has_critical_or_high": any( + v.severity in (Severity.CRITICAL, Severity.HIGH) + for v in result.vulnerabilities + ), + } + + if save_report: + self._save_report(result) + + return result + + def _scan_dependencies(self, result: ScanResult) -> None: + """Scan dependencies for known vulnerabilities.""" + # Check for npm + if (self.project_dir / "package.json").exists(): + self._run_npm_audit(result) + + # Check for Python + if (self.project_dir / "requirements.txt").exists() or ( + self.project_dir / "pyproject.toml" + ).exists(): + self._run_pip_audit(result) + + def _run_npm_audit(self, result: ScanResult) -> None: + """Run npm audit and parse results.""" + result.scans_run.append("npm_audit") + + try: + proc = subprocess.run( + ["npm", "audit", "--json"], + cwd=self.project_dir, + capture_output=True, + text=True, + timeout=120, + ) + + if proc.stdout: + try: + audit_data = json.loads(proc.stdout) + + # Parse vulnerabilities from npm audit output + vulns = audit_data.get("vulnerabilities", {}) + for pkg_name, pkg_info in vulns.items(): + severity_str = pkg_info.get("severity", "medium") + severity_map = { + "critical": Severity.CRITICAL, + "high": Severity.HIGH, + "moderate": Severity.MEDIUM, + "low": Severity.LOW, + "info": Severity.INFO, + } + severity = severity_map.get(severity_str, Severity.MEDIUM) + + via = pkg_info.get("via", []) + description = "" + if via and isinstance(via[0], dict): + description = via[0].get("title", "") + elif via and isinstance(via[0], str): + description = f"Vulnerable through {via[0]}" + + result.vulnerabilities.append( + Vulnerability( + type=VulnerabilityType.DEPENDENCY, + severity=severity, + title=f"Vulnerable dependency: {pkg_name}", + description=description or "Known vulnerability in package", + package_name=pkg_name, + package_version=pkg_info.get("range"), + recommendation=f"Run: npm update {pkg_name}", + ) + ) + except json.JSONDecodeError: + pass + + except subprocess.TimeoutExpired: + pass + except FileNotFoundError: + pass + + def _run_pip_audit(self, result: ScanResult) -> None: + """Run pip-audit and parse results.""" + result.scans_run.append("pip_audit") + + # Try pip-audit first + pip_audit_path = shutil.which("pip-audit") + if pip_audit_path: + try: + proc = subprocess.run( + ["pip-audit", "--format", "json", "-r", "requirements.txt"], + cwd=self.project_dir, + capture_output=True, + text=True, + timeout=120, + ) + + if proc.stdout: + try: + vulns = json.loads(proc.stdout) + for vuln in vulns: + severity_map = { + "CRITICAL": Severity.CRITICAL, + "HIGH": Severity.HIGH, + "MEDIUM": Severity.MEDIUM, + "LOW": Severity.LOW, + } + result.vulnerabilities.append( + Vulnerability( + type=VulnerabilityType.DEPENDENCY, + severity=severity_map.get( + vuln.get("severity", "MEDIUM"), Severity.MEDIUM + ), + title=f"Vulnerable dependency: {vuln.get('name')}", + description=vuln.get("description", ""), + package_name=vuln.get("name"), + package_version=vuln.get("version"), + cwe_id=vuln.get("id"), + recommendation=f"Upgrade to {vuln.get('fix_versions', ['latest'])[0] if vuln.get('fix_versions') else 'latest'}", + ) + ) + except json.JSONDecodeError: + pass + except (subprocess.TimeoutExpired, FileNotFoundError): + pass + + # Try safety as fallback + safety_path = shutil.which("safety") + if safety_path and not any( + v.type == VulnerabilityType.DEPENDENCY + for v in result.vulnerabilities + if v.package_name + ): + try: + proc = subprocess.run( + ["safety", "check", "--json", "-r", "requirements.txt"], + cwd=self.project_dir, + capture_output=True, + text=True, + timeout=120, + ) + + if proc.stdout: + try: + # Safety JSON format is different + safety_data = json.loads(proc.stdout) + # Parse safety output (format varies by version) + if isinstance(safety_data, list): + for item in safety_data: + if isinstance(item, list) and len(item) >= 4: + result.vulnerabilities.append( + Vulnerability( + type=VulnerabilityType.DEPENDENCY, + severity=Severity.MEDIUM, + title=f"Vulnerable dependency: {item[0]}", + description=item[3] if len(item) > 3 else "", + package_name=item[0], + package_version=item[1] if len(item) > 1 else None, + ) + ) + except json.JSONDecodeError: + pass + except (subprocess.TimeoutExpired, FileNotFoundError): + pass + + def _scan_secrets(self, result: ScanResult) -> None: + """Scan files for hardcoded secrets.""" + result.scans_run.append("secret_detection") + + # File extensions to scan + extensions = { + ".py", ".js", ".ts", ".tsx", ".jsx", + ".json", ".yaml", ".yml", ".toml", + ".env", ".env.local", ".env.example", + ".sh", ".bash", ".zsh", + ".md", ".txt", + } + + # Directories to skip + skip_dirs = { + "node_modules", "venv", ".venv", "__pycache__", + ".git", "dist", "build", ".next", + "vendor", "packages", + } + + for file_path in self._iter_files(extensions, skip_dirs): + try: + content = file_path.read_text(errors="ignore") + lines = content.split("\n") + + for pattern, title, severity, cwe_id in SECRET_PATTERNS: + for i, line in enumerate(lines, 1): + if re.search(pattern, line): + # Skip if it looks like an example or placeholder + if any( + placeholder in line.lower() + for placeholder in [ + "example", + "your_", + " 100 else line, + cwe_id=cwe_id, + recommendation="Move sensitive values to environment variables", + ) + ) + except Exception: + continue + + def _scan_code_patterns(self, result: ScanResult) -> None: + """Scan code for vulnerability patterns.""" + result.scans_run.append("code_patterns") + + # File extensions to scan + extensions = {".py", ".js", ".ts", ".tsx", ".jsx"} + + # Directories to skip + skip_dirs = { + "node_modules", "venv", ".venv", "__pycache__", + ".git", "dist", "build", ".next", + } + + for file_path in self._iter_files(extensions, skip_dirs): + try: + content = file_path.read_text(errors="ignore") + lines = content.split("\n") + + for pattern, title, vuln_type, severity, cwe_id, recommendation in CODE_PATTERNS: + for i, line in enumerate(lines, 1): + if re.search(pattern, line): + result.vulnerabilities.append( + Vulnerability( + type=vuln_type, + severity=severity, + title=title, + description="Potential vulnerability pattern detected", + file_path=str(file_path.relative_to(self.project_dir)), + line_number=i, + code_snippet=line.strip()[:100], + cwe_id=cwe_id, + recommendation=recommendation, + ) + ) + except Exception: + continue + + def _iter_files( + self, extensions: set[str], skip_dirs: set[str] + ): + """Iterate over files with given extensions, skipping certain directories.""" + for root, dirs, files in os.walk(self.project_dir): + # Skip excluded directories + dirs[:] = [d for d in dirs if d not in skip_dirs and not d.startswith(".")] + + for file in files: + file_path = Path(root) / file + if file_path.suffix in extensions or file in {".env", ".env.local", ".env.example"}: + yield file_path + + def _save_report(self, result: ScanResult) -> None: + """Save scan report to file.""" + reports_dir = self.project_dir / ".autocoder" / "security-reports" + reports_dir.mkdir(parents=True, exist_ok=True) + + timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") + report_path = reports_dir / f"security_scan_{timestamp}.json" + + with open(report_path, "w") as f: + json.dump(result.to_dict(), f, indent=2) + + +def scan_project( + project_dir: Path, + scan_dependencies: bool = True, + scan_secrets: bool = True, + scan_code: bool = True, +) -> ScanResult: + """ + Convenience function to scan a project. + + Args: + project_dir: Project directory + scan_dependencies: Run dependency audit + scan_secrets: Scan for secrets + scan_code: Scan for code patterns + + Returns: + ScanResult with findings + """ + scanner = SecurityScanner(project_dir) + return scanner.scan( + scan_dependencies=scan_dependencies, + scan_secrets=scan_secrets, + scan_code=scan_code, + ) diff --git a/server/main.py b/server/main.py index 1b01f79a..1a5628ec 100644 --- a/server/main.py +++ b/server/main.py @@ -30,15 +30,25 @@ from .routers import ( agent_router, assistant_chat_router, + cicd_router, + design_tokens_router, devserver_router, + documentation_router, expand_project_router, features_router, filesystem_router, + git_workflow_router, + import_project_router, + logs_router, projects_router, + review_router, schedules_router, + security_router, settings_router, spec_creation_router, + templates_router, terminal_router, + visual_regression_router, ) from .schemas import SetupStatus from .services.assistant_chat_session import cleanup_all_sessions as cleanup_assistant_sessions @@ -148,6 +158,16 @@ async def require_localhost(request: Request, call_next): app.include_router(assistant_chat_router) app.include_router(settings_router) app.include_router(terminal_router) +app.include_router(import_project_router) +app.include_router(logs_router) +app.include_router(security_router) +app.include_router(git_workflow_router) +app.include_router(cicd_router) +app.include_router(templates_router) +app.include_router(review_router) +app.include_router(documentation_router) +app.include_router(design_tokens_router) +app.include_router(visual_regression_router) # ============================================================================ diff --git a/server/routers/__init__.py b/server/routers/__init__.py index f4d02f51..fe48e2ae 100644 --- a/server/routers/__init__.py +++ b/server/routers/__init__.py @@ -7,15 +7,25 @@ from .agent import router as agent_router from .assistant_chat import router as assistant_chat_router +from .cicd import router as cicd_router +from .design_tokens import router as design_tokens_router from .devserver import router as devserver_router +from .documentation import router as documentation_router from .expand_project import router as expand_project_router from .features import router as features_router from .filesystem import router as filesystem_router +from .git_workflow import router as git_workflow_router +from .import_project import router as import_project_router +from .logs import router as logs_router from .projects import router as projects_router +from .review import router as review_router from .schedules import router as schedules_router +from .security import router as security_router from .settings import router as settings_router from .spec_creation import router as spec_creation_router +from .templates import router as templates_router from .terminal import router as terminal_router +from .visual_regression import router as visual_regression_router __all__ = [ "projects_router", @@ -29,4 +39,14 @@ "assistant_chat_router", "settings_router", "terminal_router", + "import_project_router", + "logs_router", + "security_router", + "git_workflow_router", + "cicd_router", + "templates_router", + "review_router", + "documentation_router", + "design_tokens_router", + "visual_regression_router", ] diff --git a/server/routers/cicd.py b/server/routers/cicd.py new file mode 100644 index 00000000..4be9a0bf --- /dev/null +++ b/server/routers/cicd.py @@ -0,0 +1,254 @@ +""" +CI/CD Router +============ + +REST API endpoints for CI/CD workflow generation. + +Endpoints: +- POST /api/cicd/generate - Generate CI/CD workflows +- GET /api/cicd/workflows - List existing workflows +- GET /api/cicd/preview - Preview workflow content +""" + +import logging +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/cicd", tags=["cicd"]) + + +def _get_project_path(project_name: str) -> Path | None: + """Get project path from registry.""" + from registry import get_project_path + + return get_project_path(project_name) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class GenerateRequest(BaseModel): + """Request to generate CI/CD workflows.""" + + project_name: str = Field(..., description="Name of the registered project") + provider: str = Field("github", description="CI provider (github, gitlab)") + workflow_types: list[str] = Field( + ["ci", "security", "deploy"], + description="Types of workflows to generate", + ) + save: bool = Field(True, description="Whether to save the workflow files") + + +class WorkflowInfo(BaseModel): + """Information about a generated workflow.""" + + name: str + filename: str + type: str + path: Optional[str] = None + + +class GenerateResponse(BaseModel): + """Response from workflow generation.""" + + provider: str + workflows: list[WorkflowInfo] + output_dir: str + message: str + + +class PreviewRequest(BaseModel): + """Request to preview a workflow.""" + + project_name: str = Field(..., description="Name of the registered project") + workflow_type: str = Field("ci", description="Type of workflow (ci, security, deploy)") + + +class PreviewResponse(BaseModel): + """Response with workflow preview.""" + + workflow_type: str + filename: str + content: str + + +class WorkflowListResponse(BaseModel): + """Response with list of existing workflows.""" + + workflows: list[WorkflowInfo] + count: int + + +# ============================================================================ +# REST Endpoints +# ============================================================================ + + +@router.post("/generate", response_model=GenerateResponse) +async def generate_workflows(request: GenerateRequest): + """ + Generate CI/CD workflows for a project. + + Detects tech stack and generates appropriate workflow files. + Supports GitHub Actions (and GitLab CI planned). + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + if not project_dir.exists(): + raise HTTPException(status_code=404, detail="Project directory not found") + + try: + if request.provider == "github": + from integrations.ci import generate_github_workflow + + workflows = [] + for wf_type in request.workflow_types: + if wf_type not in ["ci", "security", "deploy"]: + continue + + workflow = generate_github_workflow( + project_dir, + workflow_type=wf_type, + save=request.save, + ) + + path = None + if request.save: + path = str(project_dir / ".github" / "workflows" / workflow.filename) + + workflows.append( + WorkflowInfo( + name=workflow.name, + filename=workflow.filename, + type=wf_type, + path=path, + ) + ) + + return GenerateResponse( + provider="github", + workflows=workflows, + output_dir=str(project_dir / ".github" / "workflows"), + message=f"Generated {len(workflows)} workflow(s)", + ) + + else: + raise HTTPException( + status_code=400, + detail=f"Unsupported provider: {request.provider}", + ) + + except Exception as e: + logger.exception(f"Error generating workflows: {e}") + raise HTTPException(status_code=500, detail=f"Generation failed: {str(e)}") + + +@router.post("/preview", response_model=PreviewResponse) +async def preview_workflow(request: PreviewRequest): + """ + Preview a workflow without saving it. + + Returns the YAML content that would be generated. + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + if request.workflow_type not in ["ci", "security", "deploy"]: + raise HTTPException( + status_code=400, + detail=f"Invalid workflow type: {request.workflow_type}", + ) + + try: + from integrations.ci import generate_github_workflow + + workflow = generate_github_workflow( + project_dir, + workflow_type=request.workflow_type, + save=False, + ) + + return PreviewResponse( + workflow_type=request.workflow_type, + filename=workflow.filename, + content=workflow.to_yaml(), + ) + + except Exception as e: + logger.exception(f"Error previewing workflow: {e}") + raise HTTPException(status_code=500, detail=f"Preview failed: {str(e)}") + + +@router.get("/workflows/{project_name}", response_model=WorkflowListResponse) +async def list_workflows(project_name: str): + """ + List existing GitHub Actions workflows for a project. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + workflows_dir = project_dir / ".github" / "workflows" + if not workflows_dir.exists(): + return WorkflowListResponse(workflows=[], count=0) + + workflows = [] + for file in workflows_dir.glob("*.yml"): + # Determine workflow type from filename + wf_type = "custom" + if file.stem in ["ci", "security", "deploy"]: + wf_type = file.stem + + workflows.append( + WorkflowInfo( + name=file.stem.title(), + filename=file.name, + type=wf_type, + path=str(file), + ) + ) + + return WorkflowListResponse( + workflows=workflows, + count=len(workflows), + ) + + +@router.get("/workflows/{project_name}/{filename}") +async def get_workflow_content(project_name: str, filename: str): + """ + Get the content of a specific workflow file. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + # Security: validate filename + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + if not filename.endswith((".yml", ".yaml")): + raise HTTPException(status_code=400, detail="Invalid workflow filename") + + workflow_path = project_dir / ".github" / "workflows" / filename + if not workflow_path.exists(): + raise HTTPException(status_code=404, detail="Workflow not found") + + try: + content = workflow_path.read_text() + return { + "filename": filename, + "content": content, + } + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error reading workflow: {str(e)}") diff --git a/server/routers/design_tokens.py b/server/routers/design_tokens.py new file mode 100644 index 00000000..00ac7c57 --- /dev/null +++ b/server/routers/design_tokens.py @@ -0,0 +1,355 @@ +""" +Design Tokens API Router +======================== + +REST API endpoints for design tokens management. + +Endpoints: +- GET /api/design-tokens/{project_name} - Get current design tokens +- PUT /api/design-tokens/{project_name} - Update design tokens +- POST /api/design-tokens/{project_name}/generate - Generate token files +- GET /api/design-tokens/{project_name}/preview/{format} - Preview generated output +- POST /api/design-tokens/{project_name}/validate - Validate tokens +""" + +import logging +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel + +from design_tokens import DesignTokens, DesignTokensManager +from registry import get_project_path + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/design-tokens", tags=["design-tokens"]) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class ColorTokens(BaseModel): + """Color token configuration.""" + + primary: Optional[str] = "#3B82F6" + secondary: Optional[str] = "#6366F1" + accent: Optional[str] = "#F59E0B" + success: Optional[str] = "#10B981" + warning: Optional[str] = "#F59E0B" + error: Optional[str] = "#EF4444" + info: Optional[str] = "#3B82F6" + neutral: Optional[str] = "#6B7280" + + +class TypographyTokens(BaseModel): + """Typography token configuration.""" + + font_family: Optional[dict] = None + font_size: Optional[dict] = None + font_weight: Optional[dict] = None + line_height: Optional[dict] = None + + +class BorderTokens(BaseModel): + """Border token configuration.""" + + radius: Optional[dict] = None + width: Optional[dict] = None + + +class AnimationTokens(BaseModel): + """Animation token configuration.""" + + duration: Optional[dict] = None + easing: Optional[dict] = None + + +class DesignTokensRequest(BaseModel): + """Request to update design tokens.""" + + colors: Optional[dict] = None + spacing: Optional[list[int]] = None + typography: Optional[dict] = None + borders: Optional[dict] = None + shadows: Optional[dict] = None + animations: Optional[dict] = None + + +class DesignTokensResponse(BaseModel): + """Response with design tokens.""" + + colors: dict + spacing: list[int] + typography: dict + borders: dict + shadows: dict + animations: dict + + +class GenerateResponse(BaseModel): + """Response from token generation.""" + + generated_files: dict + contrast_issues: Optional[list[dict]] = None + message: str + + +class PreviewResponse(BaseModel): + """Preview of generated output.""" + + format: str + content: str + + +class ValidateResponse(BaseModel): + """Validation results.""" + + valid: bool + issues: list[dict] + + +# ============================================================================ +# Helper Functions +# ============================================================================ + + +def get_project_dir(project_name: str) -> Path: + """Get project directory from name or path.""" + project_path = get_project_path(project_name) + if project_path: + return Path(project_path) + + path = Path(project_name) + if path.exists() and path.is_dir(): + return path + + raise HTTPException(status_code=404, detail=f"Project not found: {project_name}") + + +# ============================================================================ +# Endpoints +# ============================================================================ + + +@router.get("/{project_name}", response_model=DesignTokensResponse) +async def get_design_tokens(project_name: str): + """ + Get current design tokens for a project. + + Returns the design tokens from config file or defaults. + """ + project_dir = get_project_dir(project_name) + + try: + manager = DesignTokensManager(project_dir) + tokens = manager.load() + + return DesignTokensResponse( + colors=tokens.colors, + spacing=tokens.spacing, + typography=tokens.typography, + borders=tokens.borders, + shadows=tokens.shadows, + animations=tokens.animations, + ) + except Exception as e: + logger.error(f"Error getting design tokens: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.put("/{project_name}", response_model=DesignTokensResponse) +async def update_design_tokens(project_name: str, request: DesignTokensRequest): + """ + Update design tokens for a project. + + Saves tokens to .autocoder/design-tokens.json. + """ + project_dir = get_project_dir(project_name) + + try: + manager = DesignTokensManager(project_dir) + current = manager.load() + + # Update only provided fields + if request.colors: + current.colors.update(request.colors) + if request.spacing: + current.spacing = request.spacing + if request.typography: + current.typography.update(request.typography) + if request.borders: + current.borders.update(request.borders) + if request.shadows: + current.shadows.update(request.shadows) + if request.animations: + current.animations.update(request.animations) + + manager.save(current) + + return DesignTokensResponse( + colors=current.colors, + spacing=current.spacing, + typography=current.typography, + borders=current.borders, + shadows=current.shadows, + animations=current.animations, + ) + except Exception as e: + logger.error(f"Error updating design tokens: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.post("/{project_name}/generate", response_model=GenerateResponse) +async def generate_token_files(project_name: str, output_dir: Optional[str] = None): + """ + Generate token files for a project. + + Creates: + - tokens.css - CSS custom properties + - _tokens.scss - SCSS variables + - tailwind.tokens.js - Tailwind config (if Tailwind detected) + """ + project_dir = get_project_dir(project_name) + + try: + manager = DesignTokensManager(project_dir) + + if output_dir: + result = manager.generate_all(project_dir / output_dir) + else: + result = manager.generate_all() + + contrast_issues = result.pop("contrast_issues", None) + + return GenerateResponse( + generated_files=result, + contrast_issues=contrast_issues, + message=f"Generated {len(result)} token files", + ) + except Exception as e: + logger.error(f"Error generating token files: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.get("/{project_name}/preview/{format}", response_model=PreviewResponse) +async def preview_tokens(project_name: str, format: str): + """ + Preview generated output without writing to disk. + + Args: + project_name: Project name + format: Output format (css, scss, tailwind) + """ + project_dir = get_project_dir(project_name) + + valid_formats = ["css", "scss", "tailwind"] + if format not in valid_formats: + raise HTTPException( + status_code=400, + detail=f"Invalid format. Valid formats: {', '.join(valid_formats)}", + ) + + try: + manager = DesignTokensManager(project_dir) + tokens = manager.load() + + if format == "css": + content = manager.generate_css(tokens) + elif format == "scss": + content = manager.generate_scss(tokens) + elif format == "tailwind": + content = manager.generate_tailwind_config(tokens) + else: + content = "" + + return PreviewResponse(format=format, content=content) + except Exception as e: + logger.error(f"Error previewing tokens: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.post("/{project_name}/validate", response_model=ValidateResponse) +async def validate_tokens(project_name: str): + """ + Validate design tokens for accessibility and consistency. + + Checks: + - Color contrast ratios + - Color format validity + - Spacing scale consistency + """ + project_dir = get_project_dir(project_name) + + try: + manager = DesignTokensManager(project_dir) + tokens = manager.load() + + issues = [] + + # Validate colors + import re + + hex_pattern = re.compile(r"^#([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$") + for name, value in tokens.colors.items(): + if not hex_pattern.match(value): + issues.append( + { + "type": "color_format", + "field": f"colors.{name}", + "value": value, + "message": "Invalid hex color format", + } + ) + + # Check contrast + contrast_issues = manager.validate_contrast(tokens) + for ci in contrast_issues: + issues.append( + { + "type": "contrast", + "field": f"colors.{ci['color']}", + "value": ci["value"], + "message": ci["issue"], + "suggestion": ci.get("suggestion"), + } + ) + + # Validate spacing scale + if tokens.spacing: + for i in range(1, len(tokens.spacing)): + if tokens.spacing[i] <= tokens.spacing[i - 1]: + issues.append( + { + "type": "spacing_scale", + "field": "spacing", + "value": tokens.spacing, + "message": f"Spacing scale should be increasing: {tokens.spacing[i-1]} >= {tokens.spacing[i]}", + } + ) + + return ValidateResponse(valid=len(issues) == 0, issues=issues) + except Exception as e: + logger.error(f"Error validating tokens: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.post("/{project_name}/reset") +async def reset_tokens(project_name: str): + """ + Reset design tokens to defaults. + """ + project_dir = get_project_dir(project_name) + + try: + manager = DesignTokensManager(project_dir) + tokens = DesignTokens.default() + manager.save(tokens) + + return {"reset": True, "message": "Design tokens reset to defaults"} + except Exception as e: + logger.error(f"Error resetting tokens: {e}") + raise HTTPException(status_code=500, detail=str(e)) diff --git a/server/routers/documentation.py b/server/routers/documentation.py new file mode 100644 index 00000000..4ee87fb7 --- /dev/null +++ b/server/routers/documentation.py @@ -0,0 +1,301 @@ +""" +Documentation API Router +======================== + +REST API endpoints for automatic documentation generation. + +Endpoints: +- POST /api/docs/generate - Generate documentation for a project +- GET /api/docs/{project_name} - List documentation files +- GET /api/docs/{project_name}/{filename} - Get documentation content +- POST /api/docs/preview - Preview README content +""" + +import logging +from pathlib import Path + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +from auto_documentation import DocumentationGenerator +from registry import get_project_path + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/docs", tags=["documentation"]) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class GenerateDocsRequest(BaseModel): + """Request to generate documentation.""" + + project_name: str = Field(..., description="Project name or path") + output_dir: str = Field("docs", description="Output directory for docs") + generate_readme: bool = Field(True, description="Generate README.md") + generate_api: bool = Field(True, description="Generate API documentation") + generate_setup: bool = Field(True, description="Generate setup guide") + + +class GenerateDocsResponse(BaseModel): + """Response from documentation generation.""" + + project_name: str + generated_files: dict + message: str + + +class DocFile(BaseModel): + """A documentation file.""" + + filename: str + path: str + size: int + modified: str + + +class ListDocsResponse(BaseModel): + """List of documentation files.""" + + files: list[DocFile] + count: int + + +class PreviewRequest(BaseModel): + """Request to preview README.""" + + project_name: str = Field(..., description="Project name or path") + + +class PreviewResponse(BaseModel): + """Preview of README content.""" + + content: str + project_name: str + description: Optional[str] = None + tech_stack: dict + features_count: int + endpoints_count: int + components_count: int + + +# ============================================================================ +# Helper Functions +# ============================================================================ + + +def get_project_dir(project_name: str) -> Path: + """Get project directory from name or path.""" + project_path = get_project_path(project_name) + if project_path: + return Path(project_path) + + path = Path(project_name) + if path.exists() and path.is_dir(): + return path + + raise HTTPException(status_code=404, detail=f"Project not found: {project_name}") + + +# ============================================================================ +# Endpoints +# ============================================================================ + + +@router.post("/generate", response_model=GenerateDocsResponse) +async def generate_docs(request: GenerateDocsRequest): + """ + Generate documentation for a project. + + Creates: + - README.md in project root + - SETUP.md in docs directory + - API.md in docs directory (if API endpoints found) + """ + project_dir = get_project_dir(request.project_name) + + try: + generator = DocumentationGenerator(project_dir, request.output_dir) + docs = generator.generate() + + generated = {} + + if request.generate_readme: + readme_path = generator.write_readme(docs) + generated["readme"] = str(readme_path.relative_to(project_dir)) + + if request.generate_setup: + setup_path = generator.write_setup_guide(docs) + generated["setup"] = str(setup_path.relative_to(project_dir)) + + if request.generate_api: + api_path = generator.write_api_docs(docs) + if api_path: + generated["api"] = str(api_path.relative_to(project_dir)) + + return GenerateDocsResponse( + project_name=docs.project_name, + generated_files=generated, + message=f"Generated {len(generated)} documentation files", + ) + + except Exception as e: + # Log full exception server-side but don't expose details to client + logger.exception(f"Documentation generation failed: {e}") + raise HTTPException(status_code=500, detail="Internal error generating documentation") + + +@router.get("/{project_name}", response_model=ListDocsResponse) +async def list_docs(project_name: str): + """ + List all documentation files for a project. + + Searches for Markdown files in project root and docs/ directory. + """ + project_dir = get_project_dir(project_name) + + files = [] + + # Check root for README + for md_file in ["README.md", "CHANGELOG.md", "CONTRIBUTING.md"]: + file_path = project_dir / md_file + if file_path.exists(): + stat = file_path.stat() + files.append( + DocFile( + filename=md_file, + path=md_file, + size=stat.st_size, + modified=stat.st_mtime.__str__(), + ) + ) + + # Check docs directory + docs_dir = project_dir / "docs" + if docs_dir.exists(): + for md_file in docs_dir.glob("*.md"): + stat = md_file.stat() + files.append( + DocFile( + filename=md_file.name, + path=str(md_file.relative_to(project_dir)), + size=stat.st_size, + modified=stat.st_mtime.__str__(), + ) + ) + + return ListDocsResponse(files=files, count=len(files)) + + +@router.get("/{project_name}/{filename:path}") +async def get_doc_content(project_name: str, filename: str): + """ + Get content of a documentation file. + + Args: + project_name: Project name + filename: Documentation file path (e.g., "README.md" or "docs/API.md") + """ + project_dir = get_project_dir(project_name) + + # Validate filename to prevent path traversal (including URL-encoded attacks) + file_path = (project_dir / filename).resolve() + if not file_path.is_relative_to(project_dir.resolve()): + raise HTTPException(status_code=400, detail="Invalid filename: path traversal detected") + + if not file_path.exists(): + raise HTTPException(status_code=404, detail=f"File not found: {filename}") + + if not file_path.suffix.lower() == ".md": + raise HTTPException(status_code=400, detail="Only Markdown files are supported") + + try: + content = file_path.read_text() + return {"filename": filename, "content": content} + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error reading file: {e}") + + +@router.post("/preview", response_model=PreviewResponse) +async def preview_readme(request: PreviewRequest): + """ + Preview README content without writing to disk. + + Returns the generated README content and project statistics. + """ + project_dir = get_project_dir(request.project_name) + + try: + generator = DocumentationGenerator(project_dir) + docs = generator.generate() + + # Generate README content in memory + lines = [] + lines.append(f"# {docs.project_name}\n") + + if docs.description: + lines.append(f"{docs.description}\n") + + if any(docs.tech_stack.values()): + lines.append("## Tech Stack\n") + for category, items in docs.tech_stack.items(): + if items: + lines.append(f"**{category.title()}:** {', '.join(items)}\n") + + if docs.features: + lines.append("\n## Features\n") + for f in docs.features[:10]: + status = "[x]" if f.get("status") == "completed" else "[ ]" + lines.append(f"- {status} {f['name']}") + if len(docs.features) > 10: + lines.append(f"\n*...and {len(docs.features) - 10} more features*") + + content = "\n".join(lines) + + return PreviewResponse( + content=content, + project_name=docs.project_name, + description=docs.description, + tech_stack=docs.tech_stack, + features_count=len(docs.features), + endpoints_count=len(docs.api_endpoints), + components_count=len(docs.components), + ) + + except Exception as e: + # Log full exception server-side but don't expose details to client + logger.exception(f"Preview failed: {e}") + raise HTTPException(status_code=500, detail="Internal error generating preview") + + +@router.delete("/{project_name}/{filename:path}") +async def delete_doc(project_name: str, filename: str): + """ + Delete a documentation file. + + Args: + project_name: Project name + filename: Documentation file path + """ + project_dir = get_project_dir(project_name) + + # Validate filename + if ".." in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + file_path = project_dir / filename + + if not file_path.exists(): + raise HTTPException(status_code=404, detail=f"File not found: {filename}") + + if not file_path.suffix.lower() == ".md": + raise HTTPException(status_code=400, detail="Only Markdown files can be deleted") + + try: + file_path.unlink() + return {"deleted": True, "filename": filename} + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error deleting file: {e}") diff --git a/server/routers/git_workflow.py b/server/routers/git_workflow.py new file mode 100644 index 00000000..bb8d9cac --- /dev/null +++ b/server/routers/git_workflow.py @@ -0,0 +1,283 @@ +""" +Git Workflow Router +=================== + +REST API endpoints for git workflow management. + +Endpoints: +- GET /api/git/status - Get current git status +- POST /api/git/start-feature - Start working on a feature (create branch) +- POST /api/git/complete-feature - Complete a feature (merge) +- POST /api/git/abort-feature - Abort a feature +- GET /api/git/branches - List feature branches +""" + +import logging +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/git", tags=["git-workflow"]) + + +def _get_project_path(project_name: str) -> Path | None: + """Get project path from registry.""" + from registry import get_project_path + + return get_project_path(project_name) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class StartFeatureRequest(BaseModel): + """Request to start a feature branch.""" + + project_name: str = Field(..., description="Name of the registered project") + feature_id: int = Field(..., description="Feature ID") + feature_name: str = Field(..., description="Feature name for branch naming") + + +class CompleteFeatureRequest(BaseModel): + """Request to complete a feature.""" + + project_name: str = Field(..., description="Name of the registered project") + feature_id: int = Field(..., description="Feature ID") + + +class AbortFeatureRequest(BaseModel): + """Request to abort a feature.""" + + project_name: str = Field(..., description="Name of the registered project") + feature_id: int = Field(..., description="Feature ID") + delete_branch: bool = Field(False, description="Whether to delete the branch") + + +class CommitRequest(BaseModel): + """Request to commit changes.""" + + project_name: str = Field(..., description="Name of the registered project") + feature_id: int = Field(..., description="Feature ID") + message: str = Field(..., description="Commit message") + + +class WorkflowResultResponse(BaseModel): + """Response from workflow operations.""" + + success: bool + message: str + branch_name: Optional[str] = None + previous_branch: Optional[str] = None + + +class GitStatusResponse(BaseModel): + """Response with git status information.""" + + is_git_repo: bool + mode: str + current_branch: Optional[str] = None + main_branch: Optional[str] = None + is_on_feature_branch: bool = False + current_feature_id: Optional[int] = None + has_uncommitted_changes: bool = False + feature_branches: list[str] = [] + feature_branch_count: int = 0 + + +class BranchInfo(BaseModel): + """Information about a branch.""" + + name: str + feature_id: Optional[int] = None + is_feature_branch: bool = False + is_current: bool = False + + +class BranchListResponse(BaseModel): + """Response with list of branches.""" + + branches: list[BranchInfo] + count: int + + +# ============================================================================ +# REST Endpoints +# ============================================================================ + + +@router.get("/status/{project_name}", response_model=GitStatusResponse) +async def get_git_status(project_name: str): + """ + Get current git workflow status for a project. + + Returns information about current branch, mode, and feature branches. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from git_workflow import get_workflow + + workflow = get_workflow(project_dir) + status = workflow.get_status() + + return GitStatusResponse(**status) + + except Exception as e: + logger.exception(f"Error getting git status: {e}") + raise HTTPException(status_code=500, detail=f"Failed to get status: {str(e)}") + + +@router.post("/start-feature", response_model=WorkflowResultResponse) +async def start_feature(request: StartFeatureRequest): + """ + Start working on a feature (create and checkout branch). + + In feature_branches mode, creates a new branch like 'feature/42-user-can-login'. + In trunk mode, this is a no-op. + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from git_workflow import get_workflow + + workflow = get_workflow(project_dir) + result = workflow.start_feature(request.feature_id, request.feature_name) + + return WorkflowResultResponse( + success=result.success, + message=result.message, + branch_name=result.branch_name, + previous_branch=result.previous_branch, + ) + + except Exception as e: + logger.exception(f"Error starting feature: {e}") + raise HTTPException(status_code=500, detail=f"Failed to start feature: {str(e)}") + + +@router.post("/complete-feature", response_model=WorkflowResultResponse) +async def complete_feature(request: CompleteFeatureRequest): + """ + Complete a feature (merge to main if auto_merge enabled). + + Commits any remaining changes and optionally merges the feature branch. + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from git_workflow import get_workflow + + workflow = get_workflow(project_dir) + result = workflow.complete_feature(request.feature_id) + + return WorkflowResultResponse( + success=result.success, + message=result.message, + branch_name=result.branch_name, + previous_branch=result.previous_branch, + ) + + except Exception as e: + logger.exception(f"Error completing feature: {e}") + raise HTTPException(status_code=500, detail=f"Failed to complete feature: {str(e)}") + + +@router.post("/abort-feature", response_model=WorkflowResultResponse) +async def abort_feature(request: AbortFeatureRequest): + """ + Abort a feature (discard changes, optionally delete branch). + + Returns to main branch and discards uncommitted changes. + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from git_workflow import get_workflow + + workflow = get_workflow(project_dir) + result = workflow.abort_feature(request.feature_id, request.delete_branch) + + return WorkflowResultResponse( + success=result.success, + message=result.message, + branch_name=result.branch_name, + previous_branch=result.previous_branch, + ) + + except Exception as e: + logger.exception(f"Error aborting feature: {e}") + raise HTTPException(status_code=500, detail=f"Failed to abort feature: {str(e)}") + + +@router.post("/commit", response_model=WorkflowResultResponse) +async def commit_changes(request: CommitRequest): + """ + Commit current changes for a feature. + + Adds all changes and commits with a structured message. + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from git_workflow import get_workflow + + workflow = get_workflow(project_dir) + result = workflow.commit_feature_progress(request.feature_id, request.message) + + return WorkflowResultResponse( + success=result.success, + message=result.message, + ) + + except Exception as e: + logger.exception(f"Error committing: {e}") + raise HTTPException(status_code=500, detail=f"Commit failed: {str(e)}") + + +@router.get("/branches/{project_name}", response_model=BranchListResponse) +async def list_branches(project_name: str): + """ + List all feature branches for a project. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from git_workflow import get_workflow + + workflow = get_workflow(project_dir) + branches = workflow.list_feature_branches() + + return BranchListResponse( + branches=[ + BranchInfo( + name=b.name, + feature_id=b.feature_id, + is_feature_branch=b.is_feature_branch, + is_current=b.is_current, + ) + for b in branches + ], + count=len(branches), + ) + + except Exception as e: + logger.exception(f"Error listing branches: {e}") + raise HTTPException(status_code=500, detail=f"Failed to list branches: {str(e)}") diff --git a/server/routers/import_project.py b/server/routers/import_project.py new file mode 100644 index 00000000..f2587638 --- /dev/null +++ b/server/routers/import_project.py @@ -0,0 +1,319 @@ +""" +Import Project Router +===================== + +REST and WebSocket endpoints for importing existing projects into Autocoder. + +The import flow: +1. POST /api/import/analyze - Analyze codebase, detect stack +2. POST /api/import/extract-features - Generate features from analysis +3. POST /api/import/create-features - Create features in database +""" + +import logging +import re +import sys +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/import", tags=["import-project"]) + +# Root directory +ROOT_DIR = Path(__file__).parent.parent.parent + +# Add root to path for imports +if str(ROOT_DIR) not in sys.path: + sys.path.insert(0, str(ROOT_DIR)) + + +def _get_project_path(project_name: str) -> Path | None: + """Get project path from registry.""" + from registry import get_project_path + return get_project_path(project_name) + + +def validate_path(path: str) -> bool: + """Validate path to prevent traversal attacks.""" + # Allow absolute paths but check for common attack patterns + if ".." in path or "\x00" in path: + return False + return True + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + +class AnalyzeRequest(BaseModel): + """Request to analyze a project directory.""" + path: str = Field(..., description="Absolute path to the project directory") + + +class StackInfo(BaseModel): + """Information about a detected stack.""" + name: str + category: str + confidence: float + + +class AnalyzeResponse(BaseModel): + """Response from project analysis.""" + project_dir: str + detected_stacks: list[StackInfo] + primary_frontend: Optional[str] = None + primary_backend: Optional[str] = None + database: Optional[str] = None + routes_count: int + components_count: int + endpoints_count: int + summary: str + + +class ExtractFeaturesRequest(BaseModel): + """Request to extract features from an analyzed project.""" + path: str = Field(..., description="Absolute path to the project directory") + + +class DetectedFeature(BaseModel): + """A feature extracted from codebase analysis.""" + category: str + name: str + description: str + steps: list[str] + source_type: str + source_file: Optional[str] = None + confidence: float + + +class ExtractFeaturesResponse(BaseModel): + """Response from feature extraction.""" + features: list[DetectedFeature] + count: int + by_category: dict[str, int] + summary: str + + +class CreateFeaturesRequest(BaseModel): + """Request to create features in the database.""" + project_name: str = Field(..., description="Name of the registered project") + features: list[dict] = Field(..., description="Features to create (category, name, description, steps)") + + +class CreateFeaturesResponse(BaseModel): + """Response from feature creation.""" + created: int + project_name: str + message: str + + +# ============================================================================ +# REST Endpoints +# ============================================================================ + +@router.post("/analyze", response_model=AnalyzeResponse) +async def analyze_project(request: AnalyzeRequest): + """ + Analyze a project directory to detect tech stack. + + Returns detected stacks with confidence scores, plus counts of + routes, endpoints, and components found. + """ + if not validate_path(request.path): + raise HTTPException(status_code=400, detail="Invalid path") + + project_dir = Path(request.path).resolve() + + if not project_dir.exists(): + raise HTTPException(status_code=404, detail="Directory not found") + + if not project_dir.is_dir(): + raise HTTPException(status_code=400, detail="Path is not a directory") + + try: + from analyzers import StackDetector + + detector = StackDetector(project_dir) + result = detector.detect() + + # Convert to response model + stacks = [ + StackInfo( + name=s["name"], + category=s["category"], + confidence=s["confidence"], + ) + for s in result["detected_stacks"] + ] + + return AnalyzeResponse( + project_dir=str(project_dir), + detected_stacks=stacks, + primary_frontend=result.get("primary_frontend"), + primary_backend=result.get("primary_backend"), + database=result.get("database"), + routes_count=result.get("routes_count", 0), + components_count=result.get("components_count", 0), + endpoints_count=result.get("endpoints_count", 0), + summary=result.get("summary", ""), + ) + + except Exception as e: + logger.exception(f"Error analyzing project: {e}") + raise HTTPException(status_code=500, detail=f"Analysis failed: {str(e)}") + + +@router.post("/extract-features", response_model=ExtractFeaturesResponse) +async def extract_features(request: ExtractFeaturesRequest): + """ + Extract features from an analyzed project. + + Returns a list of features ready for import, each with: + - category, name, description, steps + - source_type (route, endpoint, component, inferred) + - confidence score + """ + if not validate_path(request.path): + raise HTTPException(status_code=400, detail="Invalid path") + + project_dir = Path(request.path).resolve() + + if not project_dir.exists(): + raise HTTPException(status_code=404, detail="Directory not found") + + try: + from analyzers import extract_from_project + + result = extract_from_project(project_dir) + + # Convert to response model + features = [ + DetectedFeature( + category=f["category"], + name=f["name"], + description=f["description"], + steps=f["steps"], + source_type=f["source_type"], + source_file=f.get("source_file"), + confidence=f["confidence"], + ) + for f in result["features"] + ] + + return ExtractFeaturesResponse( + features=features, + count=result["count"], + by_category=result["by_category"], + summary=result["summary"], + ) + + except Exception as e: + logger.exception(f"Error extracting features: {e}") + raise HTTPException(status_code=500, detail=f"Feature extraction failed: {str(e)}") + + +@router.post("/create-features", response_model=CreateFeaturesResponse) +async def create_features(request: CreateFeaturesRequest): + """ + Create features in the database for a registered project. + + Takes extracted features and creates them via the feature database. + All features are created with passes=False (pending verification). + """ + # Validate project name + if not re.match(r'^[a-zA-Z0-9_-]{1,50}$', request.project_name): + raise HTTPException(status_code=400, detail="Invalid project name") + + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found in registry") + + if not project_dir.exists(): + raise HTTPException(status_code=404, detail="Project directory not found") + + if not request.features: + raise HTTPException(status_code=400, detail="No features provided") + + try: + from api.database import Feature, create_database + + # Initialize database + engine, SessionLocal = create_database(project_dir) + session = SessionLocal() + + try: + # Get starting priority + from sqlalchemy import func + max_priority = session.query(func.max(Feature.priority)).scalar() or 0 + + # Create features + created_count = 0 + for i, f in enumerate(request.features): + # Validate required fields + if not all(key in f for key in ["category", "name", "description", "steps"]): + logger.warning(f"Skipping feature missing required fields: {f}") + continue + + feature = Feature( + priority=max_priority + i + 1, + category=f["category"], + name=f["name"], + description=f["description"], + steps=f["steps"], + passes=False, + in_progress=False, + ) + session.add(feature) + created_count += 1 + + session.commit() + + return CreateFeaturesResponse( + created=created_count, + project_name=request.project_name, + message=f"Created {created_count} features for project '{request.project_name}'", + ) + + finally: + session.close() + + except Exception as e: + logger.exception(f"Error creating features: {e}") + raise HTTPException(status_code=500, detail=f"Feature creation failed: {str(e)}") + + +@router.get("/quick-detect") +async def quick_detect(path: str): + """ + Quick detection endpoint for UI preview. + + Returns only stack names and confidence without full analysis. + Useful for showing detected stack while user configures import. + """ + if not validate_path(path): + raise HTTPException(status_code=400, detail="Invalid path") + + project_dir = Path(path).resolve() + + if not project_dir.exists(): + raise HTTPException(status_code=404, detail="Directory not found") + + try: + from analyzers import StackDetector + + detector = StackDetector(project_dir) + result = detector.detect_quick() + + return { + "project_dir": str(project_dir), + "stacks": result.get("stacks", []), + "primary": result.get("primary"), + } + + except Exception as e: + logger.exception(f"Error in quick detect: {e}") + raise HTTPException(status_code=500, detail=f"Detection failed: {str(e)}") diff --git a/server/routers/logs.py b/server/routers/logs.py new file mode 100644 index 00000000..39b74210 --- /dev/null +++ b/server/routers/logs.py @@ -0,0 +1,315 @@ +""" +Logs Router +=========== + +REST API endpoints for querying and exporting structured logs. + +Endpoints: +- GET /api/logs - Query logs with filters +- GET /api/logs/timeline - Get activity timeline +- GET /api/logs/stats - Get per-agent statistics +- POST /api/logs/export - Export logs to file +""" + +import logging +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Literal, Optional + +from fastapi import APIRouter, HTTPException, Query +from fastapi.responses import FileResponse +from pydantic import BaseModel + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/logs", tags=["logs"]) + + +def _get_project_path(project_name: str) -> Path | None: + """Get project path from registry.""" + from registry import get_project_path + + return get_project_path(project_name) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class LogEntry(BaseModel): + """A structured log entry.""" + + id: int + timestamp: str + level: str + message: str + agent_id: Optional[str] = None + feature_id: Optional[int] = None + tool_name: Optional[str] = None + duration_ms: Optional[int] = None + extra: Optional[str] = None + + +class LogQueryResponse(BaseModel): + """Response from log query.""" + + logs: list[LogEntry] + total: int + limit: int + offset: int + + +class TimelineBucket(BaseModel): + """A timeline bucket with activity counts.""" + + timestamp: str + agents: dict[str, int] + total: int + errors: int + + +class TimelineResponse(BaseModel): + """Response from timeline query.""" + + buckets: list[TimelineBucket] + bucket_minutes: int + + +class AgentStats(BaseModel): + """Statistics for a single agent.""" + + agent_id: Optional[str] + total: int + info_count: int + warn_count: int + error_count: int + first_log: Optional[str] + last_log: Optional[str] + + +class StatsResponse(BaseModel): + """Response from stats query.""" + + agents: list[AgentStats] + total_logs: int + + +class ExportRequest(BaseModel): + """Request to export logs.""" + + project_name: str + format: Literal["json", "jsonl", "csv"] = "jsonl" + level: Optional[str] = None + agent_id: Optional[str] = None + feature_id: Optional[int] = None + since_hours: Optional[int] = None + + +class ExportResponse(BaseModel): + """Response from export request.""" + + filename: str + count: int + format: str + + +# ============================================================================ +# REST Endpoints +# ============================================================================ + + +@router.get("/{project_name}", response_model=LogQueryResponse) +async def query_logs( + project_name: str, + level: Optional[str] = Query(None, description="Filter by log level (debug, info, warn, error)"), + agent_id: Optional[str] = Query(None, description="Filter by agent ID"), + feature_id: Optional[int] = Query(None, description="Filter by feature ID"), + tool_name: Optional[str] = Query(None, description="Filter by tool name"), + search: Optional[str] = Query(None, description="Full-text search in message"), + since_hours: Optional[int] = Query(None, description="Filter logs from last N hours"), + limit: int = Query(100, ge=1, le=1000, description="Max results"), + offset: int = Query(0, ge=0, description="Pagination offset"), +): + """ + Query logs with filters. + + Supports filtering by level, agent, feature, tool, and full-text search. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from structured_logging import get_log_query + + query = get_log_query(project_dir) + + since = None + if since_hours: + since = datetime.now(timezone.utc) - timedelta(hours=since_hours) + + logs = query.query( + level=level, + agent_id=agent_id, + feature_id=feature_id, + tool_name=tool_name, + search=search, + since=since, + limit=limit, + offset=offset, + ) + + total = query.count(level=level, agent_id=agent_id, feature_id=feature_id, since=since) + + return LogQueryResponse( + logs=[LogEntry(**log) for log in logs], + total=total, + limit=limit, + offset=offset, + ) + + except Exception as e: + logger.exception(f"Error querying logs: {e}") + raise HTTPException(status_code=500, detail=f"Query failed: {str(e)}") + + +@router.get("/{project_name}/timeline", response_model=TimelineResponse) +async def get_timeline( + project_name: str, + since_hours: int = Query(24, ge=1, le=168, description="Hours to look back"), + bucket_minutes: int = Query(5, ge=1, le=60, description="Bucket size in minutes"), +): + """ + Get activity timeline bucketed by time intervals. + + Useful for visualizing agent activity over time. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from structured_logging import get_log_query + + query = get_log_query(project_dir) + + since = datetime.now(timezone.utc) - timedelta(hours=since_hours) + buckets = query.get_timeline(since=since, bucket_minutes=bucket_minutes) + + return TimelineResponse( + buckets=[TimelineBucket(**b) for b in buckets], + bucket_minutes=bucket_minutes, + ) + + except Exception as e: + logger.exception(f"Error getting timeline: {e}") + raise HTTPException(status_code=500, detail=f"Timeline query failed: {str(e)}") + + +@router.get("/{project_name}/stats", response_model=StatsResponse) +async def get_stats( + project_name: str, + since_hours: Optional[int] = Query(None, description="Hours to look back"), +): + """ + Get log statistics per agent. + + Shows total logs, info/warn/error counts, and time range per agent. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from structured_logging import get_log_query + + query = get_log_query(project_dir) + + since = None + if since_hours: + since = datetime.now(timezone.utc) - timedelta(hours=since_hours) + + agents = query.get_agent_stats(since=since) + total = sum(a.get("total", 0) for a in agents) + + return StatsResponse( + agents=[AgentStats(**a) for a in agents], + total_logs=total, + ) + + except Exception as e: + logger.exception(f"Error getting stats: {e}") + raise HTTPException(status_code=500, detail=f"Stats query failed: {str(e)}") + + +@router.post("/export", response_model=ExportResponse) +async def export_logs(request: ExportRequest): + """ + Export logs to a downloadable file. + + Supports JSON, JSONL, and CSV formats. + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + try: + from structured_logging import get_log_query + + query = get_log_query(project_dir) + + since = None + if request.since_hours: + since = datetime.now(timezone.utc) - timedelta(hours=request.since_hours) + + # Create temp file for export + suffix = f".{request.format}" if request.format != "jsonl" else ".jsonl" + timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") + filename = f"logs_{request.project_name}_{timestamp}{suffix}" + + # Export to project's .autocoder/exports directory + export_dir = project_dir / ".autocoder" / "exports" + export_dir.mkdir(parents=True, exist_ok=True) + output_path = export_dir / filename + + count = query.export_logs( + output_path=output_path, + format=request.format, + level=request.level, + agent_id=request.agent_id, + feature_id=request.feature_id, + since=since, + ) + + return ExportResponse( + filename=filename, + count=count, + format=request.format, + ) + + except Exception as e: + logger.exception(f"Error exporting logs: {e}") + raise HTTPException(status_code=500, detail=f"Export failed: {str(e)}") + + +@router.get("/{project_name}/download/{filename}") +async def download_export(project_name: str, filename: str): + """Download an exported log file.""" + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + # Security: validate filename to prevent path traversal + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + export_path = project_dir / ".autocoder" / "exports" / filename + if not export_path.exists(): + raise HTTPException(status_code=404, detail="Export file not found") + + return FileResponse( + path=export_path, + filename=filename, + media_type="application/octet-stream", + ) diff --git a/server/routers/review.py b/server/routers/review.py new file mode 100644 index 00000000..e1c94235 --- /dev/null +++ b/server/routers/review.py @@ -0,0 +1,349 @@ +""" +Review Agent API Router +======================= + +REST API endpoints for automatic code review. + +Endpoints: +- POST /api/review/run - Run code review on a project +- GET /api/review/reports/{project_name} - List review reports +- GET /api/review/reports/{project_name}/{filename} - Get specific report +- POST /api/review/create-features - Create features from review issues +""" + +import json +import logging +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +from registry import get_project_path +from review_agent import ReviewAgent + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/review", tags=["review"]) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class RunReviewRequest(BaseModel): + """Request to run a code review.""" + + project_name: str = Field(..., description="Project name or path") + commits: Optional[list[str]] = Field(None, description="Specific commits to review") + files: Optional[list[str]] = Field(None, description="Specific files to review") + save_report: bool = Field(True, description="Whether to save the report") + checks: Optional[dict] = Field( + None, + description="Which checks to run (dead_code, naming, error_handling, security, complexity)", + ) + + +class ReviewIssueResponse(BaseModel): + """A review issue.""" + + category: str + severity: str + title: str + description: str + file_path: str + line_number: Optional[int] = None + code_snippet: Optional[str] = None + suggestion: Optional[str] = None + + +class ReviewSummary(BaseModel): + """Summary of review results.""" + + total_issues: int + by_severity: dict + by_category: dict + + +class RunReviewResponse(BaseModel): + """Response from running a review.""" + + project_dir: str + review_time: str + commits_reviewed: list[str] + files_reviewed: list[str] + issues: list[ReviewIssueResponse] + summary: ReviewSummary + report_path: Optional[str] = None + + +class ReportListItem(BaseModel): + """A review report in the list.""" + + filename: str + review_time: str + total_issues: int + errors: int + warnings: int + + +class ReportListResponse(BaseModel): + """List of review reports.""" + + reports: list[ReportListItem] + count: int + + +class CreateFeaturesRequest(BaseModel): + """Request to create features from review issues.""" + + project_name: str = Field(..., description="Project name") + issues: list[dict] = Field(..., description="Issues to convert to features") + + +class CreateFeaturesResponse(BaseModel): + """Response from creating features.""" + + created: int + features: list[dict] + + +# ============================================================================ +# Helper Functions +# ============================================================================ + + +def get_project_dir(project_name: str) -> Path: + """Get project directory from name or path.""" + # Try to get from registry + project_path = get_project_path(project_name) + if project_path: + return Path(project_path) + + # Check if it's a direct path + path = Path(project_name) + if path.exists() and path.is_dir(): + return path + + raise HTTPException(status_code=404, detail=f"Project not found: {project_name}") + + +# ============================================================================ +# Endpoints +# ============================================================================ + + +@router.post("/run", response_model=RunReviewResponse) +async def run_code_review(request: RunReviewRequest): + """ + Run code review on a project. + + Analyzes code for common issues: + - Dead code (unused imports, variables) + - Naming convention violations + - Missing error handling + - Security vulnerabilities + - Code complexity + """ + project_dir = get_project_dir(request.project_name) + project_dir_resolved = project_dir.resolve() + + # Validate files to prevent path traversal attacks + validated_files = None + if request.files: + validated_files = [] + for file_path in request.files: + # Resolve the full path and validate it's within project boundaries + full_path = (project_dir / file_path).resolve() + if not full_path.is_relative_to(project_dir_resolved): + raise HTTPException( + status_code=400, + detail=f"Invalid file path: path traversal detected in '{file_path}'", + ) + # Store the relative path for the agent + validated_files.append(str(full_path.relative_to(project_dir_resolved))) + + # Configure checks + check_config = request.checks or {} + + try: + agent = ReviewAgent( + project_dir=project_dir, + check_dead_code=check_config.get("dead_code", True), + check_naming=check_config.get("naming", True), + check_error_handling=check_config.get("error_handling", True), + check_security=check_config.get("security", True), + check_complexity=check_config.get("complexity", True), + ) + + report = agent.review( + commits=request.commits, + files=validated_files, + ) + + report_path = None + if request.save_report: + saved_path = agent.save_report(report) + report_path = str(saved_path.relative_to(project_dir)) + + report_dict = report.to_dict() + + return RunReviewResponse( + project_dir=report_dict["project_dir"], + review_time=report_dict["review_time"], + commits_reviewed=report_dict["commits_reviewed"], + files_reviewed=report_dict["files_reviewed"], + issues=[ReviewIssueResponse(**i) for i in report_dict["issues"]], + summary=ReviewSummary(**report_dict["summary"]), + report_path=report_path, + ) + + except Exception as e: + logger.error(f"Review failed for {project_dir}: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.get("/reports/{project_name}", response_model=ReportListResponse) +async def list_reports(project_name: str): + """ + List all review reports for a project. + """ + project_dir = get_project_dir(project_name) + reports_dir = project_dir / ".autocoder" / "review-reports" + + if not reports_dir.exists(): + return ReportListResponse(reports=[], count=0) + + reports = [] + for report_file in sorted(reports_dir.glob("review_*.json"), reverse=True): + try: + with open(report_file) as f: + data = json.load(f) + + summary = data.get("summary", {}) + by_severity = summary.get("by_severity", {}) + + reports.append( + ReportListItem( + filename=report_file.name, + review_time=data.get("review_time", ""), + total_issues=summary.get("total_issues", 0), + errors=by_severity.get("error", 0), + warnings=by_severity.get("warning", 0), + ) + ) + except Exception as e: + logger.warning(f"Error reading report {report_file}: {e}") + continue + + return ReportListResponse(reports=reports, count=len(reports)) + + +@router.get("/reports/{project_name}/{filename}") +async def get_report(project_name: str, filename: str): + """ + Get a specific review report. + """ + # Validate filename FIRST to prevent path traversal + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + project_dir = get_project_dir(project_name) + report_path = project_dir / ".autocoder" / "review-reports" / filename + + if not report_path.exists(): + raise HTTPException(status_code=404, detail=f"Report not found: {filename}") + + try: + with open(report_path) as f: + return json.load(f) + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error reading report: {e}") + + +@router.post("/create-features", response_model=CreateFeaturesResponse) +async def create_features_from_issues(request: CreateFeaturesRequest): + """ + Create features from review issues. + + Converts review issues into trackable features that can be assigned + to coding agents for resolution. + """ + from api.database import Feature, get_session + + project_dir = get_project_dir(request.project_name) + db_path = project_dir / "features.db" + + if not db_path.exists(): + raise HTTPException(status_code=404, detail="Project database not found") + + created_features = [] + session = None + + try: + session = get_session(db_path) + + # Get max priority for ordering + max_priority = session.query(Feature.priority).order_by(Feature.priority.desc()).first() + current_priority = (max_priority[0] if max_priority else 0) + 1 + + for issue in request.issues: + # Create feature from issue + feature = Feature( + priority=current_priority, + category=issue.get("category", "Code Review"), + name=issue.get("name", issue.get("title", "Review Issue")), + description=issue.get("description", ""), + steps=json.dumps(issue.get("steps", ["Fix the identified issue"])), + passes=False, + in_progress=False, + ) + + session.add(feature) + current_priority += 1 + + created_features.append( + { + "priority": feature.priority, + "category": feature.category, + "name": feature.name, + "description": feature.description, + } + ) + + session.commit() + + return CreateFeaturesResponse( + created=len(created_features), + features=created_features, + ) + + except Exception as e: + logger.error(f"Failed to create features: {e}") + raise HTTPException(status_code=500, detail=str(e)) + finally: + if session is not None: + session.close() + + +@router.delete("/reports/{project_name}/{filename}") +async def delete_report(project_name: str, filename: str): + """ + Delete a specific review report. + """ + project_dir = get_project_dir(project_name) + report_path = project_dir / ".autocoder" / "review-reports" / filename + + # Validate filename to prevent path traversal + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + if not report_path.exists(): + raise HTTPException(status_code=404, detail=f"Report not found: {filename}") + + try: + report_path.unlink() + return {"deleted": True, "filename": filename} + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error deleting report: {e}") diff --git a/server/routers/security.py b/server/routers/security.py new file mode 100644 index 00000000..6d9e0fa1 --- /dev/null +++ b/server/routers/security.py @@ -0,0 +1,211 @@ +""" +Security Router +=============== + +REST API endpoints for security scanning. + +Endpoints: +- POST /api/security/scan - Run security scan on a project +- GET /api/security/reports - List scan reports +- GET /api/security/reports/{filename} - Get a specific report +""" + +import json +import logging +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/security", tags=["security"]) + + +def _get_project_path(project_name: str) -> Path | None: + """Get project path from registry.""" + from registry import get_project_path + + return get_project_path(project_name) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class ScanRequest(BaseModel): + """Request to run a security scan.""" + + project_name: str = Field(..., description="Name of the registered project") + scan_dependencies: bool = Field(True, description="Run npm audit / pip-audit") + scan_secrets: bool = Field(True, description="Scan for hardcoded secrets") + scan_code: bool = Field(True, description="Scan for code vulnerability patterns") + + +class VulnerabilityInfo(BaseModel): + """Information about a detected vulnerability.""" + + type: str + severity: str + title: str + description: str + file_path: Optional[str] = None + line_number: Optional[int] = None + code_snippet: Optional[str] = None + recommendation: Optional[str] = None + cwe_id: Optional[str] = None + package_name: Optional[str] = None + package_version: Optional[str] = None + + +class ScanSummary(BaseModel): + """Summary of scan results.""" + + total_issues: int + critical: int + high: int + medium: int + low: int + has_critical_or_high: bool + + +class ScanResponse(BaseModel): + """Response from security scan.""" + + project_dir: str + scan_time: str + vulnerabilities: list[VulnerabilityInfo] + summary: ScanSummary + scans_run: list[str] + report_saved: bool + + +class ReportListResponse(BaseModel): + """Response listing available reports.""" + + reports: list[str] + count: int + + +# ============================================================================ +# REST Endpoints +# ============================================================================ + + +@router.post("/scan", response_model=ScanResponse) +async def run_security_scan(request: ScanRequest): + """ + Run a security scan on a project. + + Scans for: + - Vulnerable dependencies (npm audit, pip-audit) + - Hardcoded secrets (API keys, passwords, tokens) + - Code vulnerability patterns (SQL injection, XSS, etc.) + + Results are saved to .autocoder/security-reports/ + """ + project_dir = _get_project_path(request.project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + if not project_dir.exists(): + raise HTTPException(status_code=404, detail="Project directory not found") + + try: + from security_scanner import scan_project + + result = scan_project( + project_dir, + scan_dependencies=request.scan_dependencies, + scan_secrets=request.scan_secrets, + scan_code=request.scan_code, + ) + + return ScanResponse( + project_dir=result.project_dir, + scan_time=result.scan_time, + vulnerabilities=[ + VulnerabilityInfo(**v.to_dict()) for v in result.vulnerabilities + ], + summary=ScanSummary(**result.summary), + scans_run=result.scans_run, + report_saved=True, + ) + + except Exception as e: + logger.exception(f"Error running security scan: {e}") + raise HTTPException(status_code=500, detail=f"Scan failed: {str(e)}") + + +@router.get("/reports/{project_name}", response_model=ReportListResponse) +async def list_reports(project_name: str): + """ + List available security scan reports for a project. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + reports_dir = project_dir / ".autocoder" / "security-reports" + if not reports_dir.exists(): + return ReportListResponse(reports=[], count=0) + + reports = sorted( + [f.name for f in reports_dir.glob("security_scan_*.json")], + reverse=True, + ) + + return ReportListResponse(reports=reports, count=len(reports)) + + +@router.get("/reports/{project_name}/{filename}") +async def get_report(project_name: str, filename: str): + """ + Get a specific security scan report. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + # Security: validate filename to prevent path traversal + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + if not filename.startswith("security_scan_") or not filename.endswith(".json"): + raise HTTPException(status_code=400, detail="Invalid report filename") + + report_path = project_dir / ".autocoder" / "security-reports" / filename + if not report_path.exists(): + raise HTTPException(status_code=404, detail="Report not found") + + try: + with open(report_path) as f: + return json.load(f) + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error reading report: {str(e)}") + + +@router.get("/latest/{project_name}") +async def get_latest_report(project_name: str): + """ + Get the most recent security scan report for a project. + """ + project_dir = _get_project_path(project_name) + if not project_dir: + raise HTTPException(status_code=404, detail="Project not found") + + reports_dir = project_dir / ".autocoder" / "security-reports" + if not reports_dir.exists(): + raise HTTPException(status_code=404, detail="No reports found") + + reports = sorted(reports_dir.glob("security_scan_*.json"), reverse=True) + if not reports: + raise HTTPException(status_code=404, detail="No reports found") + + try: + with open(reports[0]) as f: + return json.load(f) + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error reading report: {str(e)}") diff --git a/server/routers/templates.py b/server/routers/templates.py new file mode 100644 index 00000000..3f9106df --- /dev/null +++ b/server/routers/templates.py @@ -0,0 +1,328 @@ +""" +Templates Router +================ + +REST API endpoints for project templates. + +Endpoints: +- GET /api/templates - List all available templates +- GET /api/templates/{template_id} - Get template details +- POST /api/templates/preview - Preview app_spec.txt generation +- POST /api/templates/apply - Apply template to new project +""" + +import logging +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/templates", tags=["templates"]) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class TechStackInfo(BaseModel): + """Technology stack information.""" + + frontend: Optional[str] = None + backend: Optional[str] = None + database: Optional[str] = None + auth: Optional[str] = None + styling: Optional[str] = None + hosting: Optional[str] = None + + +class DesignTokensInfo(BaseModel): + """Design tokens information.""" + + colors: dict[str, str] = {} + spacing: list[int] = [] + fonts: dict[str, str] = {} + border_radius: dict[str, str] = {} + + +class TemplateInfo(BaseModel): + """Template summary information.""" + + id: str + name: str + description: str + estimated_features: int + tags: list[str] = [] + difficulty: str = "intermediate" + + +class TemplateDetail(BaseModel): + """Full template details.""" + + id: str + name: str + description: str + tech_stack: TechStackInfo + feature_categories: dict[str, list[str]] + design_tokens: DesignTokensInfo + estimated_features: int + tags: list[str] = [] + difficulty: str = "intermediate" + + +class TemplateListResponse(BaseModel): + """Response with list of templates.""" + + templates: list[TemplateInfo] + count: int + + +class PreviewRequest(BaseModel): + """Request to preview app_spec.txt.""" + + template_id: str = Field(..., description="Template identifier") + app_name: str = Field(..., description="Application name") + customizations: Optional[dict] = Field(None, description="Optional customizations") + + +class PreviewResponse(BaseModel): + """Response with app_spec.txt preview.""" + + template_id: str + app_name: str + app_spec_content: str + feature_count: int + + +class ApplyRequest(BaseModel): + """Request to apply template to a project.""" + + template_id: str = Field(..., description="Template identifier") + project_name: str = Field(..., description="Name for the new project") + project_dir: str = Field(..., description="Directory for the project") + customizations: Optional[dict] = Field(None, description="Optional customizations") + + +class ApplyResponse(BaseModel): + """Response from applying template.""" + + success: bool + project_name: str + project_dir: str + app_spec_path: str + feature_count: int + message: str + + +# ============================================================================ +# REST Endpoints +# ============================================================================ + + +@router.get("", response_model=TemplateListResponse) +async def list_templates(): + """ + List all available templates. + + Returns basic information about each template. + """ + try: + from templates import list_templates as get_templates + + templates = get_templates() + + return TemplateListResponse( + templates=[ + TemplateInfo( + id=t.id, + name=t.name, + description=t.description, + estimated_features=t.estimated_features, + tags=t.tags, + difficulty=t.difficulty, + ) + for t in templates + ], + count=len(templates), + ) + + except Exception as e: + logger.exception(f"Error listing templates: {e}") + raise HTTPException(status_code=500, detail=f"Failed to list templates: {str(e)}") + + +@router.get("/{template_id}", response_model=TemplateDetail) +async def get_template(template_id: str): + """ + Get detailed information about a specific template. + """ + try: + from templates import get_template as load_template + + template = load_template(template_id) + + if not template: + raise HTTPException(status_code=404, detail=f"Template not found: {template_id}") + + return TemplateDetail( + id=template.id, + name=template.name, + description=template.description, + tech_stack=TechStackInfo( + frontend=template.tech_stack.frontend, + backend=template.tech_stack.backend, + database=template.tech_stack.database, + auth=template.tech_stack.auth, + styling=template.tech_stack.styling, + hosting=template.tech_stack.hosting, + ), + feature_categories=template.feature_categories, + design_tokens=DesignTokensInfo( + colors=template.design_tokens.colors, + spacing=template.design_tokens.spacing, + fonts=template.design_tokens.fonts, + border_radius=template.design_tokens.border_radius, + ), + estimated_features=template.estimated_features, + tags=template.tags, + difficulty=template.difficulty, + ) + + except HTTPException: + raise + except Exception as e: + logger.exception(f"Error getting template: {e}") + raise HTTPException(status_code=500, detail=f"Failed to get template: {str(e)}") + + +@router.post("/preview", response_model=PreviewResponse) +async def preview_template(request: PreviewRequest): + """ + Preview the app_spec.txt that would be generated from a template. + + Does not create any files - just returns the content. + """ + try: + from templates import generate_app_spec, generate_features, get_template + + template = get_template(request.template_id) + if not template: + raise HTTPException(status_code=404, detail=f"Template not found: {request.template_id}") + + app_spec_content = generate_app_spec( + template, + request.app_name, + request.customizations, + ) + + features = generate_features(template) + + return PreviewResponse( + template_id=request.template_id, + app_name=request.app_name, + app_spec_content=app_spec_content, + feature_count=len(features), + ) + + except HTTPException: + raise + except Exception as e: + logger.exception(f"Error previewing template: {e}") + raise HTTPException(status_code=500, detail=f"Preview failed: {str(e)}") + + +@router.post("/apply", response_model=ApplyResponse) +async def apply_template(request: ApplyRequest): + """ + Apply a template to create a new project. + + Creates the project directory, prompts folder, and app_spec.txt. + Does NOT register the project or create features - use the projects API for that. + """ + try: + from templates import generate_app_spec, generate_features, get_template + + template = get_template(request.template_id) + if not template: + raise HTTPException(status_code=404, detail=f"Template not found: {request.template_id}") + + # Validate and create project directory + project_dir = Path(request.project_dir).resolve() + + # Prevent path traversal - ensure the path doesn't escape via ../ + if ".." in request.project_dir: + raise HTTPException(status_code=400, detail="Invalid project directory: path traversal detected") + + # Ensure project_dir is an absolute path within allowed boundaries + # (i.e., not trying to write to system directories) + project_dir_str = str(project_dir) + sensitive_paths = ["/etc", "/usr", "/bin", "/sbin", "/var", "/root", "/home/root"] + if any(project_dir_str.startswith(p) for p in sensitive_paths): + raise HTTPException(status_code=400, detail="Invalid project directory: cannot write to system paths") + + prompts_dir = project_dir / "prompts" + prompts_dir.mkdir(parents=True, exist_ok=True) + + # Generate and save app_spec.txt + app_spec_content = generate_app_spec( + template, + request.project_name, + request.customizations, + ) + + app_spec_path = prompts_dir / "app_spec.txt" + with open(app_spec_path, "w") as f: + f.write(app_spec_content) + + features = generate_features(template) + + return ApplyResponse( + success=True, + project_name=request.project_name, + project_dir=str(project_dir), + app_spec_path=str(app_spec_path), + feature_count=len(features), + message=f"Template '{template.name}' applied successfully. Register the project and run the initializer to create features.", + ) + + except HTTPException: + raise + except Exception as e: + logger.exception(f"Error applying template: {e}") + raise HTTPException(status_code=500, detail=f"Apply failed: {str(e)}") + + +@router.get("/{template_id}/features") +async def get_template_features(template_id: str): + """ + Get the features that would be created from a template. + + Returns features in bulk_create format. + """ + try: + from templates import generate_features, get_template + + template = get_template(template_id) + if not template: + raise HTTPException(status_code=404, detail=f"Template not found: {template_id}") + + features = generate_features(template) + + return { + "template_id": template_id, + "features": features, + "count": len(features), + "by_category": { + category: len(feature_names) + for category, feature_names in template.feature_categories.items() + }, + } + + except HTTPException: + raise + except Exception as e: + logger.exception(f"Error getting template features: {e}") + raise HTTPException(status_code=500, detail=f"Failed to get features: {str(e)}") diff --git a/server/routers/visual_regression.py b/server/routers/visual_regression.py new file mode 100644 index 00000000..6698dcd2 --- /dev/null +++ b/server/routers/visual_regression.py @@ -0,0 +1,419 @@ +""" +Visual Regression API Router +============================ + +REST API endpoints for visual regression testing. + +Endpoints: +- POST /api/visual/test - Run visual tests +- GET /api/visual/baselines/{project_name} - List baselines +- GET /api/visual/reports/{project_name} - List test reports +- GET /api/visual/reports/{project_name}/{filename} - Get specific report +- POST /api/visual/update-baseline - Accept current as baseline +- DELETE /api/visual/baselines/{project_name}/{name}/{viewport} - Delete baseline +- GET /api/visual/snapshot/{project_name}/{type}/{filename} - Get snapshot image +""" + +import json +import logging +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException +from fastapi.responses import FileResponse +from pydantic import BaseModel, Field + +from registry import get_project_path +from visual_regression import ( + Viewport, + VisualRegressionTester, +) + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/visual", tags=["visual-regression"]) + + +# ============================================================================ +# Request/Response Models +# ============================================================================ + + +class RouteConfig(BaseModel): + """Route configuration for testing.""" + + path: str = Field(..., description="Route path (e.g., /dashboard)") + name: Optional[str] = Field(None, description="Test name (auto-generated from path if not provided)") + wait_for: Optional[str] = Field(None, description="CSS selector to wait for before capture") + + +class ViewportConfig(BaseModel): + """Viewport configuration.""" + + name: str + width: int + height: int + + +class RunTestsRequest(BaseModel): + """Request to run visual tests.""" + + project_name: str = Field(..., description="Project name") + base_url: str = Field(..., description="Base URL (e.g., http://localhost:3000)") + routes: Optional[list[RouteConfig]] = Field(None, description="Routes to test") + threshold: float = Field(0.1, description="Diff threshold percentage") + update_baseline: bool = Field(False, description="Update baselines instead of comparing") + viewports: Optional[list[ViewportConfig]] = Field(None, description="Viewports to test") + + +class SnapshotResultResponse(BaseModel): + """Single snapshot result.""" + + name: str + viewport: str + baseline_path: Optional[str] = None + current_path: Optional[str] = None + diff_path: Optional[str] = None + diff_percentage: float = 0.0 + passed: bool = True + is_new: bool = False + error: Optional[str] = None + + +class TestSummary(BaseModel): + """Test summary statistics.""" + + total: int + passed: int + failed: int + new: int + + +class TestReportResponse(BaseModel): + """Test report response.""" + + project_dir: str + test_time: str + results: list[SnapshotResultResponse] + summary: TestSummary + + +class BaselineItem(BaseModel): + """Baseline snapshot item.""" + + name: str + viewport: str + filename: str + size: int + modified: str + + +class BaselineListResponse(BaseModel): + """List of baseline snapshots.""" + + baselines: list[BaselineItem] + count: int + + +class ReportListItem(BaseModel): + """Report list item.""" + + filename: str + test_time: str + total: int + passed: int + failed: int + + +class ReportListResponse(BaseModel): + """List of test reports.""" + + reports: list[ReportListItem] + count: int + + +class UpdateBaselineRequest(BaseModel): + """Request to update a baseline.""" + + project_name: str + name: str + viewport: str + + +# ============================================================================ +# Helper Functions +# ============================================================================ + + +def get_project_dir(project_name: str) -> Path: + """Get project directory from name or path.""" + project_path = get_project_path(project_name) + if project_path: + return Path(project_path) + + path = Path(project_name) + if path.exists() and path.is_dir(): + return path + + raise HTTPException(status_code=404, detail=f"Project not found: {project_name}") + + +# ============================================================================ +# Endpoints +# ============================================================================ + + +@router.post("/test", response_model=TestReportResponse) +async def run_tests(request: RunTestsRequest): + """ + Run visual regression tests. + + Captures screenshots of specified routes and compares with baselines. + """ + project_dir = get_project_dir(request.project_name) + + try: + # Convert routes + routes = None + if request.routes: + routes = [ + { + "path": r.path, + "name": r.name, + "wait_for": r.wait_for, + } + for r in request.routes + ] + + # Configure viewports + viewports = None + if request.viewports: + viewports = [ + Viewport(name=v.name, width=v.width, height=v.height) + for v in request.viewports + ] + + # Create tester with custom viewports + tester = VisualRegressionTester( + project_dir=project_dir, + threshold=request.threshold, + viewports=viewports or [Viewport.desktop()], + ) + + # Run tests + if routes: + report = await tester.test_routes( + request.base_url, routes, request.update_baseline + ) + else: + # Default to home page + report = await tester.test_page( + request.base_url, "home", update_baseline=request.update_baseline + ) + + # Save report + tester.save_report(report) + + # Convert to response + return TestReportResponse( + project_dir=report.project_dir, + test_time=report.test_time, + results=[ + SnapshotResultResponse(**r.to_dict()) for r in report.results + ], + summary=TestSummary( + total=report.total, + passed=report.passed, + failed=report.failed, + new=report.new, + ), + ) + + except Exception as e: + logger.error(f"Visual test failed: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.get("/baselines/{project_name}", response_model=BaselineListResponse) +async def list_baselines(project_name: str): + """ + List all baseline snapshots for a project. + """ + project_dir = get_project_dir(project_name) + + try: + tester = VisualRegressionTester(project_dir) + baselines = tester.list_baselines() + + return BaselineListResponse( + baselines=[BaselineItem(**b) for b in baselines], + count=len(baselines), + ) + except Exception as e: + logger.error(f"Error listing baselines: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.get("/reports/{project_name}", response_model=ReportListResponse) +async def list_reports(project_name: str): + """ + List all visual test reports for a project. + """ + project_dir = get_project_dir(project_name) + reports_dir = project_dir / ".visual-snapshots" / "reports" + + if not reports_dir.exists(): + return ReportListResponse(reports=[], count=0) + + reports = [] + for report_file in sorted(reports_dir.glob("visual_test_*.json"), reverse=True): + try: + with open(report_file) as f: + data = json.load(f) + + summary = data.get("summary", {}) + reports.append( + ReportListItem( + filename=report_file.name, + test_time=data.get("test_time", ""), + total=summary.get("total", 0), + passed=summary.get("passed", 0), + failed=summary.get("failed", 0), + ) + ) + except Exception as e: + logger.warning(f"Error reading report {report_file}: {e}") + + return ReportListResponse(reports=reports, count=len(reports)) + + +@router.get("/reports/{project_name}/{filename}") +async def get_report(project_name: str, filename: str): + """ + Get a specific visual test report. + """ + project_dir = get_project_dir(project_name) + + # Validate filename + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + report_path = project_dir / ".visual-snapshots" / "reports" / filename + + if not report_path.exists(): + raise HTTPException(status_code=404, detail=f"Report not found: {filename}") + + try: + with open(report_path) as f: + return json.load(f) + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error reading report: {e}") + + +@router.post("/update-baseline") +async def update_baseline(request: UpdateBaselineRequest): + """ + Accept current screenshot as new baseline. + """ + project_dir = get_project_dir(request.project_name) + + try: + tester = VisualRegressionTester(project_dir) + success = tester.update_baseline(request.name, request.viewport) + + if success: + return {"updated": True, "name": request.name, "viewport": request.viewport} + else: + raise HTTPException( + status_code=404, + detail=f"Current snapshot not found: {request.name}_{request.viewport}", + ) + except Exception as e: + logger.error(f"Error updating baseline: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.delete("/baselines/{project_name}/{name}/{viewport}") +async def delete_baseline(project_name: str, name: str, viewport: str): + """ + Delete a baseline snapshot. + """ + project_dir = get_project_dir(project_name) + + # Validate inputs + if ".." in name or "/" in name or "\\" in name: + raise HTTPException(status_code=400, detail="Invalid name") + if ".." in viewport or "/" in viewport or "\\" in viewport: + raise HTTPException(status_code=400, detail="Invalid viewport") + + try: + tester = VisualRegressionTester(project_dir) + success = tester.delete_baseline(name, viewport) + + if success: + return {"deleted": True, "name": name, "viewport": viewport} + else: + raise HTTPException( + status_code=404, + detail=f"Baseline not found: {name}_{viewport}", + ) + except Exception as e: + logger.error(f"Error deleting baseline: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + +@router.get("/snapshot/{project_name}/{snapshot_type}/{filename}") +async def get_snapshot(project_name: str, snapshot_type: str, filename: str): + """ + Get a snapshot image. + + Args: + project_name: Project name + snapshot_type: Type of snapshot (baselines, current, diffs) + filename: Image filename + """ + project_dir = get_project_dir(project_name) + + # Validate inputs + valid_types = ["baselines", "current", "diffs"] + if snapshot_type not in valid_types: + raise HTTPException( + status_code=400, + detail=f"Invalid snapshot type. Valid types: {', '.join(valid_types)}", + ) + + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + if not filename.endswith(".png"): + raise HTTPException(status_code=400, detail="Only PNG files are supported") + + snapshot_path = project_dir / ".visual-snapshots" / snapshot_type / filename + + if not snapshot_path.exists(): + raise HTTPException(status_code=404, detail=f"Snapshot not found: {filename}") + + return FileResponse(snapshot_path, media_type="image/png") + + +@router.delete("/reports/{project_name}/{filename}") +async def delete_report(project_name: str, filename: str): + """ + Delete a visual test report. + """ + project_dir = get_project_dir(project_name) + + # Validate filename + if ".." in filename or "/" in filename or "\\" in filename: + raise HTTPException(status_code=400, detail="Invalid filename") + + report_path = project_dir / ".visual-snapshots" / "reports" / filename + + if not report_path.exists(): + raise HTTPException(status_code=404, detail=f"Report not found: {filename}") + + try: + report_path.unlink() + return {"deleted": True, "filename": filename} + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error deleting report: {e}") diff --git a/server/services/assistant_database.py b/server/services/assistant_database.py index 15453109..91d44cc6 100644 --- a/server/services/assistant_database.py +++ b/server/services/assistant_database.py @@ -7,6 +7,7 @@ """ import logging +import threading from datetime import datetime, timezone from pathlib import Path from typing import Optional @@ -22,6 +23,10 @@ # Key: project directory path (as posix string), Value: SQLAlchemy engine _engine_cache: dict[str, object] = {} +# Lock for thread-safe access to the engine cache +# Prevents race conditions when multiple threads create engines simultaneously +_cache_lock = threading.Lock() + def _utc_now() -> datetime: """Return current UTC time. Replacement for deprecated datetime.utcnow().""" @@ -64,17 +69,33 @@ def get_engine(project_dir: Path): Uses a cache to avoid creating new engines for each request, which improves performance by reusing database connections. + + Thread-safe: Uses a lock to prevent race conditions when multiple threads + try to create engines simultaneously for the same project. """ cache_key = project_dir.as_posix() - if cache_key not in _engine_cache: - db_path = get_db_path(project_dir) - # Use as_posix() for cross-platform compatibility with SQLite connection strings - db_url = f"sqlite:///{db_path.as_posix()}" - engine = create_engine(db_url, echo=False) - Base.metadata.create_all(engine) - _engine_cache[cache_key] = engine - logger.debug(f"Created new database engine for {cache_key}") + # Double-checked locking for thread safety and performance + if cache_key in _engine_cache: + return _engine_cache[cache_key] + + with _cache_lock: + # Check again inside the lock in case another thread created it + if cache_key not in _engine_cache: + db_path = get_db_path(project_dir) + # Use as_posix() for cross-platform compatibility with SQLite connection strings + db_url = f"sqlite:///{db_path.as_posix()}" + engine = create_engine( + db_url, + echo=False, + connect_args={ + "check_same_thread": False, + "timeout": 30, # Wait up to 30s for locks + } + ) + Base.metadata.create_all(engine) + _engine_cache[cache_key] = engine + logger.debug(f"Created new database engine for {cache_key}") return _engine_cache[cache_key] diff --git a/server/services/autocoder_config.py b/server/services/autocoder_config.py new file mode 100644 index 00000000..68f831a0 --- /dev/null +++ b/server/services/autocoder_config.py @@ -0,0 +1,377 @@ +""" +Autocoder Enhanced Configuration +================================ + +Centralized configuration system for all Autocoder features. +Extends the basic project_config.py with support for: +- Quality Gates +- Git Workflow +- Error Recovery +- CI/CD Integration +- Import Settings +- Completion Settings + +Configuration is stored in {project_dir}/.autocoder/config.json. +""" + +import copy +import json +import logging +from pathlib import Path +from typing import Any, TypedDict + +logger = logging.getLogger(__name__) + + +# ============================================================================= +# Type Definitions for Configuration Schema +# ============================================================================= + + +class QualityChecksConfig(TypedDict, total=False): + """Configuration for individual quality checks.""" + lint: bool + type_check: bool + unit_tests: bool + custom_script: str | None + + +class QualityGatesConfig(TypedDict, total=False): + """Configuration for quality gates feature.""" + enabled: bool + strict_mode: bool + checks: QualityChecksConfig + + +class GitWorkflowConfig(TypedDict, total=False): + """Configuration for git workflow feature.""" + mode: str # "feature_branches" | "trunk" | "none" + branch_prefix: str + auto_merge: bool + + +class ErrorRecoveryConfig(TypedDict, total=False): + """Configuration for error recovery feature.""" + max_retries: int + skip_threshold: int + escalate_threshold: int + auto_clear_on_startup: bool + + +class CompletionConfig(TypedDict, total=False): + """Configuration for completion behavior.""" + auto_stop_at_100: bool + max_regression_cycles: int + prompt_before_extra_cycles: bool + + +class EnvironmentConfig(TypedDict, total=False): + """Configuration for a deployment environment.""" + url: str + auto_deploy: bool + + +class CiCdConfig(TypedDict, total=False): + """Configuration for CI/CD integration.""" + provider: str # "github" | "gitlab" | "none" + environments: dict[str, EnvironmentConfig] + + +class ImportConfig(TypedDict, total=False): + """Configuration for project import feature.""" + default_feature_status: str # "pending" | "passing" + auto_detect_stack: bool + + +class SecurityScanningConfig(TypedDict, total=False): + """Configuration for security scanning feature.""" + enabled: bool + scan_dependencies: bool + scan_secrets: bool + scan_injection_patterns: bool + fail_on_high_severity: bool + + +class LoggingConfig(TypedDict, total=False): + """Configuration for enhanced logging feature.""" + enabled: bool + level: str # "debug" | "info" | "warn" | "error" + structured_output: bool + include_timestamps: bool + max_log_file_size_mb: int + + +class AutocoderConfig(TypedDict, total=False): + """Full Autocoder configuration schema.""" + version: str + dev_command: str | None + quality_gates: QualityGatesConfig + git_workflow: GitWorkflowConfig + error_recovery: ErrorRecoveryConfig + completion: CompletionConfig + ci_cd: CiCdConfig + import_settings: ImportConfig + security_scanning: SecurityScanningConfig + logging: LoggingConfig + + +# ============================================================================= +# Default Configuration Values +# ============================================================================= + + +DEFAULT_CONFIG: AutocoderConfig = { + "version": "1.0", + "dev_command": None, + "quality_gates": { + "enabled": True, + "strict_mode": True, + "checks": { + "lint": True, + "type_check": True, + "unit_tests": False, + "custom_script": None, + }, + }, + "git_workflow": { + "mode": "none", + "branch_prefix": "feature/", + "auto_merge": False, + }, + "error_recovery": { + "max_retries": 3, + "skip_threshold": 5, + "escalate_threshold": 7, + "auto_clear_on_startup": True, + }, + "completion": { + "auto_stop_at_100": True, + "max_regression_cycles": 3, + "prompt_before_extra_cycles": False, + }, + "ci_cd": { + "provider": "none", + "environments": {}, + }, + "import_settings": { + "default_feature_status": "pending", + "auto_detect_stack": True, + }, + "security_scanning": { + "enabled": True, + "scan_dependencies": True, + "scan_secrets": True, + "scan_injection_patterns": True, + "fail_on_high_severity": False, + }, + "logging": { + "enabled": True, + "level": "info", + "structured_output": True, + "include_timestamps": True, + "max_log_file_size_mb": 10, + }, +} + + +# ============================================================================= +# Configuration Loading and Saving +# ============================================================================= + + +def _get_config_path(project_dir: Path) -> Path: + """Get the path to the project config file.""" + return project_dir / ".autocoder" / "config.json" + + +def _deep_merge(base: dict, override: dict) -> dict: + """ + Deep merge two dictionaries. + + Values from override take precedence over base. + Nested dicts are merged recursively. + + Args: + base: Base dictionary with default values + override: Dictionary with override values + + Returns: + Merged dictionary + """ + result = copy.deepcopy(base) + + for key, value in override.items(): + if key in result and isinstance(result[key], dict) and isinstance(value, dict): + result[key] = _deep_merge(result[key], value) + else: + result[key] = value + + return result + + +def load_autocoder_config(project_dir: Path) -> AutocoderConfig: + """ + Load the full Autocoder configuration with defaults. + + Reads from .autocoder/config.json and merges with defaults. + If the config file doesn't exist or is invalid, returns defaults. + + Args: + project_dir: Path to the project directory + + Returns: + Full configuration with all sections populated + """ + config_path = _get_config_path(project_dir) + + if not config_path.exists(): + logger.debug("No config file found at %s, using defaults", config_path) + return copy.deepcopy(DEFAULT_CONFIG) + + try: + with open(config_path, "r", encoding="utf-8") as f: + user_config = json.load(f) + + if not isinstance(user_config, dict): + logger.warning( + "Invalid config format in %s: expected dict, got %s", + config_path, type(user_config).__name__ + ) + return copy.deepcopy(DEFAULT_CONFIG) + + # Merge user config with defaults + merged = _deep_merge(DEFAULT_CONFIG, user_config) + return merged + + except json.JSONDecodeError as e: + logger.warning("Failed to parse config at %s: %s", config_path, e) + return copy.deepcopy(DEFAULT_CONFIG) + except OSError as e: + logger.warning("Failed to read config at %s: %s", config_path, e) + return copy.deepcopy(DEFAULT_CONFIG) + + +def save_autocoder_config(project_dir: Path, config: AutocoderConfig) -> None: + """ + Save the Autocoder configuration to disk. + + Creates the .autocoder directory if it doesn't exist. + + Args: + project_dir: Path to the project directory + config: Configuration to save + + Raises: + OSError: If the file cannot be written + """ + config_path = _get_config_path(project_dir) + config_path.parent.mkdir(parents=True, exist_ok=True) + + try: + with open(config_path, "w", encoding="utf-8") as f: + json.dump(config, f, indent=2) + logger.debug("Saved config to %s", config_path) + except OSError as e: + logger.error("Failed to save config to %s: %s", config_path, e) + raise + + +def update_autocoder_config(project_dir: Path, updates: dict[str, Any]) -> AutocoderConfig: + """ + Update specific configuration values. + + Loads current config, applies updates, and saves. + + Args: + project_dir: Path to the project directory + updates: Dictionary with values to update (can be nested) + + Returns: + Updated configuration + """ + config = load_autocoder_config(project_dir) + merged = _deep_merge(config, updates) + save_autocoder_config(project_dir, merged) + return merged + + +# ============================================================================= +# Convenience Getters for Specific Sections +# ============================================================================= + + +def get_quality_gates_config(project_dir: Path) -> QualityGatesConfig: + """Get quality gates configuration for a project.""" + config = load_autocoder_config(project_dir) + return config.get("quality_gates", DEFAULT_CONFIG["quality_gates"]) + + +def get_git_workflow_config(project_dir: Path) -> GitWorkflowConfig: + """Get git workflow configuration for a project.""" + config = load_autocoder_config(project_dir) + return config.get("git_workflow", DEFAULT_CONFIG["git_workflow"]) + + +def get_error_recovery_config(project_dir: Path) -> ErrorRecoveryConfig: + """Get error recovery configuration for a project.""" + config = load_autocoder_config(project_dir) + return config.get("error_recovery", DEFAULT_CONFIG["error_recovery"]) + + +def get_completion_config(project_dir: Path) -> CompletionConfig: + """Get completion configuration for a project.""" + config = load_autocoder_config(project_dir) + return config.get("completion", DEFAULT_CONFIG["completion"]) + + +def get_security_scanning_config(project_dir: Path) -> SecurityScanningConfig: + """Get security scanning configuration for a project.""" + config = load_autocoder_config(project_dir) + return config.get("security_scanning", DEFAULT_CONFIG["security_scanning"]) + + +def get_logging_config(project_dir: Path) -> LoggingConfig: + """Get logging configuration for a project.""" + config = load_autocoder_config(project_dir) + return config.get("logging", DEFAULT_CONFIG["logging"]) + + +# ============================================================================= +# Feature Enable/Disable Checks +# ============================================================================= + + +def is_quality_gates_enabled(project_dir: Path) -> bool: + """Check if quality gates are enabled for a project.""" + config = get_quality_gates_config(project_dir) + return config.get("enabled", True) + + +def is_strict_quality_mode(project_dir: Path) -> bool: + """Check if strict quality mode is enabled (blocks feature_mark_passing on failure).""" + config = get_quality_gates_config(project_dir) + return config.get("enabled", True) and config.get("strict_mode", True) + + +def is_security_scanning_enabled(project_dir: Path) -> bool: + """Check if security scanning is enabled for a project.""" + config = get_security_scanning_config(project_dir) + return config.get("enabled", True) + + +def is_auto_clear_on_startup_enabled(project_dir: Path) -> bool: + """Check if auto-clear stuck features on startup is enabled.""" + config = get_error_recovery_config(project_dir) + return config.get("auto_clear_on_startup", True) + + +def is_auto_stop_at_100_enabled(project_dir: Path) -> bool: + """Check if agent should auto-stop when all features pass.""" + config = get_completion_config(project_dir) + return config.get("auto_stop_at_100", True) + + +def get_git_workflow_mode(project_dir: Path) -> str: + """Get the git workflow mode for a project.""" + config = get_git_workflow_config(project_dir) + return config.get("mode", "none") diff --git a/structured_logging.py b/structured_logging.py new file mode 100644 index 00000000..e75ab171 --- /dev/null +++ b/structured_logging.py @@ -0,0 +1,636 @@ +""" +Structured Logging Module +========================= + +Enhanced logging with structured JSON format, filtering, and export capabilities. + +Features: +- JSON-formatted logs with consistent schema +- Filter by agent, feature, level +- Full-text search +- Timeline view for agent activity +- Export logs for offline analysis + +Log Format: +{ + "timestamp": "2025-01-21T10:30:00.000Z", + "level": "info|warn|error", + "agent_id": "coding-42", + "feature_id": 42, + "tool_name": "feature_mark_passing", + "duration_ms": 150, + "message": "Feature marked as passing" +} +""" + +import json +import logging +import sqlite3 +import threading +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Literal, Optional + +# Type aliases +# Note: Python's logging uses "warning" but we normalize to "warn" for consistency +LogLevel = Literal["debug", "info", "warn", "warning", "error"] + + +@dataclass +class StructuredLogEntry: + """A structured log entry with all metadata.""" + + timestamp: str + level: LogLevel + message: str + agent_id: Optional[str] = None + feature_id: Optional[int] = None + tool_name: Optional[str] = None + duration_ms: Optional[int] = None + extra: dict = field(default_factory=dict) + + def to_dict(self) -> dict: + """Convert to dictionary, excluding None values.""" + result = { + "timestamp": self.timestamp, + "level": self.level, + "message": self.message, + } + if self.agent_id: + result["agent_id"] = self.agent_id + if self.feature_id is not None: + result["feature_id"] = self.feature_id + if self.tool_name: + result["tool_name"] = self.tool_name + if self.duration_ms is not None: + result["duration_ms"] = self.duration_ms + if self.extra: + result["extra"] = self.extra + return result + + def to_json(self) -> str: + """Convert to JSON string.""" + return json.dumps(self.to_dict()) + + +class StructuredLogHandler(logging.Handler): + """ + Custom logging handler that stores structured logs in SQLite. + + Thread-safe for concurrent agent logging. + """ + + def __init__( + self, + db_path: Path, + agent_id: Optional[str] = None, + max_entries: int = 10000, + ): + super().__init__() + self.db_path = db_path + self.agent_id = agent_id + self.max_entries = max_entries + self._lock = threading.Lock() + self._init_database() + + def _init_database(self) -> None: + """Initialize the SQLite database for logs.""" + with self._lock: + # Use context manager to ensure connection is closed on errors + with sqlite3.connect(self.db_path) as conn: + cursor = conn.cursor() + + # Enable WAL mode for better concurrency with parallel agents + # WAL allows readers and writers to work concurrently without blocking + cursor.execute("PRAGMA journal_mode=WAL") + + # Create logs table + cursor.execute(""" + CREATE TABLE IF NOT EXISTS logs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + timestamp TEXT NOT NULL, + level TEXT NOT NULL, + message TEXT NOT NULL, + agent_id TEXT, + feature_id INTEGER, + tool_name TEXT, + duration_ms INTEGER, + extra TEXT + ) + """) + + # Create indexes for common queries + cursor.execute(""" + CREATE INDEX IF NOT EXISTS idx_logs_timestamp + ON logs(timestamp) + """) + cursor.execute(""" + CREATE INDEX IF NOT EXISTS idx_logs_level + ON logs(level) + """) + cursor.execute(""" + CREATE INDEX IF NOT EXISTS idx_logs_agent_id + ON logs(agent_id) + """) + cursor.execute(""" + CREATE INDEX IF NOT EXISTS idx_logs_feature_id + ON logs(feature_id) + """) + + conn.commit() + + def emit(self, record: logging.LogRecord) -> None: + """Store a log record in the database.""" + try: + # Extract structured data from record + # Normalize "warning" -> "warn" for consistency + level = record.levelname.lower() + if level == "warning": + level = "warn" + entry = StructuredLogEntry( + timestamp=datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + level=level, + message=self.format(record), + agent_id=getattr(record, "agent_id", self.agent_id), + feature_id=getattr(record, "feature_id", None), + tool_name=getattr(record, "tool_name", None), + duration_ms=getattr(record, "duration_ms", None), + extra=getattr(record, "extra", {}), + ) + + with self._lock: + # Use context manager to ensure connection is closed on errors + with sqlite3.connect(self.db_path) as conn: + cursor = conn.cursor() + + cursor.execute( + """ + INSERT INTO logs + (timestamp, level, message, agent_id, feature_id, tool_name, duration_ms, extra) + VALUES (?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + entry.timestamp, + entry.level, + entry.message, + entry.agent_id, + entry.feature_id, + entry.tool_name, + entry.duration_ms, + json.dumps(entry.extra) if entry.extra else None, + ), + ) + + # Cleanup old entries if over limit + cursor.execute("SELECT COUNT(*) FROM logs") + count = cursor.fetchone()[0] + if count > self.max_entries: + delete_count = count - self.max_entries + cursor.execute( + """ + DELETE FROM logs WHERE id IN ( + SELECT id FROM logs ORDER BY timestamp ASC LIMIT ? + ) + """, + (delete_count,), + ) + + conn.commit() + + except Exception: + self.handleError(record) + + +class StructuredLogger: + """ + Enhanced logger with structured logging capabilities. + + Usage: + logger = StructuredLogger(project_dir, agent_id="coding-1") + logger.info("Starting feature", feature_id=42) + logger.error("Test failed", feature_id=42, tool_name="playwright") + """ + + def __init__( + self, + project_dir: Path, + agent_id: Optional[str] = None, + console_output: bool = True, + ): + self.project_dir = Path(project_dir) + self.agent_id = agent_id + self.db_path = self.project_dir / ".autocoder" / "logs.db" + + # Ensure directory exists + self.db_path.parent.mkdir(parents=True, exist_ok=True) + + # Setup logger with unique name per instance to avoid handler accumulation + # across tests and multiple invocations. Include project path hash for uniqueness. + import hashlib + path_hash = hashlib.md5(str(self.project_dir).encode()).hexdigest()[:8] + logger_name = f"autocoder.{agent_id or 'main'}.{path_hash}.{id(self)}" + self.logger = logging.getLogger(logger_name) + self.logger.setLevel(logging.DEBUG) + + # Clear existing handlers (for safety, though names should be unique) + self.logger.handlers.clear() + + # Add structured handler + self.handler = StructuredLogHandler(self.db_path, agent_id) + self.handler.setFormatter(logging.Formatter("%(message)s")) + self.logger.addHandler(self.handler) + + # Add console handler if requested + if console_output: + console = logging.StreamHandler() + console.setLevel(logging.INFO) + console.setFormatter( + logging.Formatter("%(asctime)s [%(levelname)s] %(message)s") + ) + self.logger.addHandler(console) + + def _log( + self, + level: str, + message: str, + feature_id: Optional[int] = None, + tool_name: Optional[str] = None, + duration_ms: Optional[int] = None, + **extra, + ) -> None: + """Internal logging method with structured data.""" + record_extra = { + "agent_id": self.agent_id, + "feature_id": feature_id, + "tool_name": tool_name, + "duration_ms": duration_ms, + "extra": extra, + } + + # Use LogRecord extras + getattr(self.logger, level)( + message, + extra=record_extra, + ) + + def debug(self, message: str, **kwargs) -> None: + """Log debug message.""" + self._log("debug", message, **kwargs) + + def info(self, message: str, **kwargs) -> None: + """Log info message.""" + self._log("info", message, **kwargs) + + def warn(self, message: str, **kwargs) -> None: + """Log warning message.""" + self._log("warning", message, **kwargs) + + def warning(self, message: str, **kwargs) -> None: + """Log warning message (alias).""" + self._log("warning", message, **kwargs) + + def error(self, message: str, **kwargs) -> None: + """Log error message.""" + self._log("error", message, **kwargs) + + +class LogQuery: + """ + Query interface for structured logs. + + Supports filtering, searching, and aggregation. + """ + + def __init__(self, db_path: Path): + self.db_path = db_path + + def _connect(self) -> sqlite3.Connection: + """Get database connection.""" + conn = sqlite3.connect(self.db_path) + conn.row_factory = sqlite3.Row + return conn + + def query( + self, + level: Optional[LogLevel] = None, + agent_id: Optional[str] = None, + feature_id: Optional[int] = None, + tool_name: Optional[str] = None, + search: Optional[str] = None, + since: Optional[datetime] = None, + until: Optional[datetime] = None, + limit: int = 100, + offset: int = 0, + ) -> list[dict]: + """ + Query logs with filters. + + Args: + level: Filter by log level + agent_id: Filter by agent ID + feature_id: Filter by feature ID + tool_name: Filter by tool name + search: Full-text search in message + since: Start datetime + until: End datetime + limit: Max results + offset: Pagination offset + + Returns: + List of log entries as dicts + """ + conn = self._connect() + cursor = conn.cursor() + + conditions = [] + params = [] + + if level: + conditions.append("level = ?") + params.append(level) + + if agent_id: + conditions.append("agent_id = ?") + params.append(agent_id) + + if feature_id is not None: + conditions.append("feature_id = ?") + params.append(feature_id) + + if tool_name: + conditions.append("tool_name = ?") + params.append(tool_name) + + if search: + conditions.append("message LIKE ?") + # Escape LIKE wildcards to prevent unexpected query behavior + escaped_search = search.replace("%", "\\%").replace("_", "\\_") + params.append(f"%{escaped_search}%") + + if since: + conditions.append("timestamp >= ?") + # Use consistent timestamp format with stored logs (Z suffix for UTC) + params.append(since.isoformat().replace("+00:00", "Z")) + + if until: + conditions.append("timestamp <= ?") + # Use consistent timestamp format with stored logs (Z suffix for UTC) + params.append(until.isoformat().replace("+00:00", "Z")) + + where_clause = " AND ".join(conditions) if conditions else "1=1" + + query = f""" + SELECT * FROM logs + WHERE {where_clause} + ORDER BY timestamp DESC + LIMIT ? OFFSET ? + """ + params.extend([limit, offset]) + + cursor.execute(query, params) + rows = cursor.fetchall() + conn.close() + + return [dict(row) for row in rows] + + def count( + self, + level: Optional[LogLevel] = None, + agent_id: Optional[str] = None, + feature_id: Optional[int] = None, + since: Optional[datetime] = None, + ) -> int: + """Count logs matching filters.""" + conn = self._connect() + cursor = conn.cursor() + + conditions = [] + params = [] + + if level: + conditions.append("level = ?") + params.append(level) + if agent_id: + conditions.append("agent_id = ?") + params.append(agent_id) + if feature_id is not None: + conditions.append("feature_id = ?") + params.append(feature_id) + if since: + conditions.append("timestamp >= ?") + # Use consistent timestamp format with stored logs (Z suffix for UTC) + params.append(since.isoformat().replace("+00:00", "Z")) + + where_clause = " AND ".join(conditions) if conditions else "1=1" + cursor.execute(f"SELECT COUNT(*) FROM logs WHERE {where_clause}", params) + count = cursor.fetchone()[0] + conn.close() + return count + + def get_timeline( + self, + since: Optional[datetime] = None, + until: Optional[datetime] = None, + bucket_minutes: int = 5, + ) -> list[dict]: + """ + Get activity timeline bucketed by time intervals. + + Returns list of buckets with counts per agent. + """ + conn = self._connect() + cursor = conn.cursor() + + # Default to last 24 hours + if not since: + since = datetime.now(timezone.utc) - timedelta(hours=24) + if not until: + until = datetime.now(timezone.utc) + + cursor.execute( + """ + SELECT + strftime('%Y-%m-%d %H:', timestamp) || + printf('%02d', (CAST(strftime('%M', timestamp) AS INTEGER) / ?) * ?) || ':00' as bucket, + agent_id, + COUNT(*) as count, + SUM(CASE WHEN level = 'error' THEN 1 ELSE 0 END) as errors + FROM logs + WHERE timestamp >= ? AND timestamp <= ? + GROUP BY bucket, agent_id + ORDER BY bucket + """, + (bucket_minutes, bucket_minutes, since.isoformat().replace("+00:00", "Z"), until.isoformat().replace("+00:00", "Z")), + ) + + rows = cursor.fetchall() + conn.close() + + # Group by bucket + buckets = {} + for row in rows: + bucket = row["bucket"] + if bucket not in buckets: + buckets[bucket] = {"timestamp": bucket, "agents": {}, "total": 0, "errors": 0} + agent = row["agent_id"] or "main" + buckets[bucket]["agents"][agent] = row["count"] + buckets[bucket]["total"] += row["count"] + buckets[bucket]["errors"] += row["errors"] + + return list(buckets.values()) + + def get_agent_stats(self, since: Optional[datetime] = None) -> list[dict]: + """Get log statistics per agent.""" + conn = self._connect() + cursor = conn.cursor() + + params = [] + where_clause = "1=1" + if since: + where_clause = "timestamp >= ?" + # Use consistent timestamp format with stored logs (Z suffix for UTC) + params.append(since.isoformat().replace("+00:00", "Z")) + + cursor.execute( + f""" + SELECT + agent_id, + COUNT(*) as total, + SUM(CASE WHEN level = 'info' THEN 1 ELSE 0 END) as info_count, + SUM(CASE WHEN level = 'warn' OR level = 'warning' THEN 1 ELSE 0 END) as warn_count, + SUM(CASE WHEN level = 'error' THEN 1 ELSE 0 END) as error_count, + MIN(timestamp) as first_log, + MAX(timestamp) as last_log + FROM logs + WHERE {where_clause} + GROUP BY agent_id + ORDER BY total DESC + """, + params, + ) + + rows = cursor.fetchall() + conn.close() + return [dict(row) for row in rows] + + def _iter_logs( + self, + batch_size: int = 1000, + **filters, + ): + """ + Iterate over logs in batches using cursor-based pagination. + + This avoids loading all logs into memory at once. + + Args: + batch_size: Number of rows to fetch per batch + **filters: Query filters passed to query() + + Yields: + Log entries as dicts + """ + offset = 0 + while True: + batch = self.query(limit=batch_size, offset=offset, **filters) + if not batch: + break + yield from batch + offset += len(batch) + # If we got fewer than batch_size, we've reached the end + if len(batch) < batch_size: + break + + def export_logs( + self, + output_path: Path, + format: Literal["json", "jsonl", "csv"] = "jsonl", + batch_size: int = 1000, + **filters, + ) -> int: + """ + Export logs to file using cursor-based streaming. + + Args: + output_path: Output file path + format: Export format (json, jsonl, csv) + batch_size: Number of rows to fetch per batch (default 1000) + **filters: Query filters + + Returns: + Number of exported entries + """ + import csv + + output_path = Path(output_path) + output_path.parent.mkdir(parents=True, exist_ok=True) + + count = 0 + + if format == "json": + # For JSON format, we still need to collect all to produce valid JSON + # but we stream to avoid massive single query + with open(output_path, "w") as f: + f.write("[\n") + first = True + for log in self._iter_logs(batch_size=batch_size, **filters): + if not first: + f.write(",\n") + f.write(" " + json.dumps(log)) + first = False + count += 1 + f.write("\n]") + + elif format == "jsonl": + with open(output_path, "w") as f: + for log in self._iter_logs(batch_size=batch_size, **filters): + f.write(json.dumps(log) + "\n") + count += 1 + + elif format == "csv": + fieldnames = None + with open(output_path, "w", newline="") as f: + writer = None + for log in self._iter_logs(batch_size=batch_size, **filters): + if writer is None: + fieldnames = list(log.keys()) + writer = csv.DictWriter(f, fieldnames=fieldnames) + writer.writeheader() + writer.writerow(log) + count += 1 + + return count + + +def get_logger( + project_dir: Path, + agent_id: Optional[str] = None, + console_output: bool = True, +) -> StructuredLogger: + """ + Get or create a structured logger for a project. + + Args: + project_dir: Project directory + agent_id: Agent identifier (e.g., "coding-1", "initializer") + console_output: Whether to also log to console + + Returns: + StructuredLogger instance + """ + return StructuredLogger(project_dir, agent_id, console_output) + + +def get_log_query(project_dir: Path) -> LogQuery: + """ + Get log query interface for a project. + + Args: + project_dir: Project directory + + Returns: + LogQuery instance + """ + db_path = Path(project_dir) / ".autocoder" / "logs.db" + return LogQuery(db_path) diff --git a/templates/__init__.py b/templates/__init__.py new file mode 100644 index 00000000..90593418 --- /dev/null +++ b/templates/__init__.py @@ -0,0 +1,39 @@ +""" +Template Library +================ + +Pre-made templates for common application types. + +Templates provide starting points with: +- Tech stack configuration +- Pre-defined features and categories +- Design tokens +- Estimated feature count + +Available templates: +- saas-starter: Multi-tenant SaaS with auth and billing +- ecommerce: Online store with products, cart, checkout +- admin-dashboard: Admin panel with CRUD operations +- blog-cms: Blog/CMS with posts, categories, comments +- api-service: RESTful API service +""" + +from .library import ( + Template, + TemplateCategory, + generate_app_spec, + generate_features, + get_template, + list_templates, + load_template, +) + +__all__ = [ + "Template", + "TemplateCategory", + "get_template", + "list_templates", + "load_template", + "generate_app_spec", + "generate_features", +] diff --git a/templates/catalog/admin-dashboard.yaml b/templates/catalog/admin-dashboard.yaml new file mode 100644 index 00000000..1380a4a2 --- /dev/null +++ b/templates/catalog/admin-dashboard.yaml @@ -0,0 +1,83 @@ +name: "Admin Dashboard" +description: "Full-featured admin panel with CRUD operations, charts, and data tables" + +tech_stack: + frontend: "React" + backend: "FastAPI" + database: "PostgreSQL" + auth: "JWT" + styling: "Tailwind CSS" + +feature_categories: + authentication: + - "Admin login" + - "Password reset" + - "Role-based access control" + - "Session management" + + dashboard: + - "Overview page" + - "Statistics cards" + - "Charts (line, bar, pie)" + - "Recent activity" + - "Quick actions" + + user_management: + - "User list with pagination" + - "User search and filter" + - "Create new user" + - "Edit user" + - "Delete user" + - "User roles management" + - "User activity log" + + content_management: + - "Content list" + - "Create content" + - "Edit content" + - "Delete content" + - "Publish/unpublish" + - "Content categories" + + data_tables: + - "Sortable columns" + - "Filterable columns" + - "Pagination" + - "Bulk actions" + - "Export to CSV" + - "Column visibility toggle" + + settings: + - "General settings" + - "Email templates" + - "Notification settings" + - "Backup management" + - "System logs" + + notifications: + - "In-app notifications" + - "Notification center" + - "Mark as read" + - "Notification preferences" + +design_tokens: + colors: + primary: "#3B82F6" + secondary: "#8B5CF6" + accent: "#F59E0B" + background: "#F3F4F6" + sidebar: "#1F2937" + text: "#111827" + muted: "#6B7280" + spacing: [4, 8, 12, 16, 24, 32] + fonts: + heading: "Inter" + body: "Inter" + border_radius: + small: "4px" + medium: "6px" + large: "8px" + +estimated_features: 40 +tags: ["admin", "dashboard", "crud", "management"] +difficulty: "intermediate" diff --git a/templates/catalog/api-service.yaml b/templates/catalog/api-service.yaml new file mode 100644 index 00000000..9815245e --- /dev/null +++ b/templates/catalog/api-service.yaml @@ -0,0 +1,80 @@ +name: "API Service" +description: "RESTful API service with authentication, rate limiting, and documentation" + +tech_stack: + backend: "FastAPI" + database: "PostgreSQL" + auth: "JWT" + hosting: "Docker" + +feature_categories: + core_api: + - "Health check endpoint" + - "Version endpoint" + - "OpenAPI documentation" + - "Swagger UI" + - "ReDoc documentation" + + authentication: + - "User registration" + - "User login" + - "Token refresh" + - "Password reset" + - "API key authentication" + - "OAuth2 support" + + user_management: + - "Get current user" + - "Update user profile" + - "Change password" + - "Delete account" + - "List users (admin)" + + resource_crud: + - "Create resource" + - "Read resource" + - "Update resource" + - "Delete resource" + - "List resources" + - "Search resources" + - "Filter resources" + - "Paginate results" + + security: + - "Rate limiting" + - "Request validation" + - "Input sanitization" + - "CORS configuration" + - "Security headers" + + monitoring: + - "Request logging" + - "Error tracking" + - "Performance metrics" + - "Health checks" + + admin: + - "Admin endpoints" + - "User management" + - "System statistics" + - "Audit logs" + +design_tokens: + colors: + primary: "#059669" + secondary: "#0EA5E9" + accent: "#F59E0B" + background: "#F9FAFB" + text: "#111827" + spacing: [4, 8, 12, 16, 24, 32] + fonts: + heading: "Inter" + body: "Inter" + border_radius: + small: "4px" + medium: "6px" + large: "8px" + +estimated_features: 30 +tags: ["api", "rest", "backend", "microservice"] +difficulty: "intermediate" diff --git a/templates/catalog/blog-cms.yaml b/templates/catalog/blog-cms.yaml new file mode 100644 index 00000000..a95fb6e0 --- /dev/null +++ b/templates/catalog/blog-cms.yaml @@ -0,0 +1,80 @@ +name: "Blog & CMS" +description: "Content management system with blog posts, categories, and comments" + +tech_stack: + frontend: "Next.js" + backend: "Node.js/Express" + database: "PostgreSQL" + auth: "NextAuth.js" + styling: "Tailwind CSS" + +feature_categories: + public_pages: + - "Home page with featured posts" + - "Blog listing page" + - "Blog post detail page" + - "Category pages" + - "Tag pages" + - "Author pages" + - "Search results page" + - "About page" + - "Contact page" + + blog_features: + - "Post search" + - "Category filtering" + - "Tag filtering" + - "Related posts" + - "Social sharing" + - "Reading time estimate" + - "Table of contents" + + comments: + - "Comment submission" + - "Comment moderation" + - "Reply to comments" + - "Like comments" + - "Comment notifications" + + admin_content: + - "Post editor (rich text)" + - "Post preview" + - "Draft management" + - "Schedule posts" + - "Post categories" + - "Post tags" + - "Media library" + + admin_settings: + - "Site settings" + - "SEO settings" + - "Social media links" + - "Analytics integration" + + user_features: + - "Author registration" + - "Author login" + - "Author profile" + - "Author dashboard" + - "Newsletter subscription" + +design_tokens: + colors: + primary: "#0F172A" + secondary: "#3B82F6" + accent: "#F97316" + background: "#FFFFFF" + text: "#334155" + muted: "#94A3B8" + spacing: [4, 8, 12, 16, 24, 32, 48, 64] + fonts: + heading: "Merriweather" + body: "Source Sans Pro" + border_radius: + small: "2px" + medium: "4px" + large: "8px" + +estimated_features: 35 +tags: ["blog", "cms", "content", "publishing"] +difficulty: "intermediate" diff --git a/templates/catalog/ecommerce.yaml b/templates/catalog/ecommerce.yaml new file mode 100644 index 00000000..dcbcf146 --- /dev/null +++ b/templates/catalog/ecommerce.yaml @@ -0,0 +1,83 @@ +name: "E-Commerce Store" +description: "Full-featured online store with products, cart, checkout, and order management" + +tech_stack: + frontend: "Next.js" + backend: "Node.js/Express" + database: "PostgreSQL" + auth: "NextAuth.js" + styling: "Tailwind CSS" + hosting: "Vercel" + +feature_categories: + product_catalog: + - "Product listing page" + - "Product detail page" + - "Product search" + - "Category navigation" + - "Product filtering" + - "Product sorting" + - "Product image gallery" + - "Related products" + + shopping_cart: + - "Add to cart" + - "Update cart quantity" + - "Remove from cart" + - "Cart sidebar/drawer" + - "Cart page" + - "Save for later" + + checkout: + - "Guest checkout" + - "User checkout" + - "Shipping address form" + - "Shipping method selection" + - "Payment integration (Stripe)" + - "Order summary" + - "Order confirmation" + + user_account: + - "User registration" + - "User login" + - "Password reset" + - "Profile management" + - "Address book" + - "Order history" + - "Wishlist" + + admin_panel: + - "Product management" + - "Category management" + - "Order management" + - "Customer management" + - "Inventory tracking" + - "Sales reports" + - "Discount codes" + + marketing: + - "Newsletter signup" + - "Promotional banners" + - "Product reviews" + - "Rating system" + +design_tokens: + colors: + primary: "#2563EB" + secondary: "#16A34A" + accent: "#DC2626" + background: "#FFFFFF" + text: "#1F2937" + muted: "#9CA3AF" + spacing: [4, 8, 12, 16, 24, 32, 48] + fonts: + heading: "Poppins" + body: "Open Sans" + border_radius: + small: "4px" + medium: "8px" + large: "16px" + +estimated_features: 50 +tags: ["ecommerce", "store", "shopping", "payments"] +difficulty: "advanced" diff --git a/templates/catalog/saas-starter.yaml b/templates/catalog/saas-starter.yaml new file mode 100644 index 00000000..98b4f947 --- /dev/null +++ b/templates/catalog/saas-starter.yaml @@ -0,0 +1,74 @@ +name: "SaaS Starter" +description: "Multi-tenant SaaS application with authentication, billing, and dashboard" + +tech_stack: + frontend: "Next.js" + backend: "Node.js/Express" + database: "PostgreSQL" + auth: "NextAuth.js" + styling: "Tailwind CSS" + hosting: "Vercel" + +feature_categories: + authentication: + - "User registration" + - "User login" + - "Password reset" + - "Email verification" + - "OAuth login (Google, GitHub)" + - "Two-factor authentication" + - "Session management" + + multi_tenancy: + - "Organization creation" + - "Team member invitations" + - "Role management (Admin, Member)" + - "Organization settings" + - "Switch between organizations" + + billing: + - "Subscription plans display" + - "Stripe integration" + - "Payment method management" + - "Invoice history" + - "Usage tracking" + - "Plan upgrades/downgrades" + + dashboard: + - "Overview page with metrics" + - "Usage statistics charts" + - "Recent activity feed" + - "Quick actions" + + user_profile: + - "Profile settings" + - "Avatar upload" + - "Notification preferences" + - "API key management" + + admin: + - "User management" + - "Organization management" + - "System health dashboard" + - "Audit logs" + +design_tokens: + colors: + primary: "#6366F1" + secondary: "#10B981" + accent: "#F59E0B" + background: "#F9FAFB" + text: "#111827" + muted: "#6B7280" + spacing: [4, 8, 12, 16, 24, 32, 48] + fonts: + heading: "Inter" + body: "Inter" + border_radius: + small: "4px" + medium: "8px" + large: "12px" + +estimated_features: 45 +tags: ["saas", "subscription", "multi-tenant", "billing"] +difficulty: "advanced" diff --git a/templates/library.py b/templates/library.py new file mode 100644 index 00000000..5be35c39 --- /dev/null +++ b/templates/library.py @@ -0,0 +1,319 @@ +""" +Template Library Module +======================= + +Load and manage application templates for quick project scaffolding. +""" + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Optional +from xml.sax.saxutils import escape as xml_escape + +import yaml + +# Directory containing template files +TEMPLATES_DIR = Path(__file__).parent / "catalog" + + +@dataclass +class DesignTokens: + """Design tokens for consistent styling.""" + + colors: dict[str, str] = field(default_factory=dict) + spacing: list[int] = field(default_factory=list) + fonts: dict[str, str] = field(default_factory=dict) + border_radius: dict[str, str] = field(default_factory=dict) + + @classmethod + def from_dict(cls, data: dict) -> "DesignTokens": + """Create from dictionary.""" + return cls( + colors=data.get("colors", {}), + spacing=data.get("spacing", [4, 8, 12, 16, 24, 32]), + fonts=data.get("fonts", {}), + border_radius=data.get("border_radius", {}), + ) + + +@dataclass +class TechStack: + """Technology stack configuration.""" + + frontend: Optional[str] = None + backend: Optional[str] = None + database: Optional[str] = None + auth: Optional[str] = None + styling: Optional[str] = None + hosting: Optional[str] = None + + @classmethod + def from_dict(cls, data: dict) -> "TechStack": + """Create from dictionary.""" + return cls( + frontend=data.get("frontend"), + backend=data.get("backend"), + database=data.get("database"), + auth=data.get("auth"), + styling=data.get("styling"), + hosting=data.get("hosting"), + ) + + +@dataclass +class TemplateFeature: + """A feature in a template.""" + + name: str + description: str + category: str + steps: list[str] = field(default_factory=list) + priority: int = 0 + + @classmethod + def from_dict(cls, data: dict, category: str, priority: int) -> "TemplateFeature": + """Create from dictionary.""" + steps = data.get("steps", []) + if not steps: + # Generate default steps + steps = [f"Implement {data['name']}"] + + return cls( + name=data["name"], + description=data.get("description", data["name"]), + category=category, + steps=steps, + priority=priority, + ) + + +@dataclass +class TemplateCategory: + """A category of features in a template.""" + + name: str + features: list[str] + description: Optional[str] = None + + +@dataclass +class Template: + """An application template.""" + + id: str + name: str + description: str + tech_stack: TechStack + feature_categories: dict[str, list[str]] + design_tokens: DesignTokens + estimated_features: int + tags: list[str] = field(default_factory=list) + difficulty: str = "intermediate" + preview_image: Optional[str] = None + + @classmethod + def from_dict(cls, template_id: str, data: dict) -> "Template": + """Create from dictionary.""" + return cls( + id=template_id, + name=data["name"], + description=data["description"], + tech_stack=TechStack.from_dict(data.get("tech_stack", {})), + feature_categories=data.get("feature_categories", {}), + design_tokens=DesignTokens.from_dict(data.get("design_tokens", {})), + estimated_features=data.get("estimated_features", 0), + tags=data.get("tags", []), + difficulty=data.get("difficulty", "intermediate"), + preview_image=data.get("preview_image"), + ) + + def to_dict(self) -> dict: + """Convert to dictionary.""" + return { + "id": self.id, + "name": self.name, + "description": self.description, + "tech_stack": { + "frontend": self.tech_stack.frontend, + "backend": self.tech_stack.backend, + "database": self.tech_stack.database, + "auth": self.tech_stack.auth, + "styling": self.tech_stack.styling, + "hosting": self.tech_stack.hosting, + }, + "feature_categories": self.feature_categories, + "design_tokens": { + "colors": self.design_tokens.colors, + "spacing": self.design_tokens.spacing, + "fonts": self.design_tokens.fonts, + "border_radius": self.design_tokens.border_radius, + }, + "estimated_features": self.estimated_features, + "tags": self.tags, + "difficulty": self.difficulty, + } + + +def load_template(template_id: str) -> Optional[Template]: + """ + Load a template by ID. + + Args: + template_id: Template identifier (filename without extension) + + Returns: + Template instance or None if not found + """ + template_path = TEMPLATES_DIR / f"{template_id}.yaml" + + if not template_path.exists(): + return None + + try: + with open(template_path) as f: + data = yaml.safe_load(f) + return Template.from_dict(template_id, data) + except Exception: + return None + + +def list_templates() -> list[Template]: + """ + List all available templates. + + Returns: + List of Template instances + """ + templates = [] + + if not TEMPLATES_DIR.exists(): + return templates + + for file in TEMPLATES_DIR.glob("*.yaml"): + template = load_template(file.stem) + if template: + templates.append(template) + + return sorted(templates, key=lambda t: t.name) + + +def get_template(template_id: str) -> Optional[Template]: + """ + Get a specific template by ID. + + Args: + template_id: Template identifier + + Returns: + Template instance or None + """ + return load_template(template_id) + + +def generate_features(template: Template) -> list[dict]: + """ + Generate feature list from a template. + + Returns features in the format expected by feature_create_bulk. + + Args: + template: Template instance + + Returns: + List of feature dictionaries + """ + features = [] + priority = 1 + + for category, feature_names in template.feature_categories.items(): + for feature_name in feature_names: + features.append({ + "priority": priority, + "category": category.replace("_", " ").title(), + "name": feature_name, + "description": f"{feature_name} functionality for the application", + "steps": [f"Implement {feature_name}"], + "passes": False, + }) + priority += 1 + + return features + + +def generate_app_spec( + template: Template, + app_name: str, + customizations: Optional[dict] = None, +) -> str: + """ + Generate app_spec.txt content from a template. + + Args: + template: Template instance + app_name: Application name + customizations: Optional customizations to apply + + Returns: + XML content for app_spec.txt + """ + customizations = customizations or {} + + # Merge design tokens with customizations + colors = {**template.design_tokens.colors, **customizations.get("colors", {})} + + # Build XML (escape all user-provided content to prevent XML injection) + xml_parts = [ + '', + "", + f" {xml_escape(app_name)}", + f" {xml_escape(template.description)}", + "", + " ", + ] + + if template.tech_stack.frontend: + xml_parts.append(f" {xml_escape(template.tech_stack.frontend)}") + if template.tech_stack.backend: + xml_parts.append(f" {xml_escape(template.tech_stack.backend)}") + if template.tech_stack.database: + xml_parts.append(f" {xml_escape(template.tech_stack.database)}") + if template.tech_stack.auth: + xml_parts.append(f" {xml_escape(template.tech_stack.auth)}") + if template.tech_stack.styling: + xml_parts.append(f" {xml_escape(template.tech_stack.styling)}") + + xml_parts.extend([ + " ", + "", + " ", + " ", + ]) + + for color_name, color_value in colors.items(): + # Escape color name (used as tag name) and value + safe_name = xml_escape(color_name) + safe_value = xml_escape(color_value) + xml_parts.append(f" <{safe_name}>{safe_value}") + + xml_parts.extend([ + " ", + " ", + "", + " ", + ]) + + for category, feature_names in template.feature_categories.items(): + category_title = category.replace("_", " ").title() + # Escape attribute value using quoteattr pattern + safe_category = xml_escape(category_title, {'"': '"'}) + xml_parts.append(f' ') + for feature_name in feature_names: + xml_parts.append(f" {xml_escape(feature_name)}") + xml_parts.append(" ") + + xml_parts.extend([ + " ", + "", + ]) + + return "\n".join(xml_parts) diff --git a/test_agent.py b/test_agent.py new file mode 100644 index 00000000..f672ecb2 --- /dev/null +++ b/test_agent.py @@ -0,0 +1,111 @@ +""" +Unit tests for rate limit handling functions. + +Tests the parse_retry_after() and is_rate_limit_error() functions +from rate_limit_utils.py (shared module). +""" + +import unittest + +from rate_limit_utils import ( + is_rate_limit_error, + parse_retry_after, +) + + +class TestParseRetryAfter(unittest.TestCase): + """Tests for parse_retry_after() function.""" + + def test_retry_after_colon_format(self): + """Test 'Retry-After: 60' format.""" + assert parse_retry_after("Retry-After: 60") == 60 + assert parse_retry_after("retry-after: 120") == 120 + assert parse_retry_after("retry after: 30 seconds") == 30 + + def test_retry_after_space_format(self): + """Test 'retry after 60 seconds' format.""" + assert parse_retry_after("retry after 60 seconds") == 60 + assert parse_retry_after("Please retry after 120 seconds") == 120 + assert parse_retry_after("Retry after 30") == 30 + + def test_try_again_in_format(self): + """Test 'try again in X seconds' format.""" + assert parse_retry_after("try again in 120 seconds") == 120 + assert parse_retry_after("Please try again in 60s") == 60 + assert parse_retry_after("Try again in 30 seconds") == 30 + + def test_seconds_remaining_format(self): + """Test 'X seconds remaining' format.""" + assert parse_retry_after("30 seconds remaining") == 30 + assert parse_retry_after("60 seconds left") == 60 + assert parse_retry_after("120 seconds until reset") == 120 + + def test_no_match(self): + """Test messages that don't contain retry-after info.""" + assert parse_retry_after("no match here") is None + assert parse_retry_after("Connection refused") is None + assert parse_retry_after("Internal server error") is None + assert parse_retry_after("") is None + + def test_minutes_not_supported(self): + """Test that minutes are not parsed (by design).""" + # We only support seconds to avoid complexity + assert parse_retry_after("wait 5 minutes") is None + assert parse_retry_after("try again in 2 minutes") is None + + +class TestIsRateLimitError(unittest.TestCase): + """Tests for is_rate_limit_error() function.""" + + def test_rate_limit_patterns(self): + """Test various rate limit error messages.""" + assert is_rate_limit_error("Rate limit exceeded") is True + assert is_rate_limit_error("rate_limit_exceeded") is True + assert is_rate_limit_error("Too many requests") is True + assert is_rate_limit_error("HTTP 429 Too Many Requests") is True + assert is_rate_limit_error("API quota exceeded") is True + assert is_rate_limit_error("Please wait before retrying") is True + assert is_rate_limit_error("Try again later") is True + assert is_rate_limit_error("Server is overloaded") is True + assert is_rate_limit_error("Usage limit reached") is True + + def test_case_insensitive(self): + """Test that detection is case-insensitive.""" + assert is_rate_limit_error("RATE LIMIT") is True + assert is_rate_limit_error("Rate Limit") is True + assert is_rate_limit_error("rate limit") is True + assert is_rate_limit_error("RaTe LiMiT") is True + + def test_non_rate_limit_errors(self): + """Test non-rate-limit error messages.""" + assert is_rate_limit_error("Connection refused") is False + assert is_rate_limit_error("Authentication failed") is False + assert is_rate_limit_error("Invalid API key") is False + assert is_rate_limit_error("Internal server error") is False + assert is_rate_limit_error("Network timeout") is False + assert is_rate_limit_error("") is False + + +class TestExponentialBackoff(unittest.TestCase): + """Test exponential backoff calculations.""" + + def test_backoff_sequence(self): + """Test that backoff follows expected sequence.""" + # Simulating: min(60 * (2 ** retries), 3600) + expected = [60, 120, 240, 480, 960, 1920, 3600, 3600] # Caps at 3600 + for retries, expected_delay in enumerate(expected): + delay = min(60 * (2 ** retries), 3600) + assert delay == expected_delay, f"Retry {retries}: expected {expected_delay}, got {delay}" + + def test_error_backoff_sequence(self): + """Test error backoff follows expected sequence.""" + # Simulating: min(30 * retries, 300) + expected = [30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 300] # Caps at 300 + for retries in range(1, len(expected) + 1): + delay = min(30 * retries, 300) + expected_delay = expected[retries - 1] + assert delay == expected_delay, f"Retry {retries}: expected {expected_delay}, got {delay}" + + +if __name__ == "__main__": + unittest.main() diff --git a/test_security.py b/test_security.py index 1bd48d95..cc201698 100644 --- a/test_security.py +++ b/test_security.py @@ -453,6 +453,21 @@ def test_project_commands(): print(" FAIL: Non-allowed command 'rustc' should be blocked") failed += 1 + # Test 4: Empty command name is rejected + config_path.write_text("""version: 1 +commands: + - name: "" + description: Empty name should be rejected +""") + result = load_project_commands(project_dir) + if result is None: + print(" PASS: Empty command name rejected in project config") + passed += 1 + else: + print(" FAIL: Empty command name should be rejected in project config") + print(f" Got: {result}") + failed += 1 + return passed, failed diff --git a/test_structured_logging.py b/test_structured_logging.py new file mode 100644 index 00000000..a1ebb303 --- /dev/null +++ b/test_structured_logging.py @@ -0,0 +1,470 @@ +""" +Unit Tests for Structured Logging Module +========================================= + +Tests for the structured logging system that saves logs to SQLite. +""" + +import json +import sqlite3 +import tempfile +import threading +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from unittest import TestCase + +from structured_logging import ( + StructuredLogEntry, + StructuredLogHandler, + get_log_query, + get_logger, +) + + +class TestStructuredLogEntry(TestCase): + """Tests for StructuredLogEntry dataclass.""" + + def test_to_dict_minimal(self): + """Test minimal entry conversion.""" + entry = StructuredLogEntry( + timestamp="2025-01-21T10:30:00.000Z", + level="info", + message="Test message", + ) + result = entry.to_dict() + self.assertEqual(result["timestamp"], "2025-01-21T10:30:00.000Z") + self.assertEqual(result["level"], "info") + self.assertEqual(result["message"], "Test message") + # Optional fields should not be present when None + self.assertNotIn("agent_id", result) + self.assertNotIn("feature_id", result) + self.assertNotIn("tool_name", result) + + def test_to_dict_full(self): + """Test full entry with all fields.""" + entry = StructuredLogEntry( + timestamp="2025-01-21T10:30:00.000Z", + level="error", + message="Test error", + agent_id="coding-42", + feature_id=42, + tool_name="playwright", + duration_ms=150, + extra={"key": "value"}, + ) + result = entry.to_dict() + self.assertEqual(result["agent_id"], "coding-42") + self.assertEqual(result["feature_id"], 42) + self.assertEqual(result["tool_name"], "playwright") + self.assertEqual(result["duration_ms"], 150) + self.assertEqual(result["extra"], {"key": "value"}) + + def test_to_json(self): + """Test JSON serialization.""" + entry = StructuredLogEntry( + timestamp="2025-01-21T10:30:00.000Z", + level="info", + message="Test", + ) + json_str = entry.to_json() + parsed = json.loads(json_str) + self.assertEqual(parsed["message"], "Test") + + +class TestStructuredLogHandler(TestCase): + """Tests for StructuredLogHandler.""" + + def setUp(self): + """Create temporary directory for tests.""" + self.temp_dir = tempfile.mkdtemp() + self.db_path = Path(self.temp_dir) / "logs.db" + + def tearDown(self): + """Clean up temporary files.""" + import shutil + shutil.rmtree(self.temp_dir, ignore_errors=True) + + def test_creates_database(self): + """Test that handler creates database file.""" + _handler = StructuredLogHandler(self.db_path) # noqa: F841 - handler triggers DB creation + self.assertTrue(self.db_path.exists()) + + def test_creates_tables(self): + """Test that handler creates logs table.""" + _handler = StructuredLogHandler(self.db_path) # noqa: F841 - handler triggers table creation + conn = sqlite3.connect(self.db_path) + cursor = conn.cursor() + cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='logs'") + result = cursor.fetchone() + conn.close() + self.assertIsNotNone(result) + + def test_wal_mode_enabled(self): + """Test that WAL mode is enabled for concurrency.""" + _handler = StructuredLogHandler(self.db_path) # noqa: F841 - handler triggers WAL mode + conn = sqlite3.connect(self.db_path) + cursor = conn.cursor() + cursor.execute("PRAGMA journal_mode") + result = cursor.fetchone()[0] + conn.close() + self.assertEqual(result.lower(), "wal") + + +class TestStructuredLogger(TestCase): + """Tests for StructuredLogger.""" + + def setUp(self): + """Create temporary project directory.""" + self.temp_dir = tempfile.mkdtemp() + self.project_dir = Path(self.temp_dir) + + def tearDown(self): + """Clean up temporary files.""" + import shutil + shutil.rmtree(self.temp_dir, ignore_errors=True) + + def test_creates_logs_directory(self): + """Test that logger creates .autocoder directory.""" + _logger = get_logger(self.project_dir, agent_id="test", console_output=False) # noqa: F841 + autocoder_dir = self.project_dir / ".autocoder" + self.assertTrue(autocoder_dir.exists()) + + def test_creates_logs_db(self): + """Test that logger creates logs.db file.""" + _logger = get_logger(self.project_dir, agent_id="test", console_output=False) # noqa: F841 + db_path = self.project_dir / ".autocoder" / "logs.db" + self.assertTrue(db_path.exists()) + + def test_log_info(self): + """Test info level logging.""" + logger = get_logger(self.project_dir, agent_id="test-agent", console_output=False) + logger.info("Test info message", feature_id=42) + + # Query the database + query = get_log_query(self.project_dir) + logs = query.query(level="info") + self.assertEqual(len(logs), 1) + self.assertEqual(logs[0]["message"], "Test info message") + self.assertEqual(logs[0]["agent_id"], "test-agent") + self.assertEqual(logs[0]["feature_id"], 42) + + def test_log_warn(self): + """Test warning level logging.""" + logger = get_logger(self.project_dir, agent_id="test", console_output=False) + logger.warn("Test warning") + + query = get_log_query(self.project_dir) + logs = query.query(level="warning") + self.assertEqual(len(logs), 1) + # Assert on level field, not message content (more robust) + self.assertEqual(logs[0]["level"], "warning") + + def test_log_error(self): + """Test error level logging.""" + logger = get_logger(self.project_dir, agent_id="test", console_output=False) + logger.error("Test error", tool_name="playwright") + + query = get_log_query(self.project_dir) + logs = query.query(level="error") + self.assertEqual(len(logs), 1) + self.assertEqual(logs[0]["tool_name"], "playwright") + + def test_log_debug(self): + """Test debug level logging.""" + logger = get_logger(self.project_dir, agent_id="test", console_output=False) + logger.debug("Debug message") + + query = get_log_query(self.project_dir) + logs = query.query(level="debug") + self.assertEqual(len(logs), 1) + + def test_extra_fields(self): + """Test that extra fields are stored as JSON.""" + logger = get_logger(self.project_dir, agent_id="test", console_output=False) + logger.info("Test", custom_field="value", count=42) + + query = get_log_query(self.project_dir) + logs = query.query() + self.assertEqual(len(logs), 1) + extra = json.loads(logs[0]["extra"]) if logs[0]["extra"] else {} + self.assertEqual(extra.get("custom_field"), "value") + self.assertEqual(extra.get("count"), 42) + + +class TestLogQuery(TestCase): + """Tests for LogQuery.""" + + def setUp(self): + """Create temporary project directory with sample logs.""" + self.temp_dir = tempfile.mkdtemp() + self.project_dir = Path(self.temp_dir) + + # Create sample logs + logger = get_logger(self.project_dir, agent_id="coding-1", console_output=False) + logger.info("Feature started", feature_id=1) + logger.debug("Tool used", feature_id=1, tool_name="bash") + logger.error("Test failed", feature_id=1, tool_name="playwright") + + logger2 = get_logger(self.project_dir, agent_id="coding-2", console_output=False) + logger2.info("Feature started", feature_id=2) + logger2.info("Feature completed", feature_id=2) + + def tearDown(self): + """Clean up temporary files.""" + import shutil + shutil.rmtree(self.temp_dir, ignore_errors=True) + + def test_query_by_level(self): + """Test filtering by log level.""" + query = get_log_query(self.project_dir) + errors = query.query(level="error") + self.assertEqual(len(errors), 1) + self.assertEqual(errors[0]["level"], "error") + + def test_query_by_agent_id(self): + """Test filtering by agent ID.""" + query = get_log_query(self.project_dir) + logs = query.query(agent_id="coding-2") + self.assertEqual(len(logs), 2) + for log in logs: + self.assertEqual(log["agent_id"], "coding-2") + + def test_query_by_feature_id(self): + """Test filtering by feature ID.""" + query = get_log_query(self.project_dir) + logs = query.query(feature_id=1) + self.assertEqual(len(logs), 3) + for log in logs: + self.assertEqual(log["feature_id"], 1) + + def test_query_by_tool_name(self): + """Test filtering by tool name.""" + query = get_log_query(self.project_dir) + logs = query.query(tool_name="playwright") + self.assertEqual(len(logs), 1) + self.assertEqual(logs[0]["tool_name"], "playwright") + + def test_query_full_text_search(self): + """Test full-text search in messages.""" + query = get_log_query(self.project_dir) + logs = query.query(search="Feature started") + self.assertEqual(len(logs), 2) + + def test_query_with_limit(self): + """Test query with limit.""" + query = get_log_query(self.project_dir) + logs = query.query(limit=2) + self.assertEqual(len(logs), 2) + + def test_query_with_offset(self): + """Test query with offset for pagination.""" + query = get_log_query(self.project_dir) + all_logs = query.query() + offset_logs = query.query(offset=2, limit=10) + self.assertEqual(len(offset_logs), len(all_logs) - 2) + + def test_count(self): + """Test count method.""" + query = get_log_query(self.project_dir) + total = query.count() + self.assertEqual(total, 5) + + error_count = query.count(level="error") + self.assertEqual(error_count, 1) + + def test_get_agent_stats(self): + """Test agent statistics.""" + query = get_log_query(self.project_dir) + stats = query.get_agent_stats() + self.assertEqual(len(stats), 2) # coding-1 and coding-2 + + # Find coding-1 stats + coding1_stats = next((s for s in stats if s["agent_id"] == "coding-1"), None) + self.assertIsNotNone(coding1_stats) + self.assertEqual(coding1_stats["error_count"], 1) + + +class TestLogExport(TestCase): + """Tests for log export functionality.""" + + def setUp(self): + """Create temporary project directory with sample logs.""" + self.temp_dir = tempfile.mkdtemp() + self.project_dir = Path(self.temp_dir) + self.export_dir = Path(self.temp_dir) / "exports" + self.export_dir.mkdir() + + logger = get_logger(self.project_dir, agent_id="test", console_output=False) + logger.info("Test log 1") + logger.info("Test log 2") + logger.error("Test error") + + def tearDown(self): + """Clean up temporary files.""" + import shutil + shutil.rmtree(self.temp_dir, ignore_errors=True) + + def test_export_json(self): + """Test JSON export.""" + query = get_log_query(self.project_dir) + output_path = self.export_dir / "logs.json" + count = query.export_logs(output_path, format="json") + + self.assertEqual(count, 3) + self.assertTrue(output_path.exists()) + + with open(output_path) as f: + data = json.load(f) + self.assertEqual(len(data), 3) + + def test_export_jsonl(self): + """Test JSONL export.""" + query = get_log_query(self.project_dir) + output_path = self.export_dir / "logs.jsonl" + count = query.export_logs(output_path, format="jsonl") + + self.assertEqual(count, 3) + self.assertTrue(output_path.exists()) + + with open(output_path) as f: + lines = f.readlines() + self.assertEqual(len(lines), 3) + # Verify each line is valid JSON + for line in lines: + json.loads(line) + + def test_export_csv(self): + """Test CSV export.""" + query = get_log_query(self.project_dir) + output_path = self.export_dir / "logs.csv" + count = query.export_logs(output_path, format="csv") + + self.assertEqual(count, 3) + self.assertTrue(output_path.exists()) + + import csv + with open(output_path) as f: + reader = csv.DictReader(f) + rows = list(reader) + self.assertEqual(len(rows), 3) + + +class TestThreadSafety(TestCase): + """Tests for thread safety of the logging system.""" + + def setUp(self): + """Create temporary project directory.""" + self.temp_dir = tempfile.mkdtemp() + self.project_dir = Path(self.temp_dir) + + def tearDown(self): + """Clean up temporary files.""" + import shutil + shutil.rmtree(self.temp_dir, ignore_errors=True) + + def test_concurrent_writes(self): + """Test that concurrent writes don't cause database corruption.""" + num_threads = 10 + logs_per_thread = 50 + + def write_logs(thread_id): + logger = get_logger(self.project_dir, agent_id=f"thread-{thread_id}", console_output=False) + for i in range(logs_per_thread): + logger.info(f"Log {i} from thread {thread_id}", count=i) + + with ThreadPoolExecutor(max_workers=num_threads) as executor: + futures = [executor.submit(write_logs, i) for i in range(num_threads)] + for future in futures: + future.result() # Wait for all to complete + + # Verify all logs were written + query = get_log_query(self.project_dir) + total = query.count() + expected = num_threads * logs_per_thread + self.assertEqual(total, expected) + + def test_concurrent_read_write(self): + """Test that reads and writes can happen concurrently.""" + logger = get_logger(self.project_dir, agent_id="writer", console_output=False) + query = get_log_query(self.project_dir) + + # Pre-populate some logs + for i in range(10): + logger.info(f"Initial log {i}") + + read_results = [] + write_done = threading.Event() + + def writer(): + for i in range(50): + logger.info(f"Concurrent log {i}") + write_done.set() + + def reader(): + while not write_done.is_set(): + count = query.count() + read_results.append(count) + + writer_thread = threading.Thread(target=writer) + reader_thread = threading.Thread(target=reader) + + writer_thread.start() + reader_thread.start() + + writer_thread.join() + reader_thread.join() + + # Verify no errors occurred and reads returned valid counts + self.assertTrue(len(read_results) > 0) + self.assertTrue(all(r >= 10 for r in read_results)) # At least initial logs + + # Final count should be 60 (10 initial + 50 concurrent) + final_count = query.count() + self.assertEqual(final_count, 60) + + +class TestCleanup(TestCase): + """Tests for automatic log cleanup.""" + + def setUp(self): + """Create temporary project directory.""" + self.temp_dir = tempfile.mkdtemp() + self.project_dir = Path(self.temp_dir) + + def tearDown(self): + """Clean up temporary files.""" + import shutil + shutil.rmtree(self.temp_dir, ignore_errors=True) + + def test_cleanup_old_entries(self): + """Test that old entries are cleaned up when max_entries is exceeded.""" + # Create handler with low max_entries + db_path = self.project_dir / ".autocoder" / "logs.db" + db_path.parent.mkdir(parents=True, exist_ok=True) + handler = StructuredLogHandler(db_path, max_entries=10) + + # Create a logger using this handler + import logging + logger = logging.getLogger("test_cleanup") + logger.handlers.clear() + logger.addHandler(handler) + logger.setLevel(logging.DEBUG) + + # Write more than max_entries + for i in range(20): + logger.info(f"Log message {i}") + + # Query the database + conn = sqlite3.connect(db_path) + cursor = conn.cursor() + cursor.execute("SELECT COUNT(*) FROM logs") + count = cursor.fetchone()[0] + conn.close() + + # Should have at most max_entries + self.assertLessEqual(count, 10) + + +if __name__ == "__main__": + import unittest + unittest.main() diff --git a/ui/package.json b/ui/package.json index f70b9ca2..49e33735 100644 --- a/ui/package.json +++ b/ui/package.json @@ -58,6 +58,6 @@ "tw-animate-css": "^1.4.0", "typescript": "~5.7.3", "typescript-eslint": "^8.23.0", - "vite": "^7.3.0" + "vite": "^7.3.1" } } diff --git a/ui/src/components/ImportProjectModal.tsx b/ui/src/components/ImportProjectModal.tsx new file mode 100644 index 00000000..31697824 --- /dev/null +++ b/ui/src/components/ImportProjectModal.tsx @@ -0,0 +1,637 @@ +/** + * Import Project Modal Component + * + * Multi-step wizard for importing existing projects: + * 1. Select project folder + * 2. Analyze and detect tech stack + * 3. Extract features from codebase + * 4. Review and select features to import + * 5. Create features in database + */ + +import { useState, useEffect } from 'react' +import { + X, + Folder, + Search, + Layers, + CheckCircle2, + AlertCircle, + Loader2, + ArrowRight, + ArrowLeft, + Code, + Database, + Server, + Layout, + CheckSquare, + Square, + ChevronDown, + ChevronRight, +} from 'lucide-react' +import { useImportProject } from '../hooks/useImportProject' +import { useCreateProject } from '../hooks/useProjects' +import { FolderBrowser } from './FolderBrowser' + +type Step = 'folder' | 'analyzing' | 'detected' | 'features' | 'register' | 'complete' + +interface ImportProjectModalProps { + isOpen: boolean + onClose: () => void + onProjectImported: (projectName: string) => void +} + +export function ImportProjectModal({ + isOpen, + onClose, + onProjectImported, +}: ImportProjectModalProps) { + const [step, setStep] = useState('folder') + const [projectName, setProjectName] = useState('') + const [expandedCategories, setExpandedCategories] = useState>(new Set()) + const [registerError, setRegisterError] = useState(null) + + const { + state, + analyze, + extractFeatures, + createFeatures, + toggleFeature, + selectAllFeatures, + deselectAllFeatures, + reset, + } = useImportProject() + + const createProject = useCreateProject() + + // Expand all categories when features are extracted + useEffect(() => { + if (step === 'features' && state.featuresResult) { + setExpandedCategories(new Set(Object.keys(state.featuresResult.by_category))) + } + }, [step, state.featuresResult]) + + if (!isOpen) return null + + const handleFolderSelect = async (path: string) => { + setStep('analyzing') + const success = await analyze(path) + if (success) { + setStep('detected') + } + } + + const handleExtractFeatures = async () => { + const success = await extractFeatures() + if (success) { + setStep('features') + // Expand all categories by default - need to get fresh state via callback + // The featuresResult will be available after the state update from extractFeatures + } + } + + const handleContinueToRegister = () => { + // Generate default project name from path + const pathParts = state.projectPath?.split(/[/\\]/) || [] + const defaultName = pathParts[pathParts.length - 1] || 'imported-project' + setProjectName(defaultName.replace(/[^a-zA-Z0-9_-]/g, '-')) + setStep('register') + } + + const handleRegisterAndCreate = async () => { + if (!projectName.trim() || !state.projectPath) return + + setRegisterError(null) + + try { + // First register the project + await createProject.mutateAsync({ + name: projectName.trim(), + path: state.projectPath, + specMethod: 'manual', + }) + + // Then create features + const success = await createFeatures(projectName.trim()) + + if (success) { + setStep('complete') + setTimeout(() => { + onProjectImported(projectName.trim()) + handleClose() + }, 1500) + } + } catch (err) { + setRegisterError(err instanceof Error ? err.message : 'Failed to register project') + } + } + + const handleClose = () => { + setStep('folder') + setProjectName('') + setExpandedCategories(new Set()) + setRegisterError(null) + reset() + onClose() + } + + const handleBack = () => { + if (step === 'detected') { + setStep('folder') + reset() + } else if (step === 'features') { + setStep('detected') + } else if (step === 'register') { + setStep('features') + } + } + + const toggleCategory = (category: string) => { + setExpandedCategories(prev => { + const next = new Set(prev) + if (next.has(category)) { + next.delete(category) + } else { + next.add(category) + } + return next + }) + } + + const getStackIcon = (category: string) => { + switch (category.toLowerCase()) { + case 'frontend': + return + case 'backend': + return + case 'database': + return + default: + return + } + } + + // Folder selection step + if (step === 'folder') { + return ( +
+
e.stopPropagation()} + > +
+
+ +
+

+ Import Existing Project +

+

+ Select the folder containing your existing project +

+
+
+ +
+ +
+ +
+
+
+ ) + } + + // Analyzing step + if (step === 'analyzing' || state.step === 'analyzing') { + return ( +
+
e.stopPropagation()} + > +
+

+ Analyzing Project +

+ +
+ +
+
+ +
+

Detecting Tech Stack

+

+ Scanning your project for frameworks, routes, and components... +

+ +
+
+
+ ) + } + + // Error state + if (state.step === 'error') { + return ( +
+
e.stopPropagation()} + > +
+

+ Error +

+ +
+ +
+
+ +
+

Analysis Failed

+

{state.error}

+ +
+
+
+ ) + } + + // Detection results step + if (step === 'detected' && state.analyzeResult) { + const result = state.analyzeResult + return ( +
+
e.stopPropagation()} + > +
+
+ +

+ Stack Detected +

+
+ +
+ +
+ {/* Summary */} +
+

{result.summary}

+
+ + {/* Detected Stacks */} +

Detected Technologies

+
+ {result.detected_stacks.map((stack, i) => ( +
+ {getStackIcon(stack.category)} +
+
{stack.name}
+
+ {stack.category} +
+
+
+ {Math.round(stack.confidence * 100)}% +
+
+ ))} +
+ + {/* Stats */} +

Codebase Analysis

+
+
+
+ {result.routes_count} +
+
Routes
+
+
+
+ {result.endpoints_count} +
+
Endpoints
+
+
+
+ {result.components_count} +
+
Components
+
+
+
+ +
+ + +
+
+
+ ) + } + + // Features review step + if (step === 'features' && state.featuresResult) { + const result = state.featuresResult + const categories = Object.keys(result.by_category) + + // Group features by category + const featuresByCategory: Record = {} + result.features.forEach(f => { + if (!featuresByCategory[f.category]) { + featuresByCategory[f.category] = [] + } + featuresByCategory[f.category].push(f) + }) + + return ( +
+
e.stopPropagation()} + > +
+
+ +
+

+ Review Features +

+

+ {state.selectedFeatures.length} of {result.count} features selected +

+
+
+ +
+ + {/* Selection controls */} +
+ + +
+ +
+ {categories.map(category => ( +
+ + + {expandedCategories.has(category) && ( +
+ {featuresByCategory[category]?.map((feature, i) => { + const isSelected = state.selectedFeatures.some( + f => f.name === feature.name && f.category === feature.category + ) + return ( +
toggleFeature(feature)} + className={` + flex items-start gap-3 p-3 cursor-pointer transition-all + border-2 border-[var(--color-neo-border)] + ${isSelected + ? 'bg-[var(--color-neo-done-light)] border-[var(--color-neo-done)]' + : 'bg-white hover:bg-[var(--color-neo-bg-secondary)]' + } + `} + > + {isSelected ? ( + + ) : ( + + )} +
+
{feature.name}
+
+ {feature.description} +
+
+ + {feature.source_type} + + {feature.source_file && ( + + {feature.source_file} + + )} +
+
+
+ ) + })} +
+ )} +
+ ))} +
+ +
+ + +
+
+
+ ) + } + + // Register project step + if (step === 'register') { + return ( +
+
e.stopPropagation()} + > +
+

+ Register Project +

+ +
+ +
+
+ + setProjectName(e.target.value)} + placeholder="my-project" + className="neo-input" + pattern="^[a-zA-Z0-9_-]+$" + autoFocus + /> +

+ Use letters, numbers, hyphens, and underscores only. +

+
+ +
+
+
+ Features to create: + {state.selectedFeatures.length} +
+
+ Project path: + + {state.projectPath} + +
+
+
+ + {(registerError || state.error) && ( +
+ {registerError || state.error} +
+ )} + +
+ + +
+
+
+
+ ) + } + + // Complete step + if (step === 'complete') { + return ( +
+
e.stopPropagation()} + > +
+

+ Import Complete +

+
+ +
+
+ +
+

{projectName}

+

+ Project imported successfully! +

+

+ {state.createResult?.created} features created +

+
+ + Redirecting... +
+
+
+
+ ) + } + + return null +} diff --git a/ui/src/components/NewProjectModal.tsx b/ui/src/components/NewProjectModal.tsx index 38e567f6..e41efa16 100644 --- a/ui/src/components/NewProjectModal.tsx +++ b/ui/src/components/NewProjectModal.tsx @@ -10,29 +10,17 @@ */ import { useState } from 'react' -import { Bot, FileEdit, ArrowRight, ArrowLeft, Loader2, CheckCircle2, Folder } from 'lucide-react' +import { X, Bot, FileEdit, ArrowRight, ArrowLeft, Loader2, CheckCircle2, Folder, Download } from 'lucide-react' import { useCreateProject } from '../hooks/useProjects' import { SpecCreationChat } from './SpecCreationChat' import { FolderBrowser } from './FolderBrowser' +import { ImportProjectModal } from './ImportProjectModal' import { startAgent } from '../lib/api' -import { - Dialog, - DialogContent, - DialogHeader, - DialogTitle, - DialogDescription, - DialogFooter, -} from '@/components/ui/dialog' -import { Button } from '@/components/ui/button' -import { Input } from '@/components/ui/input' -import { Label } from '@/components/ui/label' -import { Alert, AlertDescription } from '@/components/ui/alert' -import { Badge } from '@/components/ui/badge' -import { Card, CardContent } from '@/components/ui/card' type InitializerStatus = 'idle' | 'starting' | 'error' -type Step = 'name' | 'folder' | 'method' | 'chat' | 'complete' +type Step = 'choose' | 'name' | 'folder' | 'method' | 'chat' | 'complete' | 'import' +type ProjectType = 'new' | 'import' type SpecMethod = 'claude' | 'manual' interface NewProjectModalProps { @@ -48,18 +36,16 @@ export function NewProjectModal({ onProjectCreated, onStepChange, }: NewProjectModalProps) { - const [step, setStep] = useState('name') + const [step, setStep] = useState('choose') + const [, setProjectType] = useState(null) const [projectName, setProjectName] = useState('') const [projectPath, setProjectPath] = useState(null) - const [_specMethod, setSpecMethod] = useState(null) + const [, setSpecMethod] = useState(null) const [error, setError] = useState(null) const [initializerStatus, setInitializerStatus] = useState('idle') const [initializerError, setInitializerError] = useState(null) const [yoloModeSelected, setYoloModeSelected] = useState(false) - // Suppress unused variable warning - specMethod may be used in future - void _specMethod - const createProject = useCreateProject() // Wrapper to notify parent of step changes @@ -89,7 +75,7 @@ export function NewProjectModal({ } const handleFolderSelect = (path: string) => { - setProjectPath(path) + setProjectPath(path) // Use selected path directly - no subfolder creation changeStep('method') } @@ -179,7 +165,8 @@ export function NewProjectModal({ } const handleClose = () => { - changeStep('name') + changeStep('choose') + setProjectType(null) setProjectName('') setProjectPath(null) setSpecMethod(null) @@ -197,13 +184,41 @@ export function NewProjectModal({ } else if (step === 'folder') { changeStep('name') setProjectPath(null) + } else if (step === 'name') { + changeStep('choose') + setProjectType(null) + } + } + + const handleProjectTypeSelect = (type: ProjectType) => { + setProjectType(type) + if (type === 'new') { + changeStep('name') + } else { + changeStep('import') } } + const handleImportComplete = (importedProjectName: string) => { + onProjectCreated(importedProjectName) + handleClose() + } + + // Import project view + if (step === 'import') { + return ( + + ) + } + // Full-screen chat view if (step === 'chat') { return ( -
+
!open && handleClose()}> - +
+
e.stopPropagation()} + > {/* Header */} - +
- +
- Select Project Location - - Select the folder to use for project {projectName}. Create a new folder or choose an existing one. - +

+ Select Project Location +

+

+ Select the folder to use for project {projectName}. Create a new folder or choose an existing one. +

- + +
{/* Folder Browser */}
@@ -242,151 +268,270 @@ export function NewProjectModal({ onCancel={handleFolderCancel} />
- - +
+
) } return ( - !open && handleClose()}> - - - +
+
e.stopPropagation()} + > + {/* Header */} +
+

+ {step === 'choose' && 'New Project'} {step === 'name' && 'Create New Project'} {step === 'method' && 'Choose Setup Method'} {step === 'complete' && 'Project Created!'} - - - - {/* Step 1: Project Name */} - {step === 'name' && ( -
-
- - setProjectName(e.target.value)} - placeholder="my-awesome-app" - pattern="^[a-zA-Z0-9_-]+$" - autoFocus - /> -

- Use letters, numbers, hyphens, and underscores only. +

+ +
+ + {/* Content */} +
+ {/* Step 0: Choose project type */} + {step === 'choose' && ( +
+

+ What would you like to do?

-
- {error && ( - - {error} - - )} - - - - - - )} - - {/* Step 2: Spec Method */} - {step === 'method' && ( -
- - How would you like to define your project? - - -
- {/* Claude option */} - !createProject.isPending && handleMethodSelect('claude')} - > - +
+ {/* New project option */} + + + {/* Import existing option */} + +
+ )} + + {/* Step 1: Project Name */} + {step === 'name' && ( +
+
+ + setProjectName(e.target.value)} + placeholder="my-awesome-app" + className="neo-input" + pattern="^[a-zA-Z0-9_-]+$" + autoFocus + /> +

+ Use letters, numbers, hyphens, and underscores only. +

+
- {error && ( - - {error} - - )} - - {createProject.isPending && ( -
- - Creating project... + {error && ( +
+ {error} +
+ )} + +
+ +
- )} + + )} + + {/* Step 2: Spec Method */} + {step === 'method' && ( +
+

+ How would you like to define your project? +

- - - -
- )} +
+ {/* Claude option */} + + + {/* Manual option */} + +
- {/* Step 3: Complete */} - {step === 'complete' && ( -
-
- + {error && ( +
+ {error} +
+ )} + + {createProject.isPending && ( +
+ + Creating project... +
+ )} + +
+ +
-

{projectName}

-

- Your project has been created successfully! -

-
- - Redirecting... + )} + + {/* Step 3: Complete */} + {step === 'complete' && ( +
+
+ +
+

+ {projectName} +

+

+ Your project has been created successfully! +

+
+ + Redirecting... +
-
- )} - -
+ )} +
+
+ ) } diff --git a/ui/src/hooks/useImportProject.ts b/ui/src/hooks/useImportProject.ts new file mode 100644 index 00000000..bd30ca26 --- /dev/null +++ b/ui/src/hooks/useImportProject.ts @@ -0,0 +1,248 @@ +/** + * Hook for managing project import workflow + * + * Handles: + * - Stack detection via API + * - Feature extraction + * - Feature creation in database + */ + +import { useState, useCallback } from 'react' +import { API_BASE_URL } from '../lib/api' + +// API response types +interface StackInfo { + name: string + category: string + confidence: number +} + +interface AnalyzeResponse { + project_dir: string + detected_stacks: StackInfo[] + primary_frontend: string | null + primary_backend: string | null + database: string | null + routes_count: number + components_count: number + endpoints_count: number + summary: string +} + +interface DetectedFeature { + category: string + name: string + description: string + steps: string[] + source_type: string + source_file: string | null + confidence: number +} + +interface ExtractFeaturesResponse { + features: DetectedFeature[] + count: number + by_category: Record + summary: string +} + +interface CreateFeaturesResponse { + created: number + project_name: string + message: string +} + +// Hook state +interface ImportState { + step: 'idle' | 'analyzing' | 'detected' | 'extracting' | 'extracted' | 'creating' | 'complete' | 'error' + projectPath: string | null + analyzeResult: AnalyzeResponse | null + featuresResult: ExtractFeaturesResponse | null + createResult: CreateFeaturesResponse | null + error: string | null + selectedFeatures: DetectedFeature[] +} + +export interface UseImportProjectReturn { + state: ImportState + analyze: (path: string) => Promise + extractFeatures: () => Promise + createFeatures: (projectName: string) => Promise + toggleFeature: (feature: DetectedFeature) => void + selectAllFeatures: () => void + deselectAllFeatures: () => void + reset: () => void +} + +const initialState: ImportState = { + step: 'idle', + projectPath: null, + analyzeResult: null, + featuresResult: null, + createResult: null, + error: null, + selectedFeatures: [], +} + +export function useImportProject(): UseImportProjectReturn { + const [state, setState] = useState(initialState) + + const analyze = useCallback(async (path: string): Promise => { + setState(prev => ({ ...prev, step: 'analyzing', projectPath: path, error: null })) + + try { + const response = await fetch(`${API_BASE_URL}/api/import/analyze`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ path }), + }) + + if (!response.ok) { + const error = await response.json() + throw new Error(error.detail || 'Failed to analyze project') + } + + const result: AnalyzeResponse = await response.json() + setState(prev => ({ + ...prev, + step: 'detected', + analyzeResult: result, + })) + return true + } catch (err) { + setState(prev => ({ + ...prev, + step: 'error', + error: err instanceof Error ? err.message : 'Analysis failed', + })) + return false + } + }, []) + + const extractFeatures = useCallback(async (): Promise => { + if (!state.projectPath) return false + + setState(prev => ({ ...prev, step: 'extracting', error: null })) + + try { + const response = await fetch(`${API_BASE_URL}/api/import/extract-features`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ path: state.projectPath }), + }) + + if (!response.ok) { + const error = await response.json() + throw new Error(error.detail || 'Failed to extract features') + } + + const result: ExtractFeaturesResponse = await response.json() + setState(prev => ({ + ...prev, + step: 'extracted', + featuresResult: result, + selectedFeatures: result.features, // Select all by default + })) + return true + } catch (err) { + setState(prev => ({ + ...prev, + step: 'error', + error: err instanceof Error ? err.message : 'Feature extraction failed', + })) + return false + } + }, [state.projectPath]) + + const createFeatures = useCallback(async (projectName: string): Promise => { + if (!state.selectedFeatures.length) return false + + setState(prev => ({ ...prev, step: 'creating', error: null })) + + try { + const features = state.selectedFeatures.map(f => ({ + category: f.category, + name: f.name, + description: f.description, + steps: f.steps, + })) + + const response = await fetch(`${API_BASE_URL}/api/import/create-features`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ project_name: projectName, features }), + }) + + if (!response.ok) { + const error = await response.json() + throw new Error(error.detail || 'Failed to create features') + } + + const result: CreateFeaturesResponse = await response.json() + setState(prev => ({ + ...prev, + step: 'complete', + createResult: result, + })) + return true + } catch (err) { + setState(prev => ({ + ...prev, + step: 'error', + error: err instanceof Error ? err.message : 'Feature creation failed', + })) + return false + } + }, [state.selectedFeatures]) + + const toggleFeature = useCallback((feature: DetectedFeature) => { + setState(prev => { + const isSelected = prev.selectedFeatures.some( + f => f.name === feature.name && f.category === feature.category + ) + + if (isSelected) { + return { + ...prev, + selectedFeatures: prev.selectedFeatures.filter( + f => !(f.name === feature.name && f.category === feature.category) + ), + } + } else { + return { + ...prev, + selectedFeatures: [...prev.selectedFeatures, feature], + } + } + }) + }, []) + + const selectAllFeatures = useCallback(() => { + setState(prev => ({ + ...prev, + selectedFeatures: prev.featuresResult?.features || [], + })) + }, []) + + const deselectAllFeatures = useCallback(() => { + setState(prev => ({ + ...prev, + selectedFeatures: [], + })) + }, []) + + const reset = useCallback(() => { + setState(initialState) + }, []) + + return { + state, + analyze, + extractFeatures, + createFeatures, + toggleFeature, + selectAllFeatures, + deselectAllFeatures, + reset, + } +} diff --git a/ui/src/lib/api.ts b/ui/src/lib/api.ts index 7ef9a8ab..50925bb1 100644 --- a/ui/src/lib/api.ts +++ b/ui/src/lib/api.ts @@ -35,6 +35,9 @@ import type { const API_BASE = '/api' +// Export for hooks that make direct fetch calls with full paths +export const API_BASE_URL = '' + async function fetchJSON(url: string, options?: RequestInit): Promise { const response = await fetch(`${API_BASE}${url}`, { ...options, diff --git a/visual_regression.py b/visual_regression.py new file mode 100644 index 00000000..d9767b02 --- /dev/null +++ b/visual_regression.py @@ -0,0 +1,513 @@ +""" +Visual Regression Testing +========================= + +Screenshot comparison testing for detecting unintended UI changes. + +Features: +- Capture screenshots after feature completion via Playwright +- Store baselines in .visual-snapshots/ +- Compare screenshots with configurable threshold +- Generate diff images highlighting changes +- Flag features for review when changes detected +- Support for multiple viewports and themes + +Configuration: +- visual_regression.enabled: Enable/disable visual testing +- visual_regression.threshold: Pixel difference threshold (default: 0.1%) +- visual_regression.viewports: List of viewport sizes to test +- visual_regression.capture_on_pass: Capture on feature pass (default: true) + +Requirements: +- Playwright must be installed: pip install playwright +- Browsers must be installed: playwright install chromium +""" + +import asyncio +import json +import logging +from dataclasses import dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Optional + +logger = logging.getLogger(__name__) + +# Check for PIL availability +try: + from PIL import Image, ImageChops, ImageDraw + + HAS_PIL = True +except ImportError: + HAS_PIL = False + logger.warning("Pillow not installed. Install with: pip install Pillow") + +# Check for Playwright availability +try: + from playwright.async_api import async_playwright + + HAS_PLAYWRIGHT = True +except ImportError: + HAS_PLAYWRIGHT = False + logger.warning("Playwright not installed. Install with: pip install playwright") + + +@dataclass +class Viewport: + """Screen viewport configuration.""" + + name: str + width: int + height: int + + @classmethod + def desktop(cls) -> "Viewport": + return cls("desktop", 1920, 1080) + + @classmethod + def tablet(cls) -> "Viewport": + return cls("tablet", 768, 1024) + + @classmethod + def mobile(cls) -> "Viewport": + return cls("mobile", 375, 667) + + +@dataclass +class SnapshotResult: + """Result of a snapshot comparison.""" + + name: str + viewport: str + baseline_path: Optional[str] = None + current_path: Optional[str] = None + diff_path: Optional[str] = None + diff_percentage: float = 0.0 + passed: bool = True + is_new: bool = False + error: Optional[str] = None + + def to_dict(self) -> dict: + return { + "name": self.name, + "viewport": self.viewport, + "baseline_path": self.baseline_path, + "current_path": self.current_path, + "diff_path": self.diff_path, + "diff_percentage": self.diff_percentage, + "passed": self.passed, + "is_new": self.is_new, + "error": self.error, + } + + +@dataclass +class TestReport: + """Visual regression test report.""" + + project_dir: str + test_time: str + results: list[SnapshotResult] = field(default_factory=list) + total: int = 0 + passed: int = 0 + failed: int = 0 + new: int = 0 + + def __post_init__(self): + if not self.test_time: + self.test_time = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") + + def to_dict(self) -> dict: + return { + "project_dir": self.project_dir, + "test_time": self.test_time, + "results": [r.to_dict() for r in self.results], + "summary": { + "total": self.total, + "passed": self.passed, + "failed": self.failed, + "new": self.new, + }, + } + + +class VisualRegressionTester: + """ + Visual regression testing using Playwright screenshots. + + Usage: + tester = VisualRegressionTester(project_dir) + report = await tester.test_page("http://localhost:3000", "homepage") + tester.save_report(report) + """ + + def __init__( + self, + project_dir: Path, + threshold: float = 0.1, + viewports: Optional[list[Viewport]] = None, + ): + self.project_dir = Path(project_dir) + self.threshold = threshold # Percentage difference allowed + self.viewports = viewports or [Viewport.desktop()] + self.snapshots_dir = self.project_dir / ".visual-snapshots" + self.baselines_dir = self.snapshots_dir / "baselines" + self.current_dir = self.snapshots_dir / "current" + self.diff_dir = self.snapshots_dir / "diffs" + + # Ensure directories exist + self.baselines_dir.mkdir(parents=True, exist_ok=True) + self.current_dir.mkdir(parents=True, exist_ok=True) + self.diff_dir.mkdir(parents=True, exist_ok=True) + + async def capture_screenshot( + self, + url: str, + name: str, + viewport: Optional[Viewport] = None, + wait_for: Optional[str] = None, + full_page: bool = True, + ) -> Path: + """ + Capture a screenshot using Playwright. + + Args: + url: URL to capture + name: Screenshot name + viewport: Viewport configuration + wait_for: CSS selector to wait for before capture + full_page: Capture full scrollable page + + Returns: + Path to saved screenshot + """ + if not HAS_PLAYWRIGHT: + raise RuntimeError("Playwright not installed. Run: pip install playwright && playwright install chromium") + + viewport = viewport or Viewport.desktop() + filename = f"{name}_{viewport.name}.png" + output_path = self.current_dir / filename + + async with async_playwright() as p: + browser = await p.chromium.launch() + page = await browser.new_page( + viewport={"width": viewport.width, "height": viewport.height} + ) + + try: + await page.goto(url, wait_until="networkidle") + + if wait_for: + await page.wait_for_selector(wait_for, timeout=10000) + + # Small delay for animations to settle + await asyncio.sleep(0.5) + + await page.screenshot(path=str(output_path), full_page=full_page) + + finally: + await browser.close() + + return output_path + + def compare_images( + self, + baseline_path: Path, + current_path: Path, + diff_path: Path, + ) -> tuple[float, bool]: + """ + Compare two images and generate diff. + + Args: + baseline_path: Path to baseline image + current_path: Path to current image + diff_path: Path to save diff image + + Returns: + Tuple of (diff_percentage, passed) + """ + if not HAS_PIL: + raise RuntimeError("Pillow not installed. Run: pip install Pillow") + + # Use context managers to ensure file handles are properly closed + with Image.open(baseline_path) as baseline_img, Image.open(current_path) as current_img: + baseline = baseline_img.convert("RGB") + current = current_img.convert("RGB") + + # Resize if dimensions differ + if baseline.size != current.size: + current = current.resize(baseline.size, Image.Resampling.LANCZOS) + + # Calculate difference + diff = ImageChops.difference(baseline, current) + + # Count different pixels + diff_data = diff.getdata() + total_pixels = baseline.size[0] * baseline.size[1] + diff_pixels = sum(1 for pixel in diff_data if sum(pixel) > 30) # Threshold for "different" + + diff_percentage = (diff_pixels / total_pixels) * 100 + + # Generate highlighted diff image + if diff_percentage > 0: + # Create diff overlay + diff_highlight = Image.new("RGBA", baseline.size, (0, 0, 0, 0)) + draw = ImageDraw.Draw(diff_highlight) + + for y in range(baseline.size[1]): + for x in range(baseline.size[0]): + pixel = diff.getpixel((x, y)) + if sum(pixel) > 30: + draw.point((x, y), fill=(255, 0, 0, 128)) # Red highlight + + # Composite with original + result = Image.alpha_composite(baseline.convert("RGBA"), diff_highlight) + result.save(diff_path) + + passed = diff_percentage <= self.threshold + + return diff_percentage, passed + + async def test_page( + self, + url: str, + name: str, + wait_for: Optional[str] = None, + update_baseline: bool = False, + ) -> TestReport: + """ + Test a page across all viewports. + + Args: + url: URL to test + name: Test name + wait_for: CSS selector to wait for + update_baseline: If True, update baselines instead of comparing + + Returns: + TestReport with results + """ + report = TestReport( + project_dir=str(self.project_dir), + test_time=datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + ) + + for viewport in self.viewports: + result = SnapshotResult(name=name, viewport=viewport.name) + + try: + # Capture current screenshot + current_path = await self.capture_screenshot( + url, name, viewport, wait_for + ) + result.current_path = str(current_path.relative_to(self.project_dir)) + + # Check for baseline + baseline_filename = f"{name}_{viewport.name}.png" + baseline_path = self.baselines_dir / baseline_filename + result.baseline_path = str(baseline_path.relative_to(self.project_dir)) + + if not baseline_path.exists() or update_baseline: + # New baseline - copy current to baseline + import shutil + + shutil.copy(current_path, baseline_path) + result.is_new = True + result.passed = True + report.new += 1 + else: + # Compare with baseline + diff_filename = f"{name}_{viewport.name}_diff.png" + diff_path = self.diff_dir / diff_filename + + diff_percentage, passed = self.compare_images( + baseline_path, current_path, diff_path + ) + + result.diff_percentage = diff_percentage + result.passed = passed + + if not passed: + result.diff_path = str(diff_path.relative_to(self.project_dir)) + report.failed += 1 + else: + report.passed += 1 + + except Exception as e: + result.error = str(e) + result.passed = False + report.failed += 1 + logger.error(f"Visual test error for {name}/{viewport.name}: {e}") + + report.results.append(result) + report.total += 1 + + return report + + async def test_routes( + self, + base_url: str, + routes: list[dict], + update_baseline: bool = False, + ) -> TestReport: + """ + Test multiple routes. + + Args: + base_url: Base URL (e.g., http://localhost:3000) + routes: List of routes to test [{"path": "/", "name": "home", "wait_for": "#app"}] + update_baseline: Update baselines instead of comparing + + Returns: + Combined TestReport + """ + combined_report = TestReport( + project_dir=str(self.project_dir), + test_time=datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + ) + + for route in routes: + url = base_url.rstrip("/") + route["path"] + name = route.get("name", route["path"].replace("/", "_").strip("_") or "home") + wait_for = route.get("wait_for") + + report = await self.test_page(url, name, wait_for, update_baseline) + + combined_report.results.extend(report.results) + combined_report.total += report.total + combined_report.passed += report.passed + combined_report.failed += report.failed + combined_report.new += report.new + + return combined_report + + def save_report(self, report: TestReport) -> Path: + """Save test report to file.""" + reports_dir = self.snapshots_dir / "reports" + reports_dir.mkdir(parents=True, exist_ok=True) + + timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") + report_path = reports_dir / f"visual_test_{timestamp}.json" + + with open(report_path, "w") as f: + json.dump(report.to_dict(), f, indent=2) + + return report_path + + def update_baseline(self, name: str, viewport: str) -> bool: + """ + Accept current screenshot as new baseline. + + Args: + name: Test name + viewport: Viewport name + + Returns: + True if successful + """ + # Validate inputs to prevent path traversal + if ".." in name or "/" in name or "\\" in name: + raise ValueError(f"Invalid test name: {name}") + if ".." in viewport or "/" in viewport or "\\" in viewport: + raise ValueError(f"Invalid viewport name: {viewport}") + + filename = f"{name}_{viewport}.png" + current_path = self.current_dir / filename + baseline_path = self.baselines_dir / filename + + if current_path.exists(): + import shutil + + shutil.copy(current_path, baseline_path) + + # Clean up diff if exists + diff_path = self.diff_dir / f"{name}_{viewport}_diff.png" + if diff_path.exists(): + diff_path.unlink() + + return True + + return False + + def list_baselines(self) -> list[dict]: + """List all baseline snapshots.""" + baselines = [] + + for file in self.baselines_dir.glob("*.png"): + stat = file.stat() + parts = file.stem.rsplit("_", 1) + name = parts[0] if len(parts) > 1 else file.stem + viewport = parts[1] if len(parts) > 1 else "desktop" + + baselines.append( + { + "name": name, + "viewport": viewport, + "filename": file.name, + "size": stat.st_size, + "modified": datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc).isoformat(), + } + ) + + return baselines + + def delete_baseline(self, name: str, viewport: str) -> bool: + """Delete a baseline snapshot.""" + # Validate inputs to prevent path traversal + if ".." in name or "/" in name or "\\" in name: + raise ValueError(f"Invalid test name: {name}") + if ".." in viewport or "/" in viewport or "\\" in viewport: + raise ValueError(f"Invalid viewport name: {viewport}") + + filename = f"{name}_{viewport}.png" + baseline_path = self.baselines_dir / filename + + if baseline_path.exists(): + baseline_path.unlink() + return True + + return False + + +async def run_visual_tests( + project_dir: Path, + base_url: str, + routes: Optional[list[dict]] = None, + threshold: float = 0.1, + update_baseline: bool = False, +) -> TestReport: + """ + Run visual regression tests for a project. + + Args: + project_dir: Project directory + base_url: Base URL to test + routes: Routes to test (default: [{"path": "/", "name": "home"}]) + threshold: Diff threshold percentage + update_baseline: Update baselines instead of comparing + + Returns: + TestReport with results + """ + if routes is None: + routes = [{"path": "/", "name": "home"}] + + tester = VisualRegressionTester(project_dir, threshold) + report = await tester.test_routes(base_url, routes, update_baseline) + tester.save_report(report) + + return report + + +def run_visual_tests_sync( + project_dir: Path, + base_url: str, + routes: Optional[list[dict]] = None, + threshold: float = 0.1, + update_baseline: bool = False, +) -> TestReport: + """Synchronous wrapper for run_visual_tests.""" + return asyncio.run( + run_visual_tests(project_dir, base_url, routes, threshold, update_baseline) + )