|
| 1 | +# Gemini Project Guide: Evolve on Click (EvOC) |
| 2 | + |
| 3 | +This document provides a comprehensive overview of the "Evolve on Click" project, specifically focusing on the `controller_microservice_v2`. It is intended as a guide for AI agents and developers to understand the project's architecture, conventions, and operational procedures. |
| 4 | + |
| 5 | +## 1. Project Overview |
| 6 | + |
| 7 | +This project is the central backend controller for the **Evolve on Click (EvOC) v2** platform, a system designed to provide a Jupyter-style notebook interface for working with Evolutionary Algorithms (EAs). |
| 8 | + |
| 9 | +The `controller_microservice_v2` is a **Go-based** microservice that acts as the brain of the operation. It handles API requests from the frontend, manages the lifecycle of notebooks and coding problems, and communicates with a `Jupyter Kernel Gateway` to execute code written by users. |
| 10 | + |
| 11 | +The broader architecture, as defined in `docker-compose.yaml` and design documents, includes: |
| 12 | +- **`controller_microservice_v2`**: The main Go service. |
| 13 | +- **`jupyter_gateway`**: Manages code execution kernels. |
| 14 | +- **`python_runner`**: The Docker environment where Python code (using libraries like `deap`) is executed. |
| 15 | +- **`cockroachdb`**: A distributed SQL database for storing all metadata related to users, problems, notebooks, and evolution runs. |
| 16 | +- **`minio`**: An S3-compatible object store for large files like plots and graphs. |
| 17 | +- **AI Pipeline**: A separate, asynchronous service (communicating via a message queue) responsible for the "code evolution" feature. |
| 18 | + |
| 19 | +## 2. Building and Running |
| 20 | + |
| 21 | +The project uses a `Makefile` for standardized development and operational commands. |
| 22 | + |
| 23 | +- **Start the entire stack (recommended for development):** |
| 24 | + ```bash |
| 25 | + make docker-up |
| 26 | + ``` |
| 27 | + |
| 28 | +- **Run the Go service locally (requires other services to be running):** |
| 29 | + ```bash |
| 30 | + make run |
| 31 | + ``` |
| 32 | + |
| 33 | +- **Build the Go binary:** |
| 34 | + ```bash |
| 35 | + make build |
| 36 | + ``` |
| 37 | + |
| 38 | +- **Run tests:** |
| 39 | + ```bash |
| 40 | + make test |
| 41 | + ``` |
| 42 | + |
| 43 | +- **Stop and clean up Docker containers:** |
| 44 | + ```bash |
| 45 | + make clean |
| 46 | + ``` |
| 47 | + |
| 48 | +## 3. Development Conventions |
| 49 | + |
| 50 | +The project follows a set of clear conventions to ensure code quality and maintainability, as outlined in `ARCHITECTURE.md`. |
| 51 | + |
| 52 | +- **Layered Architecture**: Code is strictly separated into layers: |
| 53 | + **`routes`** (API definition) -> **`controllers`** (HTTP handling) -> **`modules`** (Business Logic) -> **`repository`** (Data Access). |
| 54 | + |
| 55 | +- **API Design**: |
| 56 | + - The API is versioned under `/api/v1/`. |
| 57 | + - It follows RESTful principles. |
| 58 | + - Routing is handled by the standard library's `http.ServeMux` with explicit `METHOD /path` patterns (e.g., `POST /api/v1/kernels`). |
| 59 | + |
| 60 | +- **Database Interaction**: |
| 61 | + - The database schema is defined in `db/schema.sql`. |
| 62 | + - The **Repository Pattern** is the standard for all database operations. Business logic in the `modules` layer should not contain raw SQL queries; it must call methods on a repository interface. |
| 63 | + |
| 64 | +- **Logging**: |
| 65 | + - **`zerolog`** is used for structured logging. |
| 66 | + - The use of `fmt.Print*` or `log.Print*` is forbidden and blocked by a pre-commit hook. |
| 67 | + - The logger instance is injected as a dependency from `main.go`. |
| 68 | + |
| 69 | +- **Configuration**: |
| 70 | + - All configuration (DB URLs, tokens, etc.) is managed via **environment variables**. |
| 71 | + - There should be **no hardcoded configuration values**. The `.env` file is used for local development. |
| 72 | + |
| 73 | +- **Tooling**: |
| 74 | + - `Makefile` provides standard commands for building, running, and testing. |
| 75 | + ## Gemini Added Memories |
| 76 | +- The user has successfully started both the `auth_microservice` and the `controller_microservice_v2`. I have implemented a gRPC authentication middleware in the controller that communicates with the auth service. The `POST /api/v1/sessions` route is now protected by this middleware. The user will begin testing this new authentication flow next. |
| 77 | +- I have refactored the notebook object structure in the frontend to align with the backend's `controller_microservice_v2` and `llm_microservice` expectations. This involved renaming `type` to `cell_type`, `content` to `source`, and adding `execution_count` to code cells. I updated `useNotebookCells.js`, `useNotebook.js`, `useNotebookFetch.js`, `notebook-mapper.js`, and `useNotebookExecution.js` to ensure consistency across the application. |
| 78 | +- I have improved the "modify" and "fix" functionality for individual cells and the chat window. This includes: |
| 79 | + - Modifying `useNotebookLLM.js` to return the full API response. |
| 80 | + - Updating `useNotebook.js` to correctly process API responses, handle conditional in-cell messages (only for single cell modifications), and manage chat messages. |
| 81 | + - Enhancing `ChatWindow.js` to display LLM responses, including `changes_made`, and visually distinguish user and bot messages. |
| 82 | + - Adding in-cell messages that appear for 5 seconds after a cell is modified or fixed. |
| 83 | + - Correcting the indexing logic in `useNotebook.js` for `cells_modified` to use the cell's index instead of its ID. |
| 84 | +- I have also updated the UI of the notebook page to match the theme of the rest of the application. This included: |
| 85 | + - Changing the background to a light gray gradient. |
| 86 | + - Applying the `Geist Mono` font to the entire notebook layout. |
| 87 | + - Updating buttons in `CodeCellControls`, `ChatWindow`, `KernelControls`, and `ActionsToolbarModern` to use a teal color scheme. |
| 88 | + - Updating the code cell containers to have rounded corners and borders consistent with other card components. |
| 89 | + - Improving the output area with a teal theme, better error display, and a "Clear" button. |
| 90 | + - Implementing a confirmation popup for deleting cells, and resizing it for better fit. |
| 91 | +- I have also implemented a new "Add Cell" functionality with a single plus icon at the top and bottom centers of each cell. Clicking this plus icon opens a small popup menu allowing the user to select between adding a "Code" or "Markdown" cell at that specific index. This functionality has been implemented in both `CodeCell.js` and `MarkdownCell.js`, and the `AddCellMenu` has been refactored into a separate reusable component. |
| 92 | +- I have implemented a new, more engaging loading screen for the notebook page (`NotebookLoadingScreen.js`) that displays cycling text and icons. I also fixed a critical bug where the semantic `cell_name` was being lost during API data mapping, which involved correcting logic in `useNotebookFetch.js`, `notebook-mapper.js`, and `useNotebook.js`. Finally, I improved the cell controls UI by adding a manual close button to in-cell messages and ensuring that loading spinners and disabled states are correctly applied during LLM operations. |
| 93 | +- I have implemented a new delta-based autosave system to efficiently persist notebook changes. This includes a new `useAutosave.js` hook that tracks changes (new, modified, deleted, and reordered cells) and sends them to a `PATCH /api/v1/notebooks/{id}/cells` endpoint every 5 minutes. The UI now includes a status indicator (`Saving...`, `Last saved at...`). I also re-integrated the manual save button to use this same efficient delta-based logic. |
| 94 | + |
| 95 | +## 4. LLM Microservice (`evocv2_llm_microservice`) |
| 96 | + |
| 97 | +This service is responsible for the AI-powered generation, modification, and fixing of DEAP (Distributed Evolutionary Algorithms in Python) code notebooks. |
| 98 | + |
| 99 | +### 4.1. Project Overview |
| 100 | + |
| 101 | +The `evocv2_llm_microservice` is a **Python-based** service using **FastAPI**. It exposes a REST API to create and manage 12-cell Jupyter notebooks for evolutionary algorithms. It uses a Large Language Model (LLM) via the Groq API to understand specifications and natural language instructions. |
| 102 | + |
| 103 | +- **Architecture**: |
| 104 | + - **FastAPI**: Serves the REST API. |
| 105 | + - **LangGraph**: Orchestrates the workflow for generating, modifying, and fixing notebooks, including validation and retry loops. |
| 106 | + - **Instructor & Pydantic**: Ensures structured, validated JSON output from the LLM. |
| 107 | + - **Groq**: Provides fast LLM inference. |
| 108 | + - **Mem0**: An optional memory layer to persist session history and user preferences. |
| 109 | +- **Key Features**: |
| 110 | + - **Generate**: Creates a complete, 12-cell DEAP notebook from a flexible JSON specification in a single LLM call. |
| 111 | + - **Modify**: Updates an existing notebook based on natural language instructions. |
| 112 | + - **Fix**: Attempts to automatically repair a broken notebook using an error traceback. |
| 113 | + - **Session Management**: Maintains the state of notebooks across API calls. |
| 114 | + |
| 115 | +### 4.2. Building and Running |
| 116 | + |
| 117 | +The service is designed to be run with Docker. |
| 118 | + |
| 119 | +- **Prerequisites**: |
| 120 | + - Docker and Docker Compose |
| 121 | + - A Groq API key, which should be placed in a `.env` file. |
| 122 | + |
| 123 | +- **Set up environment variables**: |
| 124 | + ```bash |
| 125 | + cp .env.example .env |
| 126 | + # Edit .env and add your GROQ_API_KEY |
| 127 | + ``` |
| 128 | + |
| 129 | +- **Build and run with Docker Compose**: |
| 130 | + ```bash |
| 131 | + docker-compose up --build |
| 132 | + ``` |
| 133 | + |
| 134 | +- **Run locally (without Docker)**: |
| 135 | + 1. Create a Python virtual environment and activate it. |
| 136 | + 2. Install dependencies: `pip install -r requirements.txt` |
| 137 | + 3. Run the server: `uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload` |
| 138 | + |
| 139 | +- **Verify it's running**: |
| 140 | + - Access the health check endpoint: `curl http://localhost:8000/health` |
| 141 | + - Or view the OpenAPI docs: `http://localhost:8000/docs` |
| 142 | +
|
| 143 | +### 4.3. API Endpoints |
| 144 | +
|
| 145 | +- `POST /v1/generate`: Creates a new notebook. |
| 146 | +- `POST /v1/sessions/{session_id}/modify`: Modifies an existing notebook. |
| 147 | +- `POST /v1/sessions/{session_id}/fix`: Fixes a broken notebook. |
| 148 | +- `GET /v1/sessions/{session_id}`: Retrieves session details. |
| 149 | +- `GET /v1/sessions`: Lists all active sessions. |
| 150 | +
|
| 151 | +## 5. Autosave and Persistence |
| 152 | +
|
| 153 | +To ensure user work is saved efficiently without overwhelming the backend, the notebook implements a **timed, delta-based autosave system**. |
| 154 | +
|
| 155 | +### 5.1. How It Works |
| 156 | +
|
| 157 | +1. **Change Tracking**: The frontend (`useAutosave.js` hook) constantly monitors the notebook for changes. It tracks: |
| 158 | + * New cells being added. |
| 159 | + * Existing cells being modified (source code changes). |
| 160 | + * Cells being deleted. |
| 161 | + * The order of cells being changed. |
| 162 | +2. **Dirty Flag**: Any change marks the notebook as "dirty," indicating it has unsaved changes. |
| 163 | +3. **Timed Autosave**: A timer runs every 5 minutes. If the notebook is "dirty," it automatically triggers a save. |
| 164 | +4. **Manual Save**: A manual save button is also available, which triggers the same save logic immediately. |
| 165 | +5. **Delta-Based Payload**: Instead of sending the entire notebook, the save logic calculates a "delta" containing only what has changed. This delta is sent to the backend. |
| 166 | +
|
| 167 | +### 5.2. API Endpoint |
| 168 | +
|
| 169 | +- **Endpoint**: `PATCH /api/v1/notebooks/{id}/cells` |
| 170 | +- **Method**: `PATCH` |
| 171 | +- **Payload Structure**: |
| 172 | + ```json |
| 173 | + { |
| 174 | + "updated_order": ["id-1", "id-3", "id-2"], |
| 175 | + "cells_to_upsert": { |
| 176 | + "id-3": { "cell_type": "code", "source": "print('modified')" }, |
| 177 | + "id-4": { "cell_type": "markdown", "source": "## New Cell" } |
| 178 | + }, |
| 179 | + "cells_to_delete": ["id-5"] |
| 180 | + } |
| 181 | + ``` |
| 182 | + *All fields are optional. Only fields with actual changes are sent.* |
| 183 | +
|
| 184 | +This approach minimizes network traffic and backend load, providing an efficient and reliable persistence layer. |
0 commit comments