Tasks

README.md

@@ -12,6 +12,9 @@ SProto is a lightweight, self-hostable registry for managing Protobuf (`.proto`)
 *   **Simple API:** RESTful API for publishing, fetching, and listing modules and versions.
 *   **CLI Client:** `protoreg-cli` for easy interaction with the registry from the command line.
 *   **Dockerized:** Easily deployable using Docker and Docker Compose.
+*   **Dependency Management:** Declare, resolve, and fetch module dependencies automatically.
+*   **Import Path Mapping:** Use logical import paths in your `.proto` files that map to registry modules.
+*   **Local Caching:** Store and reuse downloaded dependencies to improve build performance.
 
 ## Architecture
 
@@ -21,6 +24,7 @@ The system comprises the following components:
 2.  **Registry Client (`protoreg-cli`):** A Go CLI tool used by developers to publish proto directories and fetch specific module versions.
 3.  **PostgreSQL:** Stores metadata about modules (namespace, name) and their versions (version string, artifact digest, storage key).
 4.  **MinIO (or S3):** An S3-compatible object storage server used to store the zipped Protobuf artifacts.
+5.  **Local Cache:** Stores downloaded module artifacts on the local filesystem for improved performance and offline use.
 
 ```mermaid
 graph LR
@@ -33,6 +37,10 @@ Go App in Docker"];
 Metadata")];
     Server --> S3[("MinIO / S3
 Artifacts")];
+    CLI -- "1. Resolve deps" --> CLI;
+    CLI -- "2. Pull artifacts" --> CLI;
+    CLI -- "3. Cache locally" --> Cache[("Local Cache
+~/.cache/sproto")];
 ```
 
 ## Getting Started (Local Development)
@@ -136,6 +144,30 @@ For simpler deployments or local testing without external dependencies like Post
 *   **Network Exposure:** Ensure only necessary ports are exposed to the network. The default `docker-compose.yaml` exposes the server (8080) and MinIO UI (9090). Adjust as needed.
 *   **S3 Bucket Permissions:** If using a managed S3 service, configure bucket policies appropriately to restrict access.
 
+## Dependency Management
+
+SProto now supports Buf-like dependency management for Protobuf files, allowing you to:
+
+* Declare dependencies in a `sproto.yaml` configuration file
+* Use import paths like `import "github.com/myorg/common/proto/types.proto"`
+* Automatically fetch and cache dependencies
+* Generate correct `--proto_path` arguments for protoc
+
+### Configuration File Format
+
+Dependencies are managed through a `sproto.yaml` file in the root of your proto directory:
+
+```yaml
+version: v1
+name: mycompany/myapp
+import_path: github.com/mycompany/myapp
+dependencies:
+  - namespace: mycompany
+    name: common
+    version: ">=v1.0.0 <v2.0.0"
+    import_path: github.com/mycompany/common
+```
+
 ## CLI Usage (`protoreg-cli`)
 
 The CLI tool provides commands to interact with the registry.
@@ -194,6 +226,9 @@ The CLI loads its configuration (Registry URL and API Token) with the following
     # Usage: ./protoreg-cli fetch <namespace/module_name> <version> --output <dir>
     ./protoreg-cli fetch mycompany/user v1.0.0 --output ./downloaded-protos
     # Files will be extracted to ./downloaded-protos/mycompany/user/v1.0.0/
+
+    # Fetch a module and all its dependencies
+    ./protoreg-cli fetch mycompany/auth v1.0.0 --output ./protos --with-deps
     ```
 
 4.  **`list`**: Lists modules or versions.
@@ -204,6 +239,41 @@ The CLI loads its configuration (Registry URL and API Token) with the following
     # List versions for a specific module
     ./protoreg-cli list mycompany/user
     ```
+
+### Dependency Resolution Commands
+
+1. **`resolve`**: Resolves and fetches all dependencies for a module.
+   ```bash
+   # Resolve dependencies for the current directory (using sproto.yaml)
+   ./protoreg-cli resolve
+
+   # Resolve dependencies for a specific module version
+   ./protoreg-cli resolve mycompany/common@v1.0.0
+
+   # Force re-fetching even if cached
+   ./protoreg-cli resolve --update
+   ```
+
+2. **`compile`**: Simplifies running protoc with the correct include paths:
+   ```bash
+   # Compile with resolved dependencies
+   ./protoreg-cli compile --go_out=./gen
+   ```
+
+3. **`cache`**: Manages the local module cache:
+   ```bash
+   # List cached modules
+   ./protoreg-cli cache list
+
+   # Clean the cache
+   ./protoreg-cli cache clean
+   ```
+
+### Dependency Management Documentation
+
+* [Configuration File Format](docs/sproto-yaml-spec.md) - Full specification for sproto.yaml
+* [Usage Examples](docs/usage-examples.md) - Examples of common workflows
+* [Migrating from Buf](docs/migrating-from-buf.md) - Guide for existing Buf users
 
 ## API Specification
 

Task/Overview.md

@@ -0,0 +1,37 @@
+# SProto Enhancement - Dependency Management System
+
+## Section 1: Goal
+
+### Overview
+
+Our goal is to enhance SProto to provide Buf-like dependency management capabilities for Protobuf files, while leveraging SProto's existing registry functionality. 
+
+Currently, SProto functions as a registry for storing, versioning, and retrieving Protobuf modules, but lacks the dependency management features that Buf provides. This enhancement will bridge that gap, allowing users to:
+
+1. Use Go module style imports (e.g., `import "github.com/myorg/common/proto/types.proto"`)
+2. Declare dependencies in a configuration file (similar to buf.yaml)
+3. Benefit from automatic dependency resolution and fetching
+4. Avoid manual --proto_path configuration
+5. Utilize a local cache for downloaded dependencies
+
+### Approach
+
+We will extend SProto in several ways:
+
+1. **Configuration Format**: Introduce a `sproto.yaml` configuration file format (similar to buf.yaml) - [Phase 1, Task 1.1](Phase1.md#task-11-design-and-implement-configuration-format)
+2. **Database Schema**: Extend the database schema to store import path mappings and dependency relationships - [Phase 1, Task 1.2](Phase1.md#task-12-update-database-schema)
+3. **CLI Enhancements**: Add dependency resolution capabilities to the CLI - [Phase 3](Phase3.md)
+4. **Caching Mechanism**: Implement a local cache for downloaded dependencies - [Phase 4](Phase4.md)
+5. **Import Resolution**: Create a system for mapping import paths to modules - [Phase 2, Task 2.3](Phase2.md#task-23-implement-import-path-mapping-system)
+
+We will leverage existing Go packages wherever possible to minimize reinventing the wheel, while ensuring backward compatibility with existing SProto installations.
+
+## Implementation Phases
+
+This enhancement is broken down into five major phases, each focusing on different aspects of the dependency management system:
+
+1. [**Phase 1: Configuration System and Schema Updates**](Phase1.md) - Laying the foundation with configuration format and database changes
+2. [**Phase 2: Dependency Resolution System**](Phase2.md) - Building the core dependency resolution logic
+3. [**Phase 3: CLI Enhancements**](Phase3.md) - Improving the command-line interface to support dependency management
+4. [**Phase 4: Caching and Directory Structure**](Phase4.md) - Implementing local caching for efficient dependency management
+5. [**Phase 5: Testing and Documentation**](Phase5.md) - Ensuring quality and usability through tests and comprehensive documentation