You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -4,153 +4,133 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
4
4
5
5
## Project Overview
6
6
7
-
This is a Python implementation of the Pipeline pattern, distributed as `thecodecrate-pipeline`. The package allows composing sequential stages into reusable, immutable pipelines for processing data. It's inspired by the PHP League Pipeline package.
7
+
This is `thecodecrate-pipeline`, a Python package that provides a pipeline pattern implementation for data processing. Pipelines allow chaining multiple stages together, where each stage transforms a payload before passing it to the next stage.
8
8
9
-
## Development Commands
9
+
**Key Concepts:**
10
10
11
-
### Environment Setup
12
-
```bash
13
-
# The project uses uv for dependency management
14
-
# Dependencies are managed in pyproject.toml under [dependency-groups]
15
-
uv sync
16
-
```
11
+
-**Pipeline**: Orchestrates the execution of multiple stages using a processor
12
+
-**Stage**: A callable unit that transforms a payload (input → output)
13
+
-**Processor**: Defines how stages are executed (e.g., chained, interruptible)
14
+
-**PipelineFactory**: Creates pipeline instances with predefined stages and processors
17
15
18
-
### Testing
19
-
```bash
20
-
# Run all tests
21
-
pytest
16
+
## Architecture
22
17
23
-
# Run tests with coverage
24
-
pytest --cov
18
+
### Code Organization
25
19
26
-
# Run a specific test file
27
-
pytest tests/test_pipeline.py
20
+
The codebase follows a concern-based architecture with clear separation between contracts, concerns, and implementations:
**Concerns Pattern**: The Pipeline class inherits from multiple concern classes (BasePipeline, ProcessablePipeline, StageablePipeline), each providing specific functionality. This follows a composition-over-inheritance approach where each concern is responsible for a single aspect:
40
37
41
-
# Fix auto-fixable linting issues
42
-
ruff check --fix .
38
+
-`BasePipeline`: Core cloning and factory behavior
39
+
-`ProcessablePipeline`: Handles processor management and execution (src/_lib/concerns/processable_pipeline.py)
40
+
-`StageablePipeline`: Manages stage collection and instantiation (src/_lib/concerns/stageable_pipeline.py)
43
41
44
-
# Format with black (line length: 79)
45
-
black .
42
+
**Contracts (Protocols)**: All contracts are defined using Python's `Protocol` type for structural subtyping. Implementations explicitly declare they implement these protocols via inheritance.
46
43
47
-
# Type checking is configured with strict mode in pyrightconfig.json
48
-
# Type check manually: pyright (if installed)
49
-
```
44
+
**Clonable Pattern**: Most classes inherit from `Clonable` (src/_lib/support/clonable/clonable.py), which provides immutable-style operations using deep copying. Methods like `pipe()`, `with_stages()`, and `with_processor()` return cloned instances rather than mutating the original.
50
45
51
-
### Documentation
52
-
```bash
53
-
# Build documentation locally
54
-
mkdocs serve
46
+
**ActAsFactory Pattern**: The `PipelineFactory` uses this pattern (src/_lib/support/act_as_factory/) to create pipeline instances with predefined configuration.
55
47
56
-
# Documentation is built with mkdocs-material and auto-generates API docs
57
-
# from docstrings using mkdocstrings-python
58
-
```
48
+
### Type System
59
49
60
-
### Version Management
61
-
```bash
62
-
# Bump version (uses bumpver)
63
-
bumpver update --patch # 1.26.0 -> 1.26.1
64
-
bumpver update --minor # 1.26.0 -> 1.27.0
65
-
bumpver update --major # 1.26.0 -> 2.0.0
50
+
The codebase is fully typed using generic types `T_in` and `T_out` for input/output payloads. All classes are generic over these types to ensure type safety through the pipeline chain.
66
51
67
-
# Note: bumpver automatically commits and tags, but does NOT push
68
-
```
52
+
## Development Commands
69
53
70
-
##Architecture
54
+
### Environment Setup
71
55
72
-
### Core Concepts
56
+
```bash
57
+
# Install uv package manager if not available
58
+
uv python install 3.13
59
+
uv sync --all-extras --dev
60
+
```
73
61
74
-
The codebase implements a pipeline pattern with three main abstractions:
62
+
### Testing
75
63
76
-
1.**Stage**: A callable unit that transforms input to output (`StageInterface[T_in, T_out]`)
77
-
2.**Pipeline**: An immutable chain of stages (`PipelineInterface[T_in, T_out]`)
78
-
3.**Processor**: Controls how stages are executed (`ProcessorInterface[T_in, T_out]`)
64
+
```bash
65
+
# Run all tests with coverage
66
+
uv run pytest tests --cov
79
67
80
-
### Directory Structure
68
+
# Run a specific test file
69
+
uv run pytest tests/test_pipeline.py
81
70
82
-
```
83
-
src/
84
-
├── _lib/ # Internal implementation
85
-
│ ├── pipeline/ # Core pipeline implementation
86
-
│ │ ├── pipeline.py # Main Pipeline class
87
-
│ │ ├── pipeline_factory.py # Factory for building pipelines
│ ├── chained_processor/ # Processor that chains stages
96
-
│ └── interruptible_processor/ # Processor with interruption support
97
-
└── thecodecrate_pipeline/ # Public API package
98
-
├── __init__.py # Re-exports from _lib
99
-
├── processors/ # Public processor exports
100
-
└── types/ # Public type exports
71
+
# Run a specific test
72
+
uv run pytest tests/test_pipeline.py::test_lambda_stages -v
101
73
```
102
74
103
-
### Key Design Patterns
75
+
### Linting & Formatting
76
+
77
+
```bash
78
+
# Check linting
79
+
uvx ruff check .
104
80
105
-
**Immutability**: Pipelines use copy-on-write semantics. The `pipe()` method creates a new pipeline instance with the added stage, preserving the original pipeline.
81
+
# Fix linting issues automatically
82
+
uvx ruff check --fix .
106
83
107
-
**Traits System**: The codebase uses a trait-like pattern with mixins:
108
-
-`Clonable`: Provides shallow cloning capability
109
-
-`ActAsFactory`: Enables objects to act as factories for creating instances
84
+
# Format code
85
+
uvx ruff format .
86
+
```
110
87
111
-
**Interface Segregation**: Each core concept has an interface (`*Interface`) and implementation, enabling custom implementations while maintaining type safety.
88
+
### Version Bumping
112
89
113
-
**Async-First**: All processing is async (`async def process(...)`). The processor handles both sync and async callables transparently using `inspect.isawaitable()`.
90
+
```bash
91
+
# Bump version (patch/minor/major)
92
+
uv run bumpver update --patch
93
+
uv run bumpver update --minor
94
+
uv run bumpver update --major
95
+
```
114
96
115
-
### Type System
97
+
Version is managed by bumpver and automatically updates:
116
98
117
-
The codebase uses generic type variables for type safety:
118
-
-`T_in`: Input type to a stage or pipeline
119
-
-`T_out`: Output type from a stage or pipeline (defaults to `T_in`)
99
+
-`pyproject.toml`
100
+
-`src/thecodecrate_pipeline/__init__.py`
120
101
121
-
Stages can transform types:
122
-
```python
123
-
Pipeline[int, str] # Takes int, returns str
124
-
StageInterface[int, int] # Takes int, returns int
125
-
```
102
+
### Documentation
126
103
127
-
### Processing Flow
104
+
```bash
105
+
# Build documentation locally
106
+
mkdocs serve
107
+
108
+
# Build static site
109
+
mkdocs build
110
+
```
128
111
129
-
1. A Pipeline is created with stages (either via `.pipe()` or declaratively)
130
-
2. When `.process(payload)` is called, the pipeline:
131
-
- Instantiates stages if needed (converts classes to instances)
132
-
- Delegates to the processor's `.process()` method
133
-
- The processor iterates through stages, passing output to next stage
134
-
3. Processors can customize execution (e.g., ChainedProcessor for error handling)
112
+
## Important Implementation Details
135
113
136
-
### Stream Processing
114
+
### Async Processing
137
115
138
-
Pipelines support processing `AsyncIterator` streams, allowing real-time data transformation where each stage can yield results consumed immediately by the next stage.
116
+
All pipeline processing is async. The `Pipeline.process()` method and all stage `__call__` methods are async. The base `Processor._call()` method (src/_lib/processor.py:37-52) handles both sync and async callables automatically using `inspect.isawaitable()`.
139
117
140
-
### PipelineFactory
118
+
### Stage Instantiation
141
119
142
-
Because pipelines are immutable, `PipelineFactory` provides mutable stage collection during composition. It builds the final immutable pipeline via `.build()`.
120
+
Stages can be provided as either classes or instances. The `StageablePipeline` concern automatically instantiates stage classes when needed (src/_lib/concerns/stageable_pipeline.py:67-72).
143
121
144
-
##Testing Notes
122
+
### Processor Types
145
123
146
-
- Test files use stub classes in `tests/stubs/` for consistent test fixtures
147
-
- Tests are async-aware (configured via `pytest.ini` with `pytest-asyncio`)
148
-
- Mock stages implement `StageInterface` for type safety
124
+
-`ChainedProcessor`: Default processor that executes stages sequentially (src/_lib/processors/chained_processor/)
125
+
-`InterruptibleProcessor`: Allows stages to interrupt the pipeline flow (src/_lib/processors/interruptible_processor/)
149
126
150
-
##Python Version
127
+
### Callable Invocation
151
128
152
-
Requires Python 3.13+ (specified in pyproject.toml).
129
+
The `Pipeline` class is callable and delegates to `process()` (src/_lib/concerns/processable_pipeline.py:52-62). The first parameter is positional-only to match the callable signature.
153
130
154
-
## Package Distribution
131
+
## Testing Guidelines
155
132
156
-
Built with `hatchling`. The wheel includes both `thecodecrate_pipeline` (public API) and `_api` packages (internal). The public package re-exports symbols from `_lib`.
133
+
- All async tests must be marked with `@pytest.mark.asyncio`
This package is completely transparent when dealing with exceptions. In no case will this package catch an exception or silence an error. Exceptions should be dealt with on a per-case basis, either inside a _stage_ or at the time the pipeline processes a payload.
0 commit comments