MitchellAcoustics · MitchellAcoustics · Apr 16, 2025 · Apr 16, 2025 · Apr 16, 2025 · Apr 16, 2025
diff --git a/.gitignore b/.gitignore
@@ -22,7 +22,11 @@ src/.DS_Store
 # in progress / trials
 scripts/
 
+# Generated demo outputs
+docs/demos/outputs/
+
 /.vscode/settings.json
 
 *.hdf5
-.coverage
+.coverage*
+site/
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,20 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Build/Test Commands
+- Install: `pip install -e .`
+- Run tests: `pytest` 
+- Run single test: `pytest tests/test_file.py::TestClass::test_function -v`
+- Run with coverage: `pytest --cov=src/cityseg --cov-report=term`
+
+## Code Style Guidelines
+- Follow PEP 8 for Python code style
+- Use type hints for all function parameters and return values
+- Import organization: standard library, third-party, local modules (alphabetically within groups)
+- Use docstrings for all modules, classes, and functions (including parameters and return values)
+- Error handling: Use custom exceptions from exceptions.py
+- Naming: snake_case for variables/functions, PascalCase for classes
+- Path handling: Use pathlib.Path instead of strings
+- Logging: Use loguru.logger with appropriate levels
+- Testing: Use pytest fixtures and parametrize for test cases
diff --git a/README.md b/README.md
@@ -2,6 +2,8 @@
 
 CitySeg is a flexible and efficient semantic segmentation pipeline for processing images and videos of urban environments. It supports multiple segmentation models and datasets, with capabilities for processing high-resolution inputs and comprehensive result analysis.
 
+> **Version 0.4.0** introduces a complete reorganization to a modular component-based architecture, with enhanced testing infrastructure and improved storage mechanisms. The legacy interface remains available for backward compatibility. See the [Changelog](docs/changelog.md) for details.
+
 ## Features
 
 - Support for multiple segmentation models (OneFormer)
@@ -10,8 +12,10 @@ CitySeg is a flexible and efficient semantic segmentation pipeline for processin
 - Comprehensive analysis of segmentation results, including category-wise statistics
 - Support for both image and video inputs
 - Multi-video processing capability for entire directories
-- Caching of processed segmentation maps in HDF5 format for quick re-analysis
-- Output includes segmentation maps, colored segmentations, overlay visualizations, and detailed CSV reports
+- Efficient storage using Zarr for segmentation data and Parquet for analysis results
+- Intelligent caching and workflow management with Hamilton
+- Output includes segmentation maps, colored segmentations, overlay visualizations
+- Proper resource management for videos and memory-efficient processing
 - Configurable logging with console and file outputs
 
 ## Installation
@@ -40,8 +44,11 @@ CitySeg requires the following main packages:
 - opencv-python (cv2)
 - numpy
 - Pillow (PIL)
-- h5py
 - pandas
+- xarray
+- zarr
+- pyarrow
+- hamilton
 - tqdm
 - pyyaml
 - loguru
@@ -61,9 +68,32 @@ For a complete list of dependencies, please refer to the `pyproject.toml` file.
    Optional arguments:
    - `--log-level`: Set the logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
    - `--verbose`: Enable verbose logging
+   - `--cache-dir`: Directory to use for caching intermediate results
+   - `--disable-cache`: Disable caching of intermediate results
 
 3. The pipeline will process the input and generate various outputs including segmentation maps, visualizations, and analysis reports.
 
+## Storage and Caching
+
+CitySeg uses modern data formats for efficient storage and retrieval:
+
+- **Zarr** for n-dimensional segmentation data
+  - Chunked storage for efficient access to video frames
+  - Compressed storage to reduce disk usage
+  - Metadata embedded directly with the data
+
+- **Parquet** for tabular analysis results
+  - Highly efficient columnar storage
+  - Advanced filtering and querying capabilities
+  - Great performance for analytical workloads
+
+- **Hamilton** for workflow management
+  - Automatic caching of intermediate results
+  - Precise dependency tracking between steps
+  - Function-level granularity for optimal reuse
+
+These improvements significantly reduce memory usage and improve performance, especially for large video files and when reprocessing with the same configuration.
+
 ## Configuration
 
 Here's an example of a basic configuration file:
@@ -113,21 +143,32 @@ Confirmed not to work due to issues with the Hugging Face pipeline:
 
 ## Project Structure
 
-The project is organized into several Python modules, each serving a specific purpose within the CitySeg pipeline:
+The project has been reorganized into a modular component-based architecture for better maintainability and extensibility:
 
 - `main.py`: Entry point of the application, responsible for initializing and running the segmentation pipeline.
-- `config.py`: Defines configuration classes and handles loading and validating configuration settings.
-- `pipeline.py`: Implements the core segmentation pipeline, including model loading and inference.
-- `processors.py`: Contains classes for processing images, videos, and directories, managing the segmentation workflow.
-- `segmentation_analyzer.py`: Provides functionality for analyzing segmentation results, including computing statistics and generating reports.
-- `video_file_iterator.py`: Implements an iterator for efficiently processing multiple video files in a directory.
-- `visualization_handler.py`: Handles the visualization of segmentation results using color palettes.
-- `file_handler.py`: Manages file operations related to saving and loading segmentation data and metadata.
-- `utils.py`: Provides utility functions for various tasks, including data handling and logging.
-- `palettes.py`: Defines color palettes for different datasets used in segmentation.
-- `exceptions.py`: Custom exception classes for error handling throughout the pipeline.
-
-This modular structure allows for easy maintenance and extension of the CitySeg pipeline, facilitating the addition of new features and models.
+- `core/`: Core functionality and configuration:
+  - `config.py`: Configuration classes and validation
+  - `exceptions.py`: Custom exception classes for error handling
+- `components/`: Modular components implementing core functionality:
+  - `image.py`: Image processing operations
+  - `video.py`: Video handling and frame extraction
+  - `segmentation.py`: Segmentation model integration
+  - `dataset.py`: Dataset handling and management
+  - `pipeline.py`: Pipeline creation and coordination
+- `analysis/`: Analysis and visualization:
+  - `analyzer.py`: Segmentation analysis and metrics
+  - `visualization.py`: Visualization of segmentation results
+- `storage/`: Data storage and retrieval:
+  - `storage.py`: Zarr and Parquet storage adapters
+- `utils/`: Utility functions:
+  - `common.py`: Common utility functions
+  - `palettes.py`: Color palettes for different datasets
+- `workflow/`: Hamilton-based workflow engine:
+  - `hamilton.py`: Workflow definitions and execution
+- `legacy/`: Backwards compatibility adapters:
+  - `processors.py`: Legacy processor interface
+
+This modular structure allows for easy maintenance and extension of the CitySeg pipeline, with clear separation of concerns between components.
 
 ## Logging
 

diff --git a/docs/api/analysis/analyzer.md b/docs/api/analysis/analyzer.md
@@ -0,0 +1,6 @@
+# Segmentation Analyzer
+
+::: cityseg.analysis.analyzer
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/analysis/visualization.md b/docs/api/analysis/visualization.md
@@ -0,0 +1,6 @@
+# Visualization Handler
+
+::: cityseg.analysis.visualization
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/components/dataset.md b/docs/api/components/dataset.md
@@ -0,0 +1,6 @@
+# Dataset Component
+
+::: cityseg.components.dataset
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/components/image.md b/docs/api/components/image.md
@@ -0,0 +1,6 @@
+# Image Component
+
+::: cityseg.components.image
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/components/pipeline.md b/docs/api/components/pipeline.md
@@ -0,0 +1,6 @@
+# Pipeline Component
+
+::: cityseg.components.pipeline
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/components/segmentation.md b/docs/api/components/segmentation.md
@@ -0,0 +1,6 @@
+# Segmentation Component
+
+::: cityseg.components.segmentation
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/components/video.md b/docs/api/components/video.md
@@ -0,0 +1,6 @@
+# Video Component
+
+::: cityseg.components.video
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/config.md b/docs/api/config.md
diff --git a/docs/api/core/config.md b/docs/api/core/config.md
@@ -0,0 +1,6 @@
+# Config Module
+
+::: cityseg.core.config
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/core/exceptions.md b/docs/api/core/exceptions.md
@@ -0,0 +1,6 @@
+# Exceptions Module
+
+::: cityseg.core.exceptions
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/exceptions.md b/docs/api/exceptions.md
diff --git a/docs/api/handlers.md b/docs/api/handlers.md
diff --git a/docs/api/legacy/processors.md b/docs/api/legacy/processors.md
@@ -0,0 +1,6 @@
+# Legacy Processors
+
+::: cityseg.legacy.processors
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/palettes.md b/docs/api/palettes.md
diff --git a/docs/api/pipeline.md b/docs/api/pipeline.md
diff --git a/docs/api/processors.md b/docs/api/processors.md
diff --git a/docs/api/segments_analysis.md b/docs/api/segments_analysis.md
diff --git a/docs/api/storage/storage.md b/docs/api/storage/storage.md
@@ -0,0 +1,6 @@
+# Storage Module
+
+::: cityseg.storage.storage
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/utils.md b/docs/api/utils.md
diff --git a/docs/api/utils/common.md b/docs/api/utils/common.md
@@ -0,0 +1,6 @@
+# Common Utilities
+
+::: cityseg.utils.common
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/utils/palettes.md b/docs/api/utils/palettes.md
@@ -0,0 +1,6 @@
+# Palettes Module
+
+::: cityseg.utils.palettes
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/api/workflow/hamilton.md b/docs/api/workflow/hamilton.md
@@ -0,0 +1,6 @@
+# Hamilton Workflow
+
+::: cityseg.workflow.hamilton
+    options:
+      members: true
+      parameter_headings: true
diff --git a/docs/changelog.md b/docs/changelog.md
@@ -5,7 +5,29 @@ All notable changes to CitySeg will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.4.0] - Development
+
+### Added
+- Comprehensive test fixtures system for reproducible testing
+- Integration tests focused on direct component interactions
+- Test helpers for common assertions and data generation
+- New modular component-based architecture
+
+### Changed
+- Complete codebase reorganization into modular structure
+- Refactored into component-based architecture with clear separation of concerns
+- Moved configuration and exceptions to core/ directory
+- Split processing functionality into specialized components
+- Improved legacy adapter layer for backward compatibility
+- Simplified test structure with focused integration tests
+- Enhanced storage layer with Zarr and Parquet adapters
+- Improved video resource management
+
+### Removed
+- Deprecated monolithic processor classes (moved to legacy namespace)
+- Redundant test files (archived for reference)
+
+## [0.3.1] - 2024-04-15
 
 ### Added
 - Initial release of CitySeg

diff --git a/docs/demos/README.md b/docs/demos/README.md
@@ -0,0 +1,35 @@
+# CitySeg Demos
+
+This directory contains demo scripts and notebooks that showcase how to use CitySeg.
+
+## Available Demos
+
+- `component_demo.ipynb` - Demonstrates the component-based architecture introduced in CitySeg 0.4.0
+
+## Example Inputs
+
+The `example_inputs/` directory contains sample images and videos for use with the demos:
+
+- `EustonTap-Screenshot1.png` - Sample street scene image
+- `CaledonianPark1_15s_3840x2160.mov` - Sample street scene video (15 seconds)
+
+## Running the Demos
+
+The Python scripts can be run directly:
+
+```bash
+cd docs/demos
+python component_demo.ipynb
+```
+
+Or converted to Jupyter notebooks using [jupytext](https://github.com/mwouts/jupytext):
+
+```bash
+pip install jupytext
+jupytext --to notebook component_demo.ipynb
+jupyter notebook component_demo.ipynb
+```
+
+## Note on Output Directory
+
+The demos will create an `outputs/` directory to store generated visualizations and results.