Comprehensive Codebase Refactoring: Modularization, Test Coverage, and Shell Script Consolidation

[hessius]

Overview

This issue consolidates identified refactoring needs to improve maintainability, reduce technical debt, and ensure long-term sustainability of the MeticAI codebase. The analysis revealed several areas that would benefit from architectural improvements.

Critical Issues

1. Monolithic Main File (7,234 lines)

File: coffee-relay/main.py
Problem: The main application file contains 121 functions and 7,234 lines of code, making it extremely difficult to:

• Navigate and understand the codebase
• Test individual components in isolation
• Make changes without risk of unintended side effects
• Onboard new contributors

Recommended Split:

Module	Functions	Description
api/routes/coffee.py	~5	Coffee analysis endpoints
api/routes/history.py	~10	History CRUD operations
api/routes/shots.py	~15	Shot data and analysis
api/routes/profiles.py	~12	Profile management
api/routes/system.py	~8	Status, updates, settings
api/routes/scheduling.py	~10	Scheduled shots, recurring
services/gemini_service.py	~5	AI interaction logic
services/meticulous_service.py	~5	Machine API integration
services/cache_service.py	~15	All caching (shots, images, LLM)
services/analysis_service.py	~10	Shot analysis logic
utils/file_utils.py	~5	Atomic writes, JSON handling
utils/sanitization.py	~5	Input sanitization
models/schemas.py	~3	Pydantic models (ApplyImageRequest, etc.)

2. Low Test Coverage (25%)

Current State: main.py only has 25% test coverage (369/1504 statements)
Target: Aim for 70%+ coverage on critical paths

Gaps to Address:

• Shot analysis functions (_analyze_stage_execution, _perform_local_shot_analysis)
• Profile image generation and processing
• Scheduling system (recurring schedules, shot execution)
• Machine API interaction layer
• Cache management functions

3. Shell Script Code Duplication

Problem: 7 shell scripts duplicate the same patterns:

• Color definitions (GREEN, BLUE, RED, NC) in 7 files
• Error handling patterns
• Docker interaction commands
• Progress output functions

Solution: Create a shared library file:

# scripts/lib/common.sh
source_colors() { ... }
log_info() { ... }
log_error() { ... }
check_docker() { ... }
check_prerequisites() { ... }

Scripts affected:

• local-install.sh (1,681 lines)
• update.sh (1,100 lines)
• uninstall.sh (621 lines)
• rebuild-watcher.sh (531 lines)
• web_install.sh (384 lines)
• docker-up.sh (85 lines)
• check-updates-on-start.sh (45 lines)

Moderate Issues

4. Type Hints

File: coffee-relay/main.py
Problem: Line 3852 uses any type which bypasses type checking:

def _resolve_variable(value, variables: list) -> tuple[any, str | None]:

Solution: Replace with proper Union types or define a VariableValue type alias.

5. Missing API Documentation

Problem: Many endpoints lack comprehensive docstrings with:

• Full parameter descriptions
• Response schema examples
• Error response documentation

Files to Update:

• All route handlers in main.py
• Consider adding OpenAPI examples

6. Configuration Management

Problem: Configuration is scattered:

• Environment variables in .env
• Settings in data/settings.json
• Constants hardcoded in main.py (UPDATE_CHECK_INTERVAL, MAX_UPLOAD_SIZE)

Solution: Create a unified config module:

# config.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    update_check_interval: int = 7200
    max_upload_size: int = 10 * 1024 * 1024
    gemini_api_key: str
    meticulous_ip: str
    # ...

Minor Issues

7. Logging Inconsistency

Problem: Logger is initialized twice in main.py:

# Line 22-30: Custom setup_logging()
logger = setup_logging(log_dir=log_dir)

# Line 36-37: Basic config overrides it
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

8. Docker Compose Volume Mounts

Problem: 20+ volume mounts in docker-compose.yml for coffee-relay make it:

• Hard to understand what's needed
• Complex to maintain
• Potentially insecure (mounting .git, docker.sock)

Solution: Document each mount's purpose and consider reducing where possible.

9. Dependency Versions

File: coffee-relay/requirements.txt
Observation: Some packages lack version pins:

pyMeticulous>=0.1.0
zstandard>=0.22.0
httpx>=0.26.0

Recommendation: Pin to specific versions for reproducibility.

9. Naming

"Coffee-relay" was the project name before MeticAI was fully developed. We should change this to MeticAI-server to keep similar naming as "MeticAI-web".

Suggested Implementation Order

0. Rename

1. Phase 1 - Quick Wins

   • Fix type hints (any → proper types)
   • Fix logging initialization
   • Pin dependency versions
   • Create shared shell library for colors/logging

2. Phase 2 - Modularization

   • Extract route handlers into separate files
   • Extract service layers
   • Extract utility functions
   • Update imports throughout

3. Phase 3 - Test Coverage

   • Add tests for extracted modules
   • Target 70% coverage on critical paths
   • Add integration tests for new module boundaries

4. Phase 4 - Documentation

   • Add comprehensive docstrings
   • Update API.md with new structure
   • Document configuration options

Related Files

• coffee-relay/main.py (7,234 lines)
• coffee-relay/test_main.py (7,362 lines)
• local-install.sh (1,681 lines)
• update.sh (1,100 lines)
• All shell scripts in root directory

Labels

• refactoring
• technical-debt
• maintainability
• good-first-issue (for Phase 1 items)