WIP: feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline

deployment/README.md

@@ -0,0 +1,74 @@
+# AWML Deployment Framework
+
+AWML ships a unified, task-agnostic deployment stack that turns trained PyTorch checkpoints into production-ready ONNX and TensorRT artifacts. The verification and evaluation toolchain runs across every backend, ensuring numerical parity and consistent metrics across different projects.
+
+At the center is a shared runner/pipeline/exporter architecture that teams can extend with lightweight wrappers or workflows. CenterPoint, YOLOX, CalibrationStatusClassification, and future models plug into the same export and verification flow while still layering in task-specific logic where needed.
+
+
+## Quick Start
+
+```bash
+# Deployment entrypoint
+python -m deployment.cli.main <project> <deploy_cfg.py> <model_cfg.py> [project-specific args]
+
+# Example: CenterPoint deployment
+python -m deployment.cli.main centerpoint <deploy_cfg.py> <model_cfg.py> --rot-y-axis-reference
+```
+
+## Documentation Map
+
+| Topic | Description |
+| --- | --- |
+| [`docs/overview.md`](docs/overview.md) | Design principles, key features, precision policies. |
+| [`docs/architecture.md`](docs/architecture.md) | Workflow diagram, core components, file layout. |
+| [`docs/usage.md`](docs/usage.md) | CLI usage, runner patterns, typed contexts, export modes. |
+| [`docs/configuration.md`](docs/configuration.md) | Config structure, typed schemas, backend enums. |
+| [`docs/projects.md`](docs/projects.md) | CenterPoint, YOLOX, and Calibration deployment specifics. |
+| [`docs/export_pipeline.md`](docs/export_pipeline.md) | ONNX/TRT export steps and pipeline patterns. |
+| [`docs/verification_evaluation.md`](docs/verification_evaluation.md) | Verification scenarios, evaluation metrics, core contract. |
+| [`docs/best_practices.md`](docs/best_practices.md) | Best practices, troubleshooting, roadmap. |
+| [`docs/contributing.md`](docs/contributing.md) | How to add new deployment projects end-to-end. |
+
+Refer to `deployment/docs/README.md` for the same index.
+
+## Architecture Snapshot
+
+- **Entry point** (`deployment/cli/main.py`) loads a project bundle from `deployment/projects/<project>/`.
+- **Runtime** (`deployment/runtime/*`) coordinates load → export → verify → evaluate via shared orchestrators.
+- **Exporters** live under `exporters/common/` with typed config classes; project wrappers/pipelines compose the base exporters as needed.
+- **Pipelines** are registered by each project bundle and resolved via `PipelineFactory`.
+- **Core package** (`core/`) supplies typed configs, runtime contexts, task definitions, and shared verification utilities.
+
+See [`docs/architecture.md`](docs/architecture.md) for diagrams and component details.
+
+## Export & Verification Flow
+
+1. Load the PyTorch checkpoint and run ONNX export (single or multi-file) using the injected wrappers/pipelines.
+2. Optionally build TensorRT engines with precision policies such as `auto`, `fp16`, `fp32_tf32`, or `strongly_typed`.
+3. Register artifacts via `ArtifactManager` for downstream verification and evaluation.
+4. Run verification scenarios defined in config—pipelines are resolved by backend and device, and outputs are recursively compared with typed tolerances.
+5. Execute evaluation across enabled backends and emit typed metrics.
+
+Implementation details live in [`docs/export_pipeline.md`](docs/export_pipeline.md) and [`docs/verification_evaluation.md`](docs/verification_evaluation.md).
+
+## Project Coverage
+
+- **CenterPoint** – multi-file export orchestrated by dedicated ONNX/TRT pipelines; see [`docs/projects.md`](docs/projects.md).
+- **YOLOX** – single-file export with output reshaping via `YOLOXOptElanONNXWrapper`.
+- **CalibrationStatusClassification** – binary classification deployment with identity wrappers and simplified pipelines.
+
+Each project ships its own deployment bundle under `deployment/projects/<project>/`.
+
+## Core Contract
+
+[`core_contract.md`](docs/core_contract.md) defines the boundaries between runners, orchestrators, evaluators, pipelines, and metrics interfaces. Follow the contract when introducing new logic to keep refactors safe and dependencies explicit.
+
+## Contributing & Best Practices
+
+- Start with [`docs/contributing.md`](docs/contributing.md) for the required files and patterns when adding a new deployment project.
+- Consult [`docs/best_practices.md`](docs/best_practices.md) for export patterns, troubleshooting tips, and roadmap items.
+- Keep documentation for project-specific quirks in the appropriate file under `deployment/docs/`.
+
+## License
+
+See LICENSE at the repository root.

deployment/__init__.py

@@ -0,0 +1,24 @@
+"""
+Autoware ML Unified Deployment Framework
+
+This package provides a unified, task-agnostic deployment framework for
+exporting, verifying, and evaluating machine learning models across different
+tasks (classification, detection, segmentation, etc.) and backends (ONNX,
+TensorRT).
+"""
+
+from deployment.core.config.base_config import BaseDeploymentConfig
+from deployment.core.evaluation.base_evaluator import BaseEvaluator
+from deployment.core.io.base_data_loader import BaseDataLoader
+from deployment.core.io.preprocessing_builder import build_preprocessing_pipeline
+from deployment.runtime.runner import BaseDeploymentRunner
+
+__all__ = [
+    "BaseDeploymentConfig",
+    "BaseDataLoader",
+    "BaseEvaluator",
+    "BaseDeploymentRunner",
+    "build_preprocessing_pipeline",
+]
+
+__version__ = "1.0.0"

deployment/cli/__init__.py

@@ -0,0 +1 @@
+"""Deployment CLI package."""

deployment/cli/main.py

@@ -0,0 +1,101 @@
+"""
+Single deployment entrypoint.
+
+Usage:
+    python -m deployment.cli.main <project> <deploy_cfg.py> <model_cfg.py> [project-specific args]
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+import pkgutil
+import sys
+import traceback
+from typing import List
+
+import deployment.projects as projects_pkg
+from deployment.core.config.base_config import parse_base_args
+from deployment.projects import project_registry
+
+
+def _discover_project_packages() -> List[str]:
+    """Discover project package names under deployment.projects (without importing them)."""
+
+    names: List[str] = []
+    for mod in pkgutil.iter_modules(projects_pkg.__path__):
+        if not mod.ispkg:
+            continue
+        if mod.name.startswith("_"):
+            continue
+        names.append(mod.name)
+    return sorted(names)
+
+
+def _import_and_register_project(project_name: str) -> None:
+    """Import project package, which should register itself into project_registry."""
+    importlib.import_module(f"deployment.projects.{project_name}")
+
+
+def build_parser() -> argparse.ArgumentParser:
+    """Build the unified deployment CLI parser.
+
+    This discovers `deployment.projects.<name>` bundles, imports them to trigger
+    registration into `deployment.projects.project_registry`, then creates a
+    subcommand per registered project.
+    """
+    parser = argparse.ArgumentParser(
+        description="AWML Deployment CLI",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+
+    subparsers = parser.add_subparsers(dest="project", required=True)
+
+    # Discover projects and import them so they can contribute args.
+    failed_projects: List[str] = []
+    for project_name in _discover_project_packages():
+        try:
+            _import_and_register_project(project_name)
+        except Exception as e:
+            tb = traceback.format_exc()
+            failed_projects.append(f"- {project_name}: {e}\n{tb}")
+            continue
+
+        try:
+            adapter = project_registry.get(project_name)
+        except KeyError:
+            continue
+
+        sub = subparsers.add_parser(project_name, help=f"{project_name} deployment")
+        parse_base_args(sub)  # adds deploy_cfg, model_cfg, --log-level
+        adapter.add_args(sub)
+        sub.set_defaults(_adapter_name=project_name)
+
+    if not project_registry.list_projects():
+        details = "\n".join(failed_projects) if failed_projects else "(no project packages discovered)"
+        raise RuntimeError(
+            "No deployment projects were registered. This usually means project imports failed.\n" f"{details}"
+        )
+
+    return parser
+
+
+def main(argv: List[str] | None = None) -> int:
+    """CLI entrypoint.
+
+    Args:
+        argv: Optional argv list (without program name). If None, uses `sys.argv[1:]`.
+
+    Returns:
+        Process exit code (0 for success).
+    """
+    argv = sys.argv[1:] if argv is None else argv
+    parser = build_parser()
+    args = parser.parse_args(argv)
+
+    adapter = project_registry.get(args._adapter_name)
+    return int(adapter.run(args) or 0)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

deployment/core/__init__.py

@@ -0,0 +1,103 @@
+"""Core components for deployment framework."""
+
+from deployment.core.artifacts import (
+    Artifact,
+    get_component_files,
+    resolve_artifact_path,
+    resolve_engine_path,
+    resolve_onnx_path,
+)
+from deployment.core.backend import Backend
+from deployment.core.config.base_config import (
+    BaseDeploymentConfig,
+    DeviceConfig,
+    EvaluationConfig,
+    ExportConfig,
+    ExportMode,
+    RuntimeConfig,
+    TensorRTConfig,
+    VerificationConfig,
+    VerificationScenario,
+    parse_base_args,
+    setup_logging,
+)
+from deployment.core.contexts import (
+    CalibrationExportContext,
+    CenterPointExportContext,
+    ExportContext,
+    YOLOXExportContext,
+)
+from deployment.core.evaluation.base_evaluator import (
+    EVALUATION_DEFAULTS,
+    BaseEvaluator,
+    EvalResultDict,
+    EvaluationDefaults,
+    InferenceInput,
+    ModelSpec,
+    TaskProfile,
+    VerifyResultDict,
+)
+from deployment.core.evaluation.verification_mixin import VerificationMixin
+from deployment.core.io.base_data_loader import BaseDataLoader
+from deployment.core.io.preprocessing_builder import build_preprocessing_pipeline
+from deployment.core.metrics import (
+    BaseMetricsConfig,
+    BaseMetricsInterface,
+    ClassificationMetricsConfig,
+    ClassificationMetricsInterface,
+    Detection2DMetricsConfig,
+    Detection2DMetricsInterface,
+    Detection3DMetricsConfig,
+    Detection3DMetricsInterface,
+)
+
+__all__ = [
+    # Backend and configuration
+    "Backend",
+    # Typed contexts
+    "ExportContext",
+    "YOLOXExportContext",
+    "CenterPointExportContext",
+    "CalibrationExportContext",
+    "BaseDeploymentConfig",
+    "ExportConfig",
+    "ExportMode",
+    "RuntimeConfig",
+    "TensorRTConfig",
+    "DeviceConfig",
+    "EvaluationConfig",
+    "VerificationConfig",
+    "VerificationScenario",
+    "setup_logging",
+    "parse_base_args",
+    # Constants
+    "EVALUATION_DEFAULTS",
+    "EvaluationDefaults",
+    # Data loading
+    "BaseDataLoader",
+    # Evaluation
+    "BaseEvaluator",
+    "TaskProfile",
+    "InferenceInput",
+    "EvalResultDict",
+    "VerifyResultDict",
+    "VerificationMixin",
+    # Artifacts
+    "Artifact",
+    "resolve_artifact_path",
+    "resolve_onnx_path",
+    "resolve_engine_path",
+    "get_component_files",
+    "ModelSpec",
+    # Preprocessing
+    "build_preprocessing_pipeline",
+    # Metrics interfaces (using autoware_perception_evaluation)
+    "BaseMetricsInterface",
+    "BaseMetricsConfig",
+    "Detection3DMetricsInterface",
+    "Detection3DMetricsConfig",
+    "Detection2DMetricsInterface",
+    "Detection2DMetricsConfig",
+    "ClassificationMetricsInterface",
+    "ClassificationMetricsConfig",
+]