Implement Valkey/Redis exporter and tests for graph nodes & Appearance-Based ReID Tracking

.github/copilot-instructions.md

@@ -2,7 +2,7 @@
 
 ## Architecture Overview
 
-MATA is a **task-centric, model-agnostic** computer vision framework with a llama.cpp-inspired universal loader. As of v1.9.0, it features a unified adapter system supporting multiple tasks and runtimes, plus a fully vendored ByteTrack/BotSort tracking system and an OCR evaluation pipeline.
+MATA is a **task-centric, model-agnostic** computer vision framework with a llama.cpp-inspired universal loader. As of v1.9.2, it features a unified adapter system supporting multiple tasks and runtimes, a fully vendored ByteTrack/BotSort tracking system with appearance-based ReID (single-camera and cross-camera via Valkey), and an OCR evaluation pipeline.
 
 **Universal Loading (v1.5.2+):**
 
@@ -12,9 +12,11 @@ mata.load("classify", "./model.onnx")           # Local ONNX file
 mata.load("segment", "fast-model")              # Config alias
 mata.load("depth", "depth-anything/Depth-Anything-V2-Small-hf")
 mata.load("track", "facebook/detr-resnet-50", tracker="botsort")  # 🆕 v1.8.0
+mata.load("track", "facebook/detr-resnet-50", tracker="botsort",
+          reid_model="openai/clip-vit-base-patch32")             # 🆕 v1.9.2 ReID
 ```
 
-**Object Tracking (v1.8.0):**
+**Object Tracking (v1.8.0+):**
 
 ```python
 # One-liner video/stream tracking
@@ -31,6 +33,18 @@ for result in mata.track("rtsp://cam/stream",
 # Persistent per-frame tracking
 tracker = mata.load("track", "facebook/detr-resnet-50", tracker="bytetrack")
 result = tracker.update(frame, persist=True)  # YOLO-like pattern
+
+# Appearance-based ReID (v1.9.2+) — BotSort only
+results = mata.track("video.mp4",
+    model="facebook/detr-resnet-50",
+    reid_model="openai/clip-vit-base-patch32")
+
+# Cross-camera ReID via Valkey (v1.9.2+)
+from mata.trackers import ReIDBridge
+bridge = ReIDBridge("valkey://localhost:6379", camera_id="cam-1")
+results = mata.track("rtsp://cam/stream", model="...",
+                     reid_model="openai/clip-vit-base-patch32",
+                     reid_bridge=bridge, stream=True)
 ```
 
 **Zero-Shot Capabilities:**
@@ -56,18 +70,24 @@ Task Adapters (HuggingFace/ONNX/TorchScript/PyTorch)
 VisionResult (Unified result: bbox + mask + track_id + embedding)
     ↓
 Runtime (PyTorch/ONNX Runtime/TorchScript)
-    ↓           ↓              ↓
-Export System  Tracking Layer  Evaluation Layer (v1.8.1+)
-(JSON/CSV/     (v1.8.0)        ↓
-Image/Crops)   ↓               Validator (detect/segment/classify/depth/ocr)
-               TrackingAdapter ↓
-               ↓               Metrics (DetMetrics/SegMetrics/ClassifyMetrics/
-               Vendored Trackers         DepthMetrics/OCRMetrics)  ← v1.9.0
-               (BYTETracker/           ↓
-               BOTSORT)              Printer + DatasetLoader
-               ↓ (no external dep)   (COCO/COCO-Text JSON)
-               KalmanFilter + IoU
+    ↓           ↓                    ↓
+Export System  Tracking Layer         Evaluation Layer (v1.8.1+)
+(JSON/CSV/     (v1.8.0)               ↓
+Image/Crops)   ↓                      Validator (detect/segment/classify/depth/ocr)
+               TrackingAdapter        ↓
+               ↓                      Metrics (DetMetrics/SegMetrics/ClassifyMetrics/
+               Vendored Trackers                DepthMetrics/OCRMetrics)  ← v1.9.0
+               (BYTETracker/BOTSORT)           ↓
+               ↓ (no external dep)            Printer + DatasetLoader
+               KalmanFilter + IoU             (COCO/COCO-Text JSON)
                matching + GMC
+               ↓
+               ReID Layer (🆕 v1.9.2)
+               ↓           ↓
+               ReIDAdapter  ReIDBridge (cross-camera)
+               ↓           ↓
+               HuggingFace  Valkey embedding store
+               ONNX         (publish/query/TTL eviction)
 ```
 
 **Key Design Pattern:** Task contracts over model specifics - all adapters implement the same `predict()` interface returning task-specific results (VisionResult for detect/segment, ClassifyResult, DepthResult).
@@ -150,6 +170,11 @@ pytest tests/test_tracking_visualization.py -v # Visualization/export (103 tests
 pytest tests/test_video_io.py -v           # Video I/O utilities (56 tests)
 pytest tests/test_track_node.py -v         # Track graph node (39 tests)
 
+# ReID test suites (v1.9.2)
+pytest tests/test_reid_adapter.py -v       # ReID adapter unit tests (40+ tests)
+pytest tests/test_tracking_reid.py -v      # TrackingAdapter + API integration (25+ tests)
+pytest tests/test_reid_bridge.py -v        # Cross-camera bridge (15+ tests)
+
 # VLM tool-calling test suites (v1.7.0)
 pytest tests/test_tool_schema.py -v        # Tool schema (33 tests)
 pytest tests/test_tool_registry.py -v      # Tool registry (49 tests)

CHANGELOG.md

@@ -11,6 +11,73 @@ Versions follow [Semantic Versioning](https://semver.org/).
 
 ---
 
+## [1.9.2] Beta Release - 2026-03-09
+
+### Added
+
+**Valkey / Redis Graph Pipeline Storage**
+
+- `export_valkey(result, url, key, ttl, serializer)` — serializes any MATA result type to a Valkey or Redis key with optional TTL; supports `json` (default) and `msgpack` serializers
+- `load_valkey(url, key, result_type="auto")` — deserializes a stored result back to the original type; auto-detects `VisionResult`, `ClassifyResult`, `DepthResult`, and `OCRResult` from stored payload structure
+- `publish_valkey(result, url, channel, serializer)` — fire-and-forget Pub/Sub broadcast; returns subscriber count
+- `_parse_valkey_uri()` helper supporting `valkey://host:port/key`, `valkey://host:port/db/key`, and `redis://user:pass@host:port/db/key` formats
+- `ValkeyStore` graph sink node — pass-through sink that writes an artifact to Valkey during graph execution; supports `{node}` and `{timestamp}` key template placeholders, TTL, and serializer selection
+- `ValkeyLoad` graph source node — source node with `inputs={}` that loads a stored result from Valkey and injects it into the graph as a typed artifact
+- `valkey://` and `redis://` URI scheme dispatch added to all six `result.save()` methods (`VisionResult`, `DetectResult`, `SegmentResult`, `ClassifyResult`, `DepthResult`, `OCRResult`) — existing file-based paths are fully unaffected
+- `ModelRegistry.get_valkey_connection(name="default")` — reads named Valkey connection profiles from the `storage.valkey` section of `.mata/models.yaml` or `~/.mata/models.yaml`; resolves `password_env` from environment variables; raises `ModelNotFoundError` for unknown connection names
+- YAML `storage.valkey.<name>` config schema with `url`, `db`, `ttl`, `password_env`, and `tls` fields
+- Optional dependency groups: `pip install mata[valkey]` → `valkey>=6.0.0`; `pip install mata[redis]` → `redis>=5.0.0`; both added to the `dev` extras group
+- `export_valkey`, `load_valkey`, and `publish_valkey` exported from `mata.core.exporters`
+- `ValkeyStore` and `ValkeyLoad` exported from `mata.nodes`
+- 89 new tests: 42 exporter tests (`test_valkey_exporter.py`), 33 graph node tests (`test_valkey_nodes.py`), 14 config and pub/sub tests (`test_valkey_config.py`)
+
+**Documentation**
+
+- `docs/VALKEY_GUIDE.md` — full integration guide covering installation, basic usage, graph pipeline integration, YAML configuration, streaming patterns, Pub/Sub architecture, security (TLS, `password_env`, SSRF prevention, key sanitization), performance tuning (serializer choice, TTL strategies, connection pooling, async patterns), and top-5 troubleshooting issues
+- `docs/GRAPH_API_REFERENCE.md` — new "Storage Nodes" section with full parameter tables for `ValkeyStore` and `ValkeyLoad`
+- `README.md` — Valkey added to Key Features list and Optional Dependencies table
+- `QUICKSTART.md` — new "Valkey / Redis Result Storage" section with annotated code examples
+- `QUICK_REFERENCE.md` — new "Valkey/Redis Storage Quick Reference (v1.9)" section with cheatsheet
+
+**Appearance-Based ReID Tracking**
+
+- `mata.track(..., reid_model="org/model")` — activate appearance-based re-identification for BotSort by supplying any HuggingFace image encoder ID or local `.onnx` path
+- `ReIDAdapter` — abstract base class for appearance feature extractors; L2-normalised embedding output; lazy-loaded to keep startup cost zero when ReID is unused
+- `HuggingFaceReIDAdapter` — ViT / CLIP / AutoModel architecture auto-detection (CLIP image encoder, ViT/DeiT/Swin/BEiT pooler output, generic AutoModel mean-pooling); all `transformers` imports lazy
+- `ONNXReIDAdapter` — ONNX Runtime ReID extractor; auto-detects NCHW/NHWC input layout from model metadata; supports CPU and CUDA execution providers
+- `TrackingAdapter.update()` now extracts detection crops, batch-encodes them through the ReID encoder, and injects embeddings into `BOTSORT` — activating the appearance distance branch in `get_dists()`; `Instance.embedding` populated in output `VisionResult`
+- `mata.track()` extended with `reid_model: str | None` and `with_reid: bool = False` kwargs; `with_reid=True` without `reid_model` raises `ValueError`
+- Config alias support: `reid_model` and `with_reid` keys can be declared in `.mata/models.yaml` under a `track:` alias; runtime kwargs always take precedence
+- `ReIDBridge` — cross-camera appearance store backed by Valkey/Redis; publishes L2-normalised embeddings keyed by `reid:{camera_id}:{track_id}`; `query()` returns nearest matches above cosine-similarity threshold from other cameras; uses `scan_iter` (production-safe, non-blocking); TTL-based auto-eviction; `msgpack` binary serialisation
+- `TrackingAdapter.__init__()` extended with `reid_bridge: ReIDBridge | None`; after each `update()` confirmed tracks with embeddings are published automatically; `ConnectionError` caught and logged, never raised
+- `mata.track()` / `mata.load("track", ...)` extended with `reid_bridge` kwarg; forwarded to `TrackingAdapter`
+- `ReIDAdapter`, `HuggingFaceReIDAdapter`, `ONNXReIDAdapter` exported from `mata.adapters`
+- `ReIDBridge` exported from `mata.trackers`
+- `src/mata/trackers/configs/botsort.yaml` — commented `reid_model` / `with_reid` documentation block added (v1.9.2+)
+- 80+ new tests: `test_reid_adapter.py` (ReID adapter unit tests), `test_tracking_reid.py` (TrackingAdapter + API integration), `test_reid_bridge.py` (cross-camera bridge)
+- `examples/track/reid_tracking.py` — basic single-camera ReID tracking example script
+- `examples/track/cross_camera_reid.py` — cross-camera ReID via Valkey example script
+
+**Documentation**
+
+- `docs/VALKEY_GUIDE.md` — full integration guide covering installation, basic usage, graph pipeline integration, YAML configuration, streaming patterns, Pub/Sub architecture, security (TLS, `password_env`, SSRF prevention, key sanitization), performance tuning (serializer choice, TTL strategies, connection pooling, async patterns), and top-5 troubleshooting issues
+- `docs/GRAPH_API_REFERENCE.md` — new "Storage Nodes" section with full parameter tables for `ValkeyStore` and `ValkeyLoad`
+- `README.md` — Valkey added to Key Features list and Optional Dependencies table; ReID tracking section added with single-camera and cross-camera usage examples
+- `QUICKSTART.md` — new "Valkey / Redis Result Storage" section with annotated code examples
+- `QUICK_REFERENCE.md` — new "Valkey/Redis Storage Quick Reference (v1.9)" section with cheatsheet
+- `docs/VALIDATION_GUIDE.md` — ReID tracking validation notes added
+
+### Changed
+
+- `mata.nodes.__all__` extended with `ValkeyStore` and `ValkeyLoad`
+- `mata.core.exporters.__init__` extended with `export_valkey`, `load_valkey`, `publish_valkey`
+- `mata.track()` signature extended with `reid_model`, `with_reid`, `reid_bridge` kwargs (backward-compatible defaults)
+- `TrackingAdapter.__init__()` extended with `reid_encoder`, `reid_bridge` kwargs (both default to `None`; zero overhead when unused)
+- `BOTSORT.get_dists()` appearance-distance branch now reachable when `encoder` is set via `reid_encoder`
+- ByteTrack vs BotSort ReID comparison table in `README.md` updated to reflect v1.9.2 BotSort support
+
+---
+
 ## [1.9.1] - 2026-03-08
 
 ### Changed

QUICKSTART.md

@@ -366,6 +366,75 @@ print(f"mAP@50-95: {metrics.box.map:.3f}")
 All four tasks are supported — detection, segmentation, classification, and depth.
 See the [Validation Guide](docs/VALIDATION_GUIDE.md) for dataset setup, full API reference, and metrics details.
 
+## Valkey / Redis Result Storage
+
+Persist any MATA result to [Valkey](https://valkey.io/) or Redis for distributed pipelines, cross-process sharing, or caching.
+
+### Install
+
+```bash
+pip install mata[valkey]   # valkey-py (recommended)
+pip install mata[redis]    # redis-py (alternative)
+```
+
+### Save and load a result
+
+```bash
+# Start a local Valkey server (or use an existing Redis server)
+docker run -d -p 6379:6379 valkey/valkey:latest
+```
+
+```python
+import mata
+
+# Run detection and save result to Valkey
+result = mata.run("detect", "image.jpg", model="PekingU/rtdetr_r18vd")
+result.save("valkey://localhost:6379/detections:frame_001")
+
+# Load it back later (in a different process or service)
+from mata.core.exporters import load_valkey
+loaded = load_valkey(url="valkey://localhost:6379", key="detections:frame_001")
+print(loaded)  # equivalent VisionResult
+```
+
+### Use in a graph pipeline with `ValkeyStore` / `ValkeyLoad`
+
+```python
+import mata
+from mata.nodes import Detect, Filter, ValkeyStore, ValkeyLoad, Fuse
+from mata.core.graph import Graph
+
+detector = mata.load("detect", "PekingU/rtdetr_r18vd")
+
+# Pipeline A — detect and persist
+store_graph = (
+    Graph()
+    .then(Detect(using="detr", out="dets"))
+    .then(Filter(src="dets", score_gt=0.4, out="filtered"))
+    .then(ValkeyStore(
+        src="filtered",
+        url="valkey://localhost:6379",
+        key="pipeline:detections:{timestamp}",
+        ttl=60,  # expires after 60 s
+    ))
+)
+mata.infer("frame.jpg", graph=store_graph, providers={"detr": detector})
+
+# Pipeline B — load and annotate (in a separate service)
+load_graph = (
+    Graph()
+    .then(ValkeyLoad(
+        url="valkey://localhost:6379",
+        key="pipeline:detections:latest",
+        out="dets",
+    ))
+    .then(Fuse(detections="dets", out="annotated"))
+)
+result = mata.infer("frame.jpg", graph=load_graph, providers={})
+```
+
+See the [Graph API Reference](docs/GRAPH_API_REFERENCE.md#storage-nodes) for full parameter documentation.
+
 ## Next Steps
 
 1. **Read the full documentation**: [README.md](README.md)

QUICK_REFERENCE.md

@@ -15,6 +15,7 @@
 | [Object Tracking](#-object-tracking-quick-reference-v18) | v1.8 |
 | [OCR / Text Extraction](#-ocr--text-extraction-quick-reference-v19) | v1.9 |
 | [Evaluation](#-evaluation-quick-reference-v18) | v1.8 |
+| [Valkey/Redis Storage](#-valkeyredis-storage-quick-reference-v19) | v1.9 |
 
 ---
 
@@ -1583,7 +1584,117 @@ metrics = mata.val(
 
 ---
 
-**Version:** 1.8.1
-**Date:** February 20, 2026
+## 🗄️ Valkey/Redis Storage Quick Reference (v1.9)
+
+### Installation
+
+```bash
+pip install mata[valkey]   # valkey-py (recommended)
+pip install mata[redis]    # redis-py (alternative)
+```
+
+### `result.save()` — URI scheme
+
+```python
+# Any result type supports valkey:// and redis:// URIs directly in save()
+result.save("valkey://localhost:6379/my_key")          # basic
+result.save("valkey://localhost:6379/0/my_key")        # with DB number
+result.save("redis://localhost:6379/my_key")           # redis-py fallback
+result.save("valkey://localhost:6379/my_key", ttl=300) # with TTL (seconds)
+```
+
+### Low-level exporter
+
+```python
+from mata.core.exporters import export_valkey, load_valkey, publish_valkey
+
+# Store
+export_valkey(result, url="valkey://localhost:6379", key="my_key", ttl=3600)
+
+# Load
+loaded = load_valkey(url="valkey://localhost:6379", key="my_key")
+
+# Load with explicit result type (skip auto-detection)
+loaded = load_valkey(url="valkey://localhost:6379", key="my_key",
+                     result_type="vision")  # or "classify", "depth", "ocr"
+
+# Pub/Sub publish (fire-and-forget)
+n_receivers = publish_valkey(result, url="valkey://localhost:6379",
+                              channel="detections:stream")
+```
+
+### URI formats
+
+| Format | Example |
+| ------ | ------- |
+| Basic | `valkey://localhost:6379/key` |
+| With DB | `valkey://localhost:6379/0/key` |
+| With auth | `valkey://user:pass@host:6379/0/key` |
+| Redis | `redis://localhost:6379/key` |
+| Redis TLS | `rediss://host:6379/key` |
+
+### Graph nodes: `ValkeyStore` / `ValkeyLoad`
+
+```python
+from mata.nodes import ValkeyStore, ValkeyLoad
+
+# Sink node — store artifact and pass it through unchanged
+ValkeyStore(
+    src="filtered",                       # artifact name in graph context
+    url="valkey://localhost:6379",
+    key="pipeline:{node}:{timestamp}",    # {node} and {timestamp} placeholders
+    ttl=3600,                             # optional TTL in seconds
+    serializer="json",                    # "json" (default) or "msgpack"
+    out="filtered",                       # optional override for output name
+)
+
+# Source node — load artifact as graph entry point
+ValkeyLoad(
+    url="valkey://localhost:6379",
+    key="upstream:detections:latest",
+    result_type="auto",                   # or "vision", "classify", "depth", "ocr"
+    out="dets",
+)
+```
+
+### Auto-detection of result type
+
+| Key in stored data | Detected type | Output artifact |
+| ------------------ | ------------- | --------------- |
+| `instances` | `vision` | `Detections` |
+| `predictions` | `classify` | `Classifications` |
+| `depth` | `depth` | `DepthMap` |
+| `regions` | `ocr` | _(raw dict)_ |
+
+### Named connections (YAML config)
+
+```yaml
+# .mata/models.yaml
+storage:
+  valkey:
+    default:
+      url: "valkey://localhost:6379"
+      db: 0
+      ttl: 3600
+    production:
+      url: "valkey://prod-cluster:6379"
+      password_env: "VALKEY_PASSWORD"  # read from env, never stored in plaintext
+      tls: true
+```
+
+```python
+from mata.core.model_registry import ModelRegistry
+
+registry = ModelRegistry()
+conn = registry.get_valkey_connection("production")
+# → {"url": "valkey://...", "password": "<from env>", "tls": True}
+```
+
+**Documentation:** [Graph API Reference — Storage Nodes](docs/GRAPH_API_REFERENCE.md#storage-nodes)
+
+---
+
+**Version:** 1.9.0
+**Date:** March 9, 2026
 **Status:** ✅ Production Ready
 ````