Refactor device layer: consolidate servers, direct disk writes, reduce latency

[pskeshu]

Note: This analysis is based on the agent branch.

cc @subindevs

Background

The device layer architecture has evolved organically and now has several inefficiencies that introduce latency and confusion. This issue documents the current state, problems, and provides a refactoring roadmap.

Why Current Design Exists (Benefits to Preserve)

1. Process Isolation - RPyC keeps application crashes from crashing hardware
2. Bluesky Templates - Plan generation provides safe scaffold for agentic access
3. Stage Limits - Ophyd devices enforce physical safety limits
4. Restricted MMCore Access - Agents don't get raw hardware access

---

Current Architecture

Startup Sequence (Production)

1. start_server.py      (RPyC server with MMCore on port 18861)
2. start_services.bat   (launches simple_server.py + sam_server.py)
3. launch_copilot.py    (orchestrator)

Components

Component	Port	Role
start_server.py	18861	RPyC MMCore server - initializes MMCore, creates RunEngine + devices, but unused by copilot
backend/simple_server.py	60610	HTTP microscope API - connects to MMCore via RPyC, creates separate RunEngine + devices, actually used by copilot
backend/sam_server.py	18862	RPyC SAM detection server
launch_copilot.py	-	Orchestrator - connects to simple_server (HTTP) and SAM server (RPyC)
queue_server_startup.py	-	Bluesky queueserver - unused in current startup

Data Flow (Current - Multiple Hops)

Physical Hardware
    ↓
MMCore (in start_server.py via RPyC)
    ↓
RPyC bridge to simple_server.py  ← HOP 1
    ↓
Ophyd Device (in simple_server.py)
    ↓
RunEngine executes plan
    ↓
JSON serialize numpy array
    ↓
HTTP response to copilot  ← HOP 2 (full data transfer)
    ↓
Copilot stores to DataStore
    ↓
VizServer retrieves from DataStore

---

Problems

1. RPyC/HTTP Latency for Large Image Stacks

• Volume acquisition returns entire numpy arrays through HTTP as JSON
• Pickle/JSON serialization + network transfer introduces significant latency
• Becomes worse with larger volumes (e.g., 200 slices × 2048 × 2048)

2. Duplicate Hardware Initialization

• start_server.py creates MMCore + RunEngine + devices
• backend/simple_server.py creates separate RunEngine + devices
• queue_server_startup.py creates yet another set
• Multiple independent instances, unclear which is canonical

3. Device Wiring Confusion

• start_server.py has RunEngine and devices but they're not used
• simple_server.py is the actual control server used by copilot
• Unclear which system controls hardware

4. Data Persistence Location

• Data flows: Control layer → HTTP → Client → DataStore
• Should be: Control layer → Direct disk write → Pass UID via HTTP
• Affects: DataStore, VizServer, ImageManager

---

Target Architecture

Startup (2 scripts)

1. start_device_layer.py   (all hardware: MMCore + devices + HTTP + SAM)
2. launch_copilot.py       (application layer)

Data Flow (Target - Direct Writes)

Physical Hardware
    ↓
MMCore (in start_device_layer.py)
    ↓
Ophyd Device + RunEngine
    ↓
Write volume to disk: {storage}/volumes/{uid}.tif
    ↓
HTTP response: {uid, path, shape, dtype, session_id}  ← METADATA ONLY
    ↓
Copilot registers UID in DataStore index
    ↓
VizServer reads from disk by UID

---

Refactoring Tasks

Task 1: Consolidate Hardware Control Layer into Single Script

Create start_device_layer.py that:

• Initializes MMCore directly
• Creates single set of Ophyd devices
• Creates single RunEngine
• Starts HTTP server for plan execution
• Starts SAM server (or integrates into same process)
• Uses HTTP only for application communication

Task 2: Direct Data Writing from Control Layer

• Write volumes directly to disk in the control layer
• Return UID + metadata instead of full numpy arrays
• Flat storage with session in metadata:

  {storage_path}/volumes/{uid}.tif
  

• Session association tracked in index/metadata, not file path
• Return in HTTP response: {uid, path, shape, dtype, session_id}
• Pass session_id from copilot to control layer when acquiring

Task 3: Update DataStore Integration

• Modify gently/core/data_store.py to index files written by control layer
• Implement register_external_file(uid, path, metadata) method
• Ensure lineage tracking still works with external writes

Task 4: Update VizServer

• Modify gently/visualization/server.py to serve data by UID
• Watch for new files or receive push notifications from control layer
• Ensure existing /api/images/{uid} endpoints work with new storage

Task 5: Update Copilot/Orchestrator

• Modify gently/agent/queue_server_client.py to work with UID-based responses
• Load data from disk when needed instead of receiving through HTTP
• Update ImageManager to handle new data flow

Task 6: Clean Up Unused Code

• Remove or document queue_server_startup.py if not used
• Clarify which server pattern is canonical
• Remove start_services.bat (no longer needed)

---

Files to Modify/Create

File	Change
NEW start_device_layer.py	Single script for device layer
start_server.py	Remove or merge
backend/simple_server.py	Merge, add direct disk writes
backend/sam_server.py	Merge or keep separate
start_services.bat	Remove
gently/core/data_store.py	Add register_external_file()
gently/visualization/server.py	Update to serve from new storage
gently/agent/queue_server_client.py	Handle UID-based responses
gently/agent/image_manager.py	Load from disk by UID
launch_copilot.py	Update connection to unified device layer

[subindevs]

Revised Architecture Proposal

After analyzing the agent branch architecture, I've evaluated the original proposal and suggest some modifications that improve on the core ideas.

Original Issues Confirmed ✅

Problem	Verified
Duplicate RunEngine/devices	✅ start_server.py and simple_server.py each create separate instances
Data over HTTP	✅ simple_server.py serializes numpy arrays to JSON via serialize_value()
Unused RPyC server	✅ Copilot uses HTTP, not the RPyC endpoint

---

Key Modification: Shared Memory Pool

Instead of disk-first writes, use a memory-first approach with async disk persistence:

┌────────────────────────────────────────────────────────────────┐
│                      Device Layer                               │
│                                                                 │
│  ┌──────────┐     ┌─────────────────────────────────────────┐  │
│  │ Acquire  │────▶│         SharedMemoryPool (2GB)          │  │
│  │ Volume   │     │  ┌───────┬───────┬───────┬─────────┐    │  │
│  └──────────┘     │  │ uid_1 │ uid_2 │ uid_3 │   ...   │    │  │
│                   │  │ 50MB  │ 80MB  │ 50MB  │         │    │  │
│                   │  └───────┴───────┴───────┴─────────┘    │  │
│                   │         ▲               │               │  │
│                   │         │ LRU evict     │ async write   │  │
│                   └─────────┼───────────────┼───────────────┘  │
│                             │               ▼                   │
│                   ┌─────────┴────┐   ┌──────────────┐          │
│                   │  Copilot/    │   │ Disk Storage │          │
│                   │  VizServer   │   │ {uid}.tif    │          │
│                   │  (zero-copy) │   └──────────────┘          │
│                   └──────────────┘                              │
└────────────────────────────────────────────────────────────────┘

HTTP Response: {uid, shape, dtype, session_id, in_memory, on_disk}

Benefits over Disk-First

Aspect	Disk-First (Original)	Memory-First (Revised)
Immediate access latency	Disk I/O	Zero-copy shared memory
Acquisition blocking	Blocked until write completes	Non-blocking (async write)
Hot data access	Always load from disk	In-memory if recent
Cross-process sharing	File-based	multiprocessing.shared_memory

---

Additional Features

1. Session Tracking (First-Class)

pool.store(uid, volume, session_id="session_2024_01_14", metadata={...})
pool.get_session_volumes("session_2024_01_14")  # -> [uid1, uid2, ...]
pool.list_sessions()  # -> [{session_id, volume_count, total_mb}, ...]
pool.delete_session("session_2024_01_14")

2. User-Configurable Cleanup Policy

@dataclass 
class CleanupPolicy:
    max_age_days: Optional[int] = None        # Delete files older than X days
    max_total_gb: Optional[float] = None      # Delete oldest when exceeding X GB
    keep_min_per_session: int = 0             # Always keep at least N per session
    auto_cleanup_enabled: bool = True         # Enable/disable auto cleanup

3. Crash Recovery

• Persistent index.json tracks all volumes and sessions
• On restart, index is loaded and reconciled with disk files
• No data loss if process crashes

---

Revised Task List

Task 1: Create SharedMemoryPool Module

• Implement SharedMemoryPool class in gently/core/memory_pool.py
• LRU eviction within 2GB budget
• Async disk writer thread (TIFF + zlib)
• Session tracking with session_id grouping
• Configurable CleanupPolicy
• Persistent index for crash recovery

Task 2: Consolidate Device Layer

• Create start_device_layer.py combining:
  • MMCore initialization
  • Single RunEngine + Ophyd devices
  • HTTP API (from simple_server.py)
  • SharedMemoryPool integration
• Keep SAM server separate (GPU memory isolation)
• Remove duplicate start_server.py initialization

Task 3: Update HTTP API

• Modify plan responses to return {uid, shape, dtype, session_id} only
• Add endpoints:
  • GET /api/volumes/{uid} - retrieve volume data
  • GET /api/volumes/{uid}/status - get status without data
  • GET /api/sessions - list sessions
  • GET /api/sessions/{id}/volumes - list volumes in session
  • DELETE /api/sessions/{id} - delete session
  • GET /api/storage/stats - storage statistics
  • POST /api/storage/cleanup - manual cleanup trigger
  • GET/PUT /api/storage/policy - cleanup policy

Task 4: Update Copilot Integration

• Modify queue_server_client.py to use UID-based retrieval
• Update image_manager.py to attach to shared memory
• Pass session_id when acquiring volumes

Task 5: Update VizServer

• Retrieve data via SharedMemoryPool (zero-copy when in memory)
• Use EventBus for new volume notifications (already exists)

Task 6: Cleanup

• Remove start_server.py (merged into device layer)
• Remove start_services.bat
• Update documentation

---

Configuration

# config.yml additions
storage:
  memory_budget_gb: 2
  storage_dir: "data/volumes"
  cleanup_policy:
    max_age_days: 30
    max_total_gb: 100
    keep_min_per_session: 5
    auto_cleanup_enabled: true

---

Summary

This revision keeps the core insight from the original proposal (UID-based references, no data over HTTP) while adding:

1. Shared memory for zero-copy cross-process access
2. Async disk writes to avoid blocking acquisition
3. LRU eviction to manage memory budget
4. Session tracking as a first-class feature
5. Configurable cleanup for disk space management
6. Crash recovery via persistent index

The complexity increase is well-encapsulated in the SharedMemoryPool class.

[pskeshu]

Great work on the revised architecture proposal, @subindevs! The memory-first approach with SharedMemoryPool is exactly what we need.

Data latency is critical for this application - acquired volumes need to be immediately available in the frontend. The zero-copy shared memory approach with async disk persistence addresses this perfectly.

I'm approving this direction. Please prioritize working on this implementation as early as possible.

Key points to ensure:

• Memory-first with 2GB SharedMemoryPool
• Async disk writes (non-blocking acquisition)
• LRU eviction for memory management
• Session tracking as first-class feature
• Crash recovery via persistent index
• Cross-system image sharing must not break - VizServer and other components need seamless access to volumes whether they're in shared memory or on disk

Looking forward to seeing this come together!

[subindevs]

@pskeshu sorry to divert from the main issue again, please check if this matches with your idea of your project

Project Restructuring Proposal

Following the approved SharedMemoryPool architecture, here's a proposal to restructure the project for reusability and clean separation of concerns.

Core Insight

gently should be a generic AI agent for microscopy — not tied to any specific hardware. The hardware-specific code moves to a separate backend package.

---

Architecture

                         gently
                   (AI agent, generic)
                   ┌─────────────────┐
                   │ Copilot logic    │
                   │ SAM client       │
                   │ VizServer        │
                   │ Experiment       │
                   │   orchestration  │
                   │                  │
                   │ MicroscopeBackend│ ◄── abstract interface
                   └────────▲─────────┘
                            │ implements
            ┌───────────────┼───────────────┐
            │               │               │
    dispim-control     pymmcore-plus     Bluesky/EPICS
     (our diSPIM)      (Micro-Manager)   (other setups)

---

gently — The AI Agent (this repo, refocused)

A pip-installable, backend-agnostic microscopy AI agent:

• Copilot / agent decision-making
• SAM integration (segmentation for ROI detection)
• VizServer / frontend
• Autofocus strategies
• Multi-position planning
• Experiment orchestration

Defines an abstract interface that any backend must implement:

# gently/interface.py
class MicroscopeBackend(Protocol):
    def acquire_volume(self, num_slices: int, ...) -> str: ...
    def get_volume(self, uid: str) -> np.ndarray: ...
    def move_stage(self, x: float, y: float, z: float): ...
    def get_position(self) -> dict: ...
    def snap_image(self) -> np.ndarray: ...
    def list_sessions(self) -> list: ...
    ...

No hardware dependencies. No MMCore, no Ophyd, no EPICS imports.

---

dispim-control (new repo) — Our Hardware Backend

Implements gently's MicroscopeBackend interface for the diSPIM:

• MMCore initialization + ASI Tiger controller
• Ophyd device classes (DiSPIMVolumeScanner, DiSPIMCamera, DiSPIMPiezo, etc.)
• RunEngine + Bluesky plans
• SharedMemoryPool (from this issue)
• Data persistence (TIFF, crash recovery index)
• Session tracking, cleanup policies

---

Usage (user's launcher script)

Each lab wires their backend to gently:

# run_dispim.py
from gently import Copilot
from dispim_control import DiSPIMBackend

backend = DiSPIMBackend(config="dispim.yml")
copilot = Copilot(backend=backend)
copilot.run()

Or with a different backend:

# run_micromanager.py
from gently import Copilot
from some_other_package import MicroManagerBackend

backend = MicroManagerBackend(config_file="mm_config.cfg")
copilot = Copilot(backend=backend)
copilot.run()

---

SAM — Standalone Microservice

• GPU-isolated (separate process)
• Simple API: send image, get segmentation mask
• Called by gently for decision-making

---

What Moves Where

Stays in gently	Moves to dispim-control
Copilot / AI agent logic	gently/devices.py
Abstract MicroscopeBackend interface	gently/plans.py
SAM client	start_server.py, simple_server.py
VizServer / frontend	hardware_triggered_scan.py
Experiment orchestration	SharedMemoryPool (new)
Autofocus strategies (generic)	MMCore config, calibration

---

Benefits

• gently becomes reusable — any microscope lab can use it with their own backend (Micro-Manager, EPICS, Bluesky, custom)
• Clean dependency tree — gently has no hardware dependencies, easy to install
• Testable — mock the backend interface, test AI logic without hardware
• dispim-control is standalone — can be used without gently for manual acquisition scripts
• Same-machine performance — direct function calls, no HTTP/IPC overhead when running together

---

Relation to SharedMemoryPool (this issue)

The SharedMemoryPool lives inside dispim-control as an implementation detail of the backend. From gently's perspective, it just calls backend.get_volume(uid) and gets a numpy array — whether that comes from shared memory, disk, or anywhere else is the backend's concern.

---

How does this look? Happy to refine the interface definition or any other details.

[pskeshu]

@subindevs this is really exciting — I love the dispim-control separation idea. The pattern of wiring a backend to Copilot with lab-specific implementations is elegant:

backend = DiSPIMBackend(config="dispim.yml")
copilot = Copilot(backend=backend)

Even though we currently only have one lab's hardware, this architecture positions us to scale immediately when that changes. The clean separation means any Micro-Manager, EPICS, or custom Bluesky setup could plug in without touching the AI agent logic.

I think we should keep this in the same issue rather than splitting — it's all part of the same major architectural refactor, and the SharedMemoryPool design directly informs how the MicroscopeBackend interface should look.

Request: Data Flow Diagrams

Before we finalize the interface, could you sketch out how data travels through the system? Specifically:

1. Acquisition → Storage flow: How does a volume get from hardware → SharedMemoryPool → disk? What's the sequence of calls?

2. VizServer consumption: How does VizServer access volumes? Does it connect to SharedMemoryPool directly, or go through the backend interface? What about when data is in memory vs. evicted to disk?

3. Perception agents: The perception system needs both live frames (for real-time detection) and stored volumes (for analysis). How do these agents consume data? Same interface as VizServer, or different paths for live vs. historical?

4. Cross-process boundaries: Where are the process boundaries? If SharedMemoryPool lives in the device layer process, how do VizServer and Copilot (separate processes) access it?

A diagram showing these data paths would really help us validate that the MicroscopeBackend protocol has the right methods before we commit to it.

Looking forward to seeing this come together!

[subindevs]

@pskeshu Here's an update on the restructuring work and the data flow diagrams you requested.

Progress Update

I've implemented the project restructuring on a new branch based on agent. The work is split across two repos:

• gently (refactored): subindevs/gently@refactor/project-restructuring-v2
• dispim-control (new backend): subindevs/dispim-control@refactor/project-restructuring-v2

What changed

gently is now backend-agnostic:

• New gently/interface.py defines a MicroscopeBackend protocol
• All agent code uses context.get('backend') instead of context.get('client')
• self.client renamed to self.backend throughout copilot, orchestrator, CLI
• Hardware files removed (devices, plans, config, calibration, servers)
• No pymmcore, ophyd, or bluesky dependencies in gently

dispim-control implements the backend:

• DiSPIMBackend extends QueueServerClient (inherits full interface)
• All hardware code: devices, plans, calibration, detection, servers
• Start scripts in scripts/

Data Flow Diagrams (answering your earlier question)

1. Acquisition → Storage

DiSPIM Hardware
    │
    ▼
dispim_control.DiSPIMBackend
    │  backend = DiSPIMBackend()          # direct Python import
    │  result = await backend.acquire_volume()
    │  (in-process, no HTTP/IPC)
    ▼
Returns Dict with numpy array (in-memory, zero-copy)
    │
    ▼
gently.core.data_store.TiledStore.store(volume)
    ├── In-memory index (DataReference with UID)
    └── Disk persistence (TIFF to storage_path)

2. VizServer Consumption

Copilot (after acquisition)
    │
    │  viz_server.add_image(image, metadata)
    │  (direct Python call, same process)
    ▼
VizServer.ImageStore (in-memory cache)
    └── WebSocket broadcast to browser clients

3. Perception Agents

Copilot
    │  volume = result['volume']     # numpy array, in-memory
    │  image_b64 = encode(projection)
    ▼
PerceptionEngine.perceive(image_b64, volume)
    │  (direct call, same process)
    ▼
Claude API (external HTTPS) → PerceptionResult

4. Cross-Process Boundaries

┌─────────────────────────────────────────────────────┐
│              SINGLE PYTHON PROCESS                  │
│                                                     │
│  from gently.agent import MicroscopyCopilot         │
│  from dispim_control import DiSPIMBackend           │
│                                                     │
│  backend = DiSPIMBackend(config="dispim.yml")       │
│  copilot = MicroscopyCopilot(backend=backend)       │
│                                                     │
│  In-Process (no IPC):                               │
│  • DiSPIMBackend (hardware control)                 │
│  • TiledStore (data persistence)                    │
│  • PerceptionManager (VLM reasoning)                │
│  • EventBus (pub/sub messaging)                     │
│  • VizServer (can be in-process or separate)        │
└─────────────────────────────────────────────────────┘
              │
              │ External only:
              ▼
    Claude API (HTTPS)
    SAM Server (rpyc, if separate GPU)
    Browser clients (WebSocket)

How to test

# Clone dispim-control
git clone https://github.com/subindevs/dispim-control.git
cd dispim-control
git checkout refactor/project-restructuring-v2
pip install -e .

# Switch gently to the restructured branch
cd /path/to/gently
git remote add subin https://github.com/subindevs/gently.git
git fetch subin
git checkout subin/refactor/project-restructuring-v2

# Verify imports
python -c "from gently import MicroscopeBackend; print('OK')"
python -c "from dispim_control import DiSPIMBackend; print('OK')"
python -c "from gently.agent import MicroscopyCopilot; print('OK')"

# Launch (offline mode)
python launch_copilot.py --offline

Would appreciate your feedback on the interface design and data flow before we proceed with the SharedMemoryPool implementation.

[pskeshu]

@subindevs Pulled both branches and verified imports — the repo split looks good and we want to move forward with it.

A few issues we hit when testing on actual hardware:

Missing client.py

simple_server.py:92 does from client import get_mmc, but client.py wasn't included in dispim-control. We had to recreate it manually.

Data flow doesn't match the diagrams

The diagrams describe a single Python process with direct in-process calls and no HTTP/IPC. But DiSPIMBackend extends QueueServerClient, which connects via aiohttp (HTTP to port 60610) and rpyc (to MMCore on 18861, SAM on 18862). The actual call chain is still:

MMCore rpyc (18861) → rpyc → simple_server HTTP (60610) → HTTP → QueueServerClient → copilot

This is the same multi-hop architecture the issue was opened to fix.

Startup is still 3 processes

The target was 2 scripts (start_device_layer.py + launch_copilot.py). We still need 3:

1. scripts/start_server_simple.py (rpyc MMCore)
2. dispim_control/simple_server.py (HTTP API)
3. launch_copilot.py

---

The repo separation is the right direction and we should keep it. Reducing the data hops is the actual problem this issue set out to solve, so we should tackle that as part of this refactor too.

What are your thoughts?

[subindevs]

Gently-Side Refactor Complete

Completed the backend-agnostic cleanup on refactor/project-restructuring-v2. Here's what changed:

Files Removed

• Root-level DiSPIM scripts: client.py, calibrate_embryo_piezo_galvo.py, multi_embryo_calibration.py, spim_hardware_triggering_reference.py, microscope_mcp_server.py, image_processing_utils.py, segment_embryo_nuclei.py, realtime_hatching_detector.py
• Service launch scripts: all .bat and .ps1 files (start_services, stop_services, restart_services, start_backend, start_frontend)
• backend/ directory: sam_server.py, agent_api.py, start_visualization_server.py
• diagnostics/ directory: all three scripts

Code Changes

• gently/gently.py: Removed _register_standard_services() (hardcoded ports 18861/18862/60610), connect_microscope(), connect_sam_server(), ServiceClient usage. Fixed default storage_path from D:/Gently to ./experiment_data.
• launch_copilot.py: Removed top-level from dispim_control import DiSPIMBackend. Added _create_backend() factory with lazy import. Added CLI args: --backend, --backend-url, --sam-host, --sam-port, --storage-dir. Generalized status labels ("Queue Server" → "Backend", "SAM Server" → "Detection Server").
• gently/agent/copilot.py + prompts.py: Renamed queue_server → backend in connection status dict and system prompt.
• gently/agent/tools/hardware_tools.py + timelapse_orchestrator.py: Fixed docstrings QueueServerClient → MicroscopeBackend.
• gently/interface.py: Removed DiSPIM-specific usage example from docstring.
• gently/agent/startup.py: Banner text "diSPIM Control" → "Adaptive Microscopy".
• requirements_copilot.txt: Removed aiohttp, napari, and commented-out bluesky/ophyd lines.

Verification

• Zero QueueServerClient references remain in gently/
• Zero hardcoded ports (18861/60610) in gently/ package code
• requirements_copilot.txt has no rpyc or aiohttp
• launch_copilot.py can run in --offline mode without dispim_control installed

---

Guidance for dispim-control

With gently cleaned up, here's what dispim-control needs to complete the data-hop reduction:

1. Include files from gently

• client.py — The get_mmc() rpyc utility (deleted from gently)
• backend/sam_server.py — The rpyc SAM server wrapper (deleted from gently). It imports gently.agent.sam_detection and gently.coordinates, which are stable public APIs.
• Service launch scripts — The .bat/.ps1 scripts (deleted from gently), updated with correct paths.

2. Rewrite DiSPIMBackend

Currently extends QueueServerClient which connects via HTTP (port 60610) + rpyc (port 18861). The call chain is:

MMCore rpyc (18861) → rpyc → simple_server HTTP (60610) → HTTP → QueueServerClient → copilot

Target: Direct MMCore access, eliminating the HTTP hop. DiSPIMBackend should initialize MMCore directly (or via a single rpyc connection) and implement all gently.interface.MicroscopeBackend methods with in-process calls.

3. Consolidate to 2 processes

• Process 1: SAM Server (GPU-isolated, rpyc) — only needed for detection
• Process 2: Backend + Copilot — DiSPIMBackend with direct MMCore access, MicroscopyCopilot, VizServer, all in one process

4. SharedMemoryPool (optional optimization)

For zero-copy volume access between acquisition and analysis within the single process. Since everything runs in-process now, this becomes a memory management optimization rather than an IPC requirement.

[subindevs]

Opened a detailed issue for the remaining dispim-control work: subindevs/dispim-control#1

Covers all remaining tasks from this issue:

• Task 1: Include missing files from gently (client.py, sam_server.py, startup scripts)
• Task 2: Rewrite DiSPIMBackend for direct MMCore access (eliminate HTTP/rpyc hops)
• Task 3: Consolidate startup to 2 processes
• Task 4: SharedMemoryPool (optional optimization)

Once dispim-control#1 is complete, this issue can be closed.