Skip to content

RobotFlow-Labs/zed-sdk-mlx

BranchesTags

Repository files navigation

ZED Open Capture -- macOS / Apple Silicon / MLX Fork

A macOS and Apple Silicon MLX port of the Stereolabs open capture stack for the ZED 2, ZED 2i, and ZED Mini stereo cameras

*** not compatible with GMSL2 devices: ZED X, ZED X Mini, and ZED X One ***

macOS / Apple Silicon • Port Status • Key Features • Build and install • Run • Documentation • Examples • Known issues • Related • License


This repository is the zed-sdk-mlx fork of zed-open-capture. It replaces the Linux/V4L2 video backend with AVFoundation, replaces CUDA/OpenCV depth with MLX on Apple Silicon, and keeps the upstream C++ API shape intact. The architecture notes live in porting_plan.md, the execution backlog in TODO.md, and the macOS bring-up workflow in docs/macos.md.

macOS / Apple Silicon

What works

  • Raw stereo capture -- AVFoundation backend, validated on ZED 2i at 2560x720 (YUYV/yuvs)
  • Sensor capture -- IMU, magnetometer, barometer, temperature via hidapi (same as Linux)
  • Video/sensor sync -- timestamp-fallback sync (no hardware gate on macOS)
  • Calibration -- auto-loads ~/zed/settings/SN<serial>.conf from Stereolabs servers
  • Stereo rectification -- OpenCV remap, same path as Linux
  • MLX disparity and depth -- Apple Silicon GPU-accelerated stereo matching
  • Point cloud export -- rectified colored PLY from MLX depth
  • Live preview with tuning -- shared-memory capture-to-MLX pipeline with interactive trackbars and persistent presets

Performance

On a ZED 2i at 1280x720 with the default live preset (pyramid_factor=2, refinement_radius=0, window_size=5):

Metric Value
MLX compute per frame ~8 ms
Live loop (with cv2 display) ~69 ms (~14.5 FPS)
Offline eval (fast preset) ~160 ms (~6.25 FPS)
Offline eval (full resolution) ~1054 ms (~0.93 FPS)

The live bottleneck is Python/OpenCV display overhead, not MLX compute.

Prerequisites

  • macOS 13+ on Apple Silicon (M1 or later)
  • Python 3.12+
  • uv (Python environment manager)
  • Homebrew: brew install hidapi opencv
  • CMake 3.1+, Xcode command line tools

Quick start

git clone https://github.com/RobotFlow-Labs/zed-sdk-mlx.git
cd zed-sdk-mlx

# Bootstrap environment and build
make sync
make configure-zed-open-capture
make build-zed-open-capture
brew install opencv
make build-examples

# Grant camera permission (first time only)
make request-camera-access-terminal

# Smoke test -- opens Terminal, captures a frame
make smoke-zed-terminal

MLX depth

# Live MLX disparity viewer
make live-mlx-disparity-terminal

# Routed depth demo (same as upstream depth example, routes to MLX on macOS)
make run-depth-example-terminal

# Live stereo tuner (saves presets to ~/zed/settings/mlx_live_stereo.json)
make run-tune-stereo-example-terminal

# One-shot evaluation report
make mlx-stereo-eval

# Point cloud export
make mlx-point-cloud

Known limitations

Camera controls are limited to auto/locked mode toggle only. See the full table below for details.

Features that could not be ported to macOS

The following Linux features are not available on macOS due to a single root cause: the macOS kernel UVC driver has no public API (UVCIOC_CTRL_QUERY equivalent) for forwarding USB class-specific control requests to the device. We built the full vendor Extension Unit transport layer (IOKit, XU unit 0x04, I2C sensor register bridge) and confirmed every request STALLs because the kernel intercepts them. This is a confirmed macOS platform limitation, not a code bug.

Feature Linux path macOS status Root cause
Manual exposure (duration/ISO) V4L2 + vendor XU register I/O Not portable Kernel blocks USB control transfers; AVFoundation exposureDuration/iso are iOS-only
Manual gain control Vendor XU sensor register write Not portable Same kernel block; no AVFoundation gain API on any platform
Manual white balance (temperature/gains) Vendor XU + V4L2 Not portable deviceWhiteBalanceGains/setWhiteBalanceModeLocked(with:) are iOS-only
Exposure compensation/bias V4L2 Not portable setExposureTargetBias is iOS-only
ROI-based exposure metering V4L2 Not portable API exists on macOS but USB cameras return unsupported
ISP register read/write Vendor XU (unit 0x04, selector 0x02) Not portable Full protocol implemented, kernel STALLs all class-specific transfers
LED control Vendor XU Not portable Uses same blocked XU path
Hardware video/sensor sync gate Low-level camera signaling Not portable macOS AVFoundation does not expose hardware sync signals

What we ship instead:

  • Auto/locked mode toggle for exposure and white balance (the only controls macOS allows via AVFoundation)
  • Timestamp-based sync fallback (validated at ~2.5 ms median offset on ZED 2i)
  • All blocked features are capability-gated with clear error messages at runtime
  • The full vendor XU transport code is preserved in src/uvc_controls_macos.mm for future use if Apple adds a UVC control query API or if the project moves to a full libusb capture path

Other limitations

  • OpenCV depth fallback: The CPU/OpenCV SGBM depth path works but is not the primary path on macOS. Use the MLX targets instead.
  • Multi-camera: Build support exists but has not been validated on macOS hardware with multiple ZED cameras.
  • ROS2 wrapper: Deferred until the base capture stack stabilizes.

Port Status

Surface Linux macOS Classification
Raw video capture V4L2 AVFoundation Native parity
Sensor capture (IMU, mag, baro, temp) hidapi hidapi Native parity
Video/sensor sync Hardware timestamp Timestamp fallback macOS replacement
Calibration loading File I/O File I/O Native parity
Stereo rectification OpenCV OpenCV Native parity
Video example V4L2 + OpenCV AVFoundation + OpenCV Native parity
Multi-camera example V4L2 AVFoundation macOS replacement (needs validation)
Camera control example V4L2 ioctls Mode toggle only Capability-gated partial
Depth example OpenCV SGBM MLX disparity macOS replacement
Tune-stereo example OpenCV SGBM GUI MLX live tuner macOS replacement
Point cloud export OpenCV + PLY MLX + PLY Native parity
Point cloud viewer OpenGL -- Deferred
Evaluation harness N/A MLX eval + JSON/Markdown New capability
ROS2 wrapper Native -- Deferred

Classification legend:

  • Native parity -- same or equivalent functionality, works today
  • macOS replacement -- different implementation, equivalent user-facing workflow
  • Capability-gated partial -- reduced functionality due to platform API limits, clearly documented
  • Deferred -- not yet implemented, planned for future work

Key Features

  • Open source C++ capture library compatible with C++11 standard
  • Video Capture
    • raw YUV 4:2:2 data format
    • Camera controls
    • HIGH-RESOLUTION mode with USB3
    • compatibility VGA mode with USB2
  • Sensor Data Capture
    • 6-DOF IMU (3-DOF accelerometer + 3-DOF gyroscope)
    • 3-DOF Magnetometer (Only ZED 2 and ZED 2i)
    • Barometer (Only ZED 2 and ZED 2i)
    • Sensors temperature (Only ZED 2 and ZED 2i)
  • Sensors/video Synchronization
  • Portable
    • Tested on Linux
    • Tested on macOS / Apple Silicon (this fork)
    • Tested on x64, ARM
  • Small Size
    • ~100KB library size
    • libusb, hidapi dependencies
  • Complete set of examples
    • Video capture
    • Camera control
    • Stereo rectification
    • IMU, magnetometer, and barometer data capture
    • Video and sensor synchronization
    • Disparity/Depth/Point Cloud extraction using MLX (macOS) or OpenCV Transparent API (Linux)
    • Depth tuning using MLX live tuner (macOS) or OpenCV control GUI (Linux)

Description

The ZED Open Capture is a multi-platform, open-source C++ library for low-level camera and sensor capture for the ZED stereo camera family. It doesn't require CUDA and therefore can be used on many desktop and embedded platforms.

The open-source library provides methods to access raw video frames, calibration data, camera controls, and raw data from the USB3 camera sensors (on ZED 2, ZED 2i, and ZED Mini). A synchronization mechanism is provided to get the correct sensor data associated with a video frame.

Note: While in the ZED SDK all output data is calibrated and compensated, here the extracted raw data is not corrected by the camera and sensor calibration parameters. You can retrieve camera and sensor calibration data using the ZED SDK to correct your camera data see zed_open_capture_rectify_example example.

Build and install

Prerequisites

Linux

  • GCC (v7.5+)
  • CMake (v3.1+)
  • OpenCV (v3.4.0+) -- optional, for examples
sudo apt install build-essential cmake
sudo apt install libusb-1.0-0-dev libhidapi-libusb0 libhidapi-dev
sudo apt install libopencv-dev libopencv-viz-dev  # optional

macOS / Apple Silicon

  • macOS 13+ on Apple Silicon (M1 or later)
  • Xcode command line tools
  • CMake (v3.1+)
  • Homebrew
  • Python 3.12+ and uv (for MLX depth pipeline)
brew install hidapi opencv cmake

See docs/macos.md for the full macOS workflow.

Clone the repository

git clone https://github.com/RobotFlow-Labs/zed-sdk-mlx.git
cd zed-sdk-mlx

Add udev rule (Linux only)

Stereo cameras such as ZED 2 and ZED Mini have built-in sensors (e.g. IMU) that are identified as USB HID devices. To be able to access the USB HID device, you must add a udev rule contained in the udev folder:

cd udev
bash install_udev_rule.sh
cd ..

Build

Linux

mkdir build
cd build
cmake ..
make -j$(nproc)

macOS

make sync
make configure-zed-open-capture
make build-zed-open-capture
brew install opencv
make build-examples

Build only the library

mkdir build
cd build
cmake .. -DBUILD_EXAMPLES=OFF
make -j$(nproc)

Build only the video capture library

mkdir build
cd build
cmake .. -DBUILD_SENSORS=OFF -DBUILD_EXAMPLES=OFF
make -j$(nproc)

Build only the sensor capture library

mkdir build
cd build
cmake .. -DBUILD_VIDEO=OFF -DBUILD_EXAMPLES=OFF
make -j$(nproc)

Install

To install the library, go to the build folder and launch the following commands:

sudo make install
sudo ldconfig

Run

Get video data

Include the videocapture.hpp header, declare a VideoCapture object, and retrieve a video frame (in YUV 4:2:2 format) with getLastFrame():

#include "videocapture.hpp"

sl_oc::video::VideoCapture cap;
cap.initializeVideo();
const sl_oc::video::Frame frame = cap.getLastFrame();

Get sensor data

Include the SensorCapture header, declare a SensorCapture object, get a list of available devices, initialize the first one, and finally retrieve sensor data:

#include "sensorcapture.hpp"

sl_oc::sensors::SensorCapture sens;
std::vector<int> devs = sens.getDeviceList();
sens.initializeSensors( devs[0] );
const sl_oc::sensors::data::Imu imuData = sens.getLastIMUData(5000);
const sl_oc::sensors::data::Magnetometer magData = sens.getLastMagnetometerData(100);
const sl_oc::sensors::data::Environment envData = sens.getLastEnvironmentData(100);
const sl_oc::sensors::data::Temperature tempData = sens.getLastCameraTemperatureData(100);

Running the examples

After building, the following examples are available:

Running on macOS

Use the Terminal-hosted make targets (required for camera permission prompting):

make run-video-example-terminal
make run-control-example-terminal
make run-rectify-example-terminal
make run-sync-example-terminal
make run-depth-example-terminal
make run-tune-stereo-example-terminal
make smoke-sensors

Running on Linux

zed_open_capture_video_example
zed_open_capture_multicam_video_example
zed_open_capture_control_example
zed_open_capture_rectify_example
zed_open_capture_sensors_example
zed_open_capture_sync_example
zed_open_capture_depth_example
zed_open_capture_depth_tune_stereo

Note: OpenCV is used in the examples for controls, display, and depth extraction.

Documentation

The API is documented in the Include.h files. It is also generated as a Doxygen for simpler navigation: https://stereolabs.github.io/zed-open-capture

You can also generate the documentation locally in HTML format (with Doxygen) using the commands below. Access the docs by opening doc/html/index.html in your web browser.

sudo apt-get install -y doxygen # if not previously installed
cd doc
./generate_doc.sh

Coordinates system

The coordinate system is only used for sensor data. The given IMU and Magnetometer data are expressed in the RAW coordinate system as shown below

Known issues

macOS camera controls

Camera controls on macOS are limited to auto/locked mode toggle. The macOS kernel UVC driver blocks sideband USB control transfers, and the fine-grained numeric exposure, ISO, gain, and white-balance APIs available on iOS are not exposed on macOS for external USB cameras. See claude_supervision.md for the full technical investigation.

macOS camera permission

Inside automated environments (e.g., Codex), tccd attributes camera access to the host process, which may lack the camera entitlement required for prompting. Use the *-terminal make targets to trigger the normal macOS camera prompt through Terminal.

OpenGL version (Linux / embedded)

On some embedded devices, like Raspberry Pi 4, the depth extraction example can crash with the following error:

vtkShaderProgram.cxx:438 ERR| vtkShaderProgram (0x23a611c0): 0:1(10): error: GLSL 1.50 is not supported. Supported versions are: 1.10, 1.20, 1.00 ES, and 3.00 ES

to correctly execute the example application it is necessary to change the default OpenGL version:

export MESA_GL_VERSION_OVERRIDE=3.2

you can permanently add this configuration by adding the above command as the last line of the ~/.bashrc file.

Related

License

This library is licensed under the MIT License.

Support

If you need assistance go to our Community site at https://community.stereolabs.com/

About

No description, website, or topics provided.

Resources

Readme

License

MIT license

Contributing

Contributing
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages