Skip to content

Feat: Ingest Vision Camera #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gmemmy merged 3 commits into from
Oct 10, 2025
Merged

Feat: Ingest Vision Camera #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1def3a8
feat: ingest backpressure + per-track stats
gmemmy Sep 25, 2025
3b64bfd
Merge branch 'main' into feat/vision-camera-ingest
gmemmy Oct 2, 2025
7fd5004
chore: introduce cameera prcoessor
gmemmy Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

# VSCode
.vscode/
.cursor/
jsconfig.json

# Xcode
Expand Down
9 changes: 8 additions & 1 deletion example/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
import {AppRegistry} from 'react-native';
import {SafeAreaProvider} from 'react-native-safe-area-context';
import App from './src/App';
import {name as appName} from './app.json';

AppRegistry.registerComponent(appName, () => App);
const Root = () => (
<SafeAreaProvider>
<App />
</SafeAreaProvider>
);

AppRegistry.registerComponent(appName, () => Root);
10 changes: 10 additions & 0 deletions example/ios/Podfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2341,6 +2341,12 @@ PODS:
- React-utils (= 0.81.1)
- SocketRocket
- SocketRocket (0.7.1)
- VisionCamera (4.7.2):
- VisionCamera/Core (= 4.7.2)
- VisionCamera/React (= 4.7.2)
- VisionCamera/Core (4.7.2)
- VisionCamera/React (4.7.2):
- React-Core
- VisionRtc (0.1.0):
- boost
- DoubleConversion
Expand Down Expand Up @@ -2447,6 +2453,7 @@ DEPENDENCIES:
- ReactCodegen (from `build/generated/ios`)
- ReactCommon/turbomodule/core (from `../node_modules/react-native/ReactCommon`)
- SocketRocket (~> 0.7.1)
- VisionCamera (from `../node_modules/react-native-vision-camera`)
- VisionRtc (from `../..`)
- WebRTC-SDK
- Yoga (from `../node_modules/react-native/ReactCommon/yoga`)
Expand Down Expand Up @@ -2600,6 +2607,8 @@ EXTERNAL SOURCES:
:path: build/generated/ios
ReactCommon:
:path: "../node_modules/react-native/ReactCommon"
VisionCamera:
:path: "../node_modules/react-native-vision-camera"
VisionRtc:
:path: "../.."
Yoga:
Expand Down Expand Up @@ -2678,6 +2687,7 @@ SPEC CHECKSUMS:
ReactCodegen: 4d203eddf6f977caa324640a20f92e70408d648b
ReactCommon: ce5d4226dfaf9d5dacbef57b4528819e39d3a120
SocketRocket: d4aabe649be1e368d1318fdf28a022d714d65748
VisionCamera: 30b358b807324c692064f78385e9a732ce1bebfe
VisionRtc: 56e770d48b49da73a130f18a3ca12148a12126e8
WebRTC-SDK: 69d4e56b0b4b27d788e87bab9b9a1326ed05b1e3
Yoga: 11c9686a21e2cd82a094a723649d9f4507200fb0
Expand Down
10 changes: 6 additions & 4 deletions example/ios/VisionRtcExample.xcodeproj/project.pbxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,18 +257,19 @@
isa = XCBuildConfiguration;
baseConfigurationReference = 3B4392A12AC88292D35C810B /* Pods-VisionRtcExample.debug.xcconfig */;
buildSettings = {
ENABLE_PREVIEWS = NO;
MACH_O_TYPE = mh_execute;
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
CLANG_ENABLE_MODULES = YES;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 9W7U4QA43U;
ENABLE_BITCODE = NO;
ENABLE_PREVIEWS = NO;
INFOPLIST_FILE = VisionRtcExample/Info.plist;
IPHONEOS_DEPLOYMENT_TARGET = 15.1;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
);
MACH_O_TYPE = mh_execute;
MARKETING_VERSION = 1.0;
OTHER_LDFLAGS = (
"$(inherited)",
Expand All @@ -287,17 +288,18 @@
isa = XCBuildConfiguration;
baseConfigurationReference = 5709B34CF0A7D63546082F79 /* Pods-VisionRtcExample.release.xcconfig */;
buildSettings = {
ENABLE_PREVIEWS = NO;
MACH_O_TYPE = mh_execute;
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
CLANG_ENABLE_MODULES = YES;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 9W7U4QA43U;
ENABLE_PREVIEWS = NO;
INFOPLIST_FILE = VisionRtcExample/Info.plist;
IPHONEOS_DEPLOYMENT_TARGET = 15.1;
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/Frameworks",
);
MACH_O_TYPE = mh_execute;
MARKETING_VERSION = 1.0;
OTHER_LDFLAGS = (
"$(inherited)",
Expand Down
4 changes: 4 additions & 0 deletions example/ios/VisionRtcExample/Info.plist
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,9 @@
</array>
<key>UIViewControllerBasedStatusBarAppearance</key>
<false/>
<key>NSCameraUsageDescription</key>
<string>This app requires camera access for live video.</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app may use the microphone for audio capture.</string>
</dict>
</plist>
3 changes: 2 additions & 1 deletion example/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
"@react-native/new-app-screen": "0.81.1",
"react": "19.1.0",
"react-native": "0.81.1",
"react-native-safe-area-context": "^5.6.1"
"react-native-safe-area-context": "^5.6.1",
"react-native-vision-camera": "^4.7.2"
},
"devDependencies": {
"@babel/core": "^7.25.2",
Expand Down
191 changes: 165 additions & 26 deletions example/src/App.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,66 @@
import * as React from 'react';
import {Button, Text, StyleSheet, View} from 'react-native';
import {SafeAreaView} from 'react-native-safe-area-context';
import {Button, Text, StyleSheet, View, findNodeHandle} from 'react-native';
import {SafeAreaView, useSafeAreaInsets} from 'react-native-safe-area-context';
import {Camera, useCameraDevice} from 'react-native-vision-camera';
import {
createVisionCameraSource,
createWebRTCTrack,
updateSource,
updateTrack,
disposeSource,
disposeTrack,
getStats,
VisionRTCView,
} from 'react-native-vision-rtc';
import TestFramework from './test-framework';

export default function App() {
const insets = useSafeAreaInsets();
const cameraRef = React.useRef<Camera>(null);
const device = useCameraDevice('back');
Comment on lines +19 to +20
Copy link

@coderabbitai coderabbitai bot Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Type error: useRef should allow null.

This won’t type-check in TS.

Apply:

-  const cameraRef = React.useRef<Camera>(null);
+  const cameraRef = React.useRef<Camera | null>(null);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const cameraRef = React.useRef<Camera>(null);
const device = useCameraDevice('back');
const cameraRef = React.useRef<Camera | null>(null);
const device = useCameraDevice('back');
🤖 Prompt for AI Agents 
In example/src/App.tsx around lines 18-19, the useRef generic currently excludes
null which causes a TypeScript type error; update the ref type to allow null
(e.g., change the generic to Camera | null) so the initial null value
type-checks, and ensure any usages are null-checked or non-null asserted where
appropriate.

const [trackId, setTrackId] = React.useState<string | null>(null);
const [showTests, setShowTests] = React.useState(false);
const [sourceId, setSourceId] = React.useState<string | null>(null);
const [stats, setStats] = React.useState<{
fps: number;
producedFps: number;
deliveredFps: number;
droppedFrames: number;
} | null>(null);
const [creating, setCreating] = React.useState(false);
const [torch, setTorch] = React.useState(false);
const [facing, setFacing] = React.useState<'front' | 'back'>('back');
const [backpressure, setBackpressure] = React.useState<
'drop-late' | 'latest-wins' | 'throttle'
>('drop-late');

const ensurePermissions = React.useCallback(async () => {
const cam = await Camera.getCameraPermissionStatus();
if (cam !== 'granted') {
const res = await Camera.requestCameraPermission();
if (res !== 'granted') throw new Error('Camera permission not granted');
}
}, []);

const onStart = async () => {
if (creating || trackId) return;
setCreating(true);
let newId: string | null = null;
try {
const {trackId: id} = await createWebRTCTrack(
{__nativeSourceId: 'null'},
await ensurePermissions();
const node = findNodeHandle(cameraRef.current);
if (!node) throw new Error('Camera view not ready');
const {__nativeSourceId} = await createVisionCameraSource(node);
setSourceId(__nativeSourceId);
const created = await createWebRTCTrack(
{__nativeSourceId},
{
fps: 30,
resolution: {width: 1280, height: 720},
backpressure,
}
);
newId = id;
setTrackId(id);
newId = created.trackId;
setTrackId(created.trackId);
} catch (err) {
Comment on lines +49 to 64
Copy link

@coderabbitai coderabbitai bot Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Dispose source on failure to avoid leaks.

If track creation fails after creating the Vision source, the source is never disposed.

Apply:

   setCreating(true);
   let newId: string | null = null;
+  let createdSourceId: string | null = null;
   try {
     await ensurePermissions();
     const node = findNodeHandle(cameraRef.current);
     if (!node) throw new Error('Camera view not ready');
-    const {__nativeSourceId} = await createVisionCameraSource(node);
-    setSourceId(__nativeSourceId);
+    const {__nativeSourceId} = await createVisionCameraSource(node);
+    createdSourceId = __nativeSourceId;
+    setSourceId(__nativeSourceId);
     const created = await createWebRTCTrack(
       {__nativeSourceId},
       {
         fps: 30,
         resolution: {width: 1280, height: 720},
         backpressure,
       }
     );
     newId = created.trackId;
     setTrackId(created.trackId);
   } catch (err) {
     if (newId) {
       try {
         await disposeTrack(newId);
       } catch {}
     }
+    if (createdSourceId) {
+      try {
+        await disposeSource(createdSourceId);
+      } catch {}
+    }
     console.error('Failed to start WebRTC track', err);
   } finally {
     setCreating(false);
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
await ensurePermissions();
const node = findNodeHandle(cameraRef.current);
if (!node) throw new Error('Camera view not ready');
const {__nativeSourceId} = await createVisionCameraSource(node);
setSourceId(__nativeSourceId);
const created = await createWebRTCTrack(
{__nativeSourceId},
{
fps: 30,
resolution: {width: 1280, height: 720},
backpressure,
}
);
newId = id;
setTrackId(id);
newId = created.trackId;
setTrackId(created.trackId);
} catch (err) {
setCreating(true);
let newId: string | null = null;
let createdSourceId: string | null = null;
try {
await ensurePermissions();
const node = findNodeHandle(cameraRef.current);
if (!node) throw new Error('Camera view not ready');
const {__nativeSourceId} = await createVisionCameraSource(node);
createdSourceId = __nativeSourceId;
setSourceId(__nativeSourceId);
const created = await createWebRTCTrack(
{__nativeSourceId},
{
fps: 30,
resolution: {width: 1280, height: 720},
backpressure,
}
);
newId = created.trackId;
setTrackId(created.trackId);
} catch (err) {
if (newId) {
try {
await disposeTrack(newId);
} catch {}
}
if (createdSourceId) {
try {
await disposeSource(createdSourceId);
} catch {}
}
console.error('Failed to start WebRTC track', err);
} finally {
setCreating(false);
}
🤖 Prompt for AI Agents 
In example/src/App.tsx around lines 47 to 62, the Vision camera source created
via createVisionCameraSource is not disposed if subsequent createWebRTCTrack
fails, leaking native resources; update the try/catch (or add a finally) to
store the created __nativeSourceId in an outer variable and, on error or in
finally when track creation did not complete, call and await the appropriate
source-disposal API (e.g., disposeVisionCameraSource or equivalent) only if
__nativeSourceId exists, then clear the variable; ensure disposal errors are
caught/logged but do not mask the original error.

if (newId) {
try {
Expand All @@ -43,13 +74,15 @@ export default function App() {
};

const onStop = async () => {
if (!trackId) return;
if (!trackId && !sourceId) return;
try {
await disposeTrack(trackId);
if (trackId) await disposeTrack(trackId);
if (sourceId) await disposeSource(sourceId);
} catch (e) {
console.warn('Failed to dispose track', e);
} finally {
setTrackId(null);
setSourceId(null);
setStats(null);
}
};
Expand All @@ -62,26 +95,119 @@ export default function App() {
};
}, [trackId]);

const onGetStats = async () => {
const s = await getStats();
if (s) setStats({fps: s.fps, droppedFrames: s.droppedFrames});
const onGetStats = React.useCallback(async () => {
const s = await getStats(trackId ?? undefined);
if (s)
setStats({
producedFps: s.producedFps,
deliveredFps: s.deliveredFps,
droppedFrames: s.droppedFrames,
});
}, [trackId]);

React.useEffect(() => {
if (!trackId) return;
const t = setInterval(onGetStats, 1000);
return () => clearInterval(t);
}, [trackId, onGetStats]);

const onFlip = async () => {
if (!sourceId) return;
const next = facing === 'back' ? 'front' : 'back';
setFacing(next);
// Turning torch off when flipping avoids devices without torch (e.g., front)
setTorch(false);
await updateSource(sourceId, {position: next});
};

const onTorch = async () => {
if (!sourceId) return;
const next = !torch;
setTorch(next);
await updateSource(sourceId, {torch: next});
};

const onFps = async (fps: number) => {
if (!trackId) return;
console.log('onFps', fps);
await updateTrack(trackId, {fps});
};

const onToggleBackpressure = async () => {
const order: Array<'drop-late' | 'latest-wins' | 'throttle'> = [
'drop-late',
'latest-wins',
'throttle',
];
const currentIndex = order.indexOf(backpressure);
const idx = currentIndex === -1 ? 0 : (currentIndex + 1) % order.length;
const next = order[idx] as 'drop-late' | 'latest-wins' | 'throttle';
setBackpressure(next);
if (trackId) {
await updateTrack(trackId, {backpressure: next});
}
};

if (showTests) {
return <TestFramework onBack={() => setShowTests(false)} />;
}

return (
<SafeAreaView edges={['top']} style={styles.container}>
<View style={styles.controls}>
<Button title="Start" onPress={onStart} />
<Button title="Stop" onPress={onStop} />
<Button title="Get Stats" onPress={onGetStats} />
</View>
<View style={styles.preview}>
<VisionRTCView trackId={trackId ?? ''} />
<View style={styles.hud} pointerEvents="none">
{device && (
<Camera
ref={cameraRef}
style={StyleSheet.absoluteFill}
device={device}
isActive={true}
torch={torch && device?.hasTorch ? 'on' : 'off'}
/>
)}
<VisionRTCView trackId={trackId ?? ''}>
<Text style={[styles.trackId, {marginTop: insets.top + 20}]}>
{trackId ?? 'No track id'}
</Text>
</VisionRTCView>
<View style={styles.hud}>
<Text style={styles.hudText}>
{`fps: ${stats?.fps ?? 0} drops: ${stats?.droppedFrames ?? 0}`}
{`prod: ${stats?.producedFps ?? 0} deliv: ${stats?.deliveredFps ?? 0} drops: ${stats?.droppedFrames ?? 0}`}
</Text>
</View>
</View>
<View style={styles.controls}>
<View style={styles.btn}>
<Button title="Start" onPress={onStart} />
</View>
<View style={styles.btn}>
<Button title="Stop" onPress={onStop} />
</View>
<View style={styles.btn}>
<Button title="Flip" onPress={onFlip} />
</View>
<View style={styles.btn}>
<Button
title={torch ? 'Torch Off' : 'Torch On'}
onPress={onTorch}
disabled={!device?.hasTorch}
/>
</View>
<View style={styles.btn}>
<Button title="15 fps" onPress={() => onFps(15)} />
</View>
<View style={styles.btn}>
<Button title="30 fps" onPress={() => onFps(30)} />
</View>
<View style={styles.btn}>
<Button
title={`BP: ${backpressure}`}
onPress={onToggleBackpressure}
/>
</View>
<View style={styles.btn}>
<Button title="🧪 Tests" onPress={() => setShowTests(true)} />
</View>
</View>
</SafeAreaView>
);
}
Expand All @@ -90,30 +216,43 @@ const styles = StyleSheet.create({
container: {
flex: 1,
backgroundColor: '#ffff',
justifyContent: 'center',
alignItems: 'center',
},
controls: {
Comment on lines 216 to 222
Copy link

@coderabbitai coderabbitai bot Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Color value typo.

'#ffff' is unusual; use '#fff' or '#ffffff'.

Apply:

-    backgroundColor: '#ffff',
+    backgroundColor: '#fff',
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
container: {
flex: 1,
backgroundColor: '#ffff',
justifyContent: 'center',
alignItems: 'center',
},
controls: {
container: {
flex: 1,
backgroundColor: '#fff',
justifyContent: 'center',
alignItems: 'center',
},
controls: {
🤖 Prompt for AI Agents 
In example/src/App.tsx around lines 207 to 213, the container style uses an
invalid/typo color value '#ffff'; replace it with a valid 3- or 6-digit hex like
'#fff' or '#ffffff' (e.g., set backgroundColor to '#ffffff') so the style uses a
correct hex color.

padding: 12,
flexDirection: 'row',
justifyContent: 'space-around',
alignItems: 'center',
flexWrap: 'wrap',
},
btn: {minWidth: 120, margin: 6},
preview: {
flex: 1,
alignItems: 'stretch',
justifyContent: 'center',
position: 'absolute',
top: 0,
left: 0,
right: 0,
bottom: 0,
},
fill: {flex: 1},
hud: {
position: 'absolute',
bottom: 8,
right: 8,
bottom: 30,
right: 0,
left: 0,
alignItems: 'center',
backgroundColor: 'rgba(0,0,0,0.4)',
paddingHorizontal: 8,
paddingVertical: 4,
borderRadius: 4,
},
hudText: {
color: '#fff',
fontSize: 12,
fontSize: 14,
},
trackId: {
color: 'gray',
fontSize: 14,
textAlign: 'center',
},
});
Loading
Loading