Dual Camera USB Streamer

This Android app captures dual rear camera streams and raw microphone audio, then sends them to your PC via USB for depth perception detection and audio processing.

Features

• ✅ Dual rear camera capture (1280x720 YUV format)
• ✅ Raw stereo audio capture (44.1kHz, 16-bit PCM)
• ✅ USB serial communication with custom protocol
• ✅ Real-time streaming with timestamps
• ✅ Modern Jetpack Compose UI
• ✅ Permission handling
• ✅ Error reporting and statistics

Android App Setup

Prerequisites

• Android device with dual rear cameras
• USB cable
• Android API level 24+ (Android 7.0+)

Installation

1. Build and install the APK on your Android device
2. Enable Developer Options and USB Debugging
3. Grant Camera and Microphone permissions when prompted

Usage

1. Connect your Android device to PC via USB
2. Launch the app
3. Tap "Start Streaming" to begin capturing and sending data
4. Monitor the status indicators and statistics
5. Tap "Stop Streaming" when done

PC-Side Receiver Setup

Prerequisites

cd pc_receiver
pip install -r requirements.txt

Configuration

1. Edit usb_receiver.py
2. Update the COM_PORT variable:
   • Windows: "COM3", "COM4", etc.
   • Linux: "/dev/ttyUSB0", "/dev/ttyACM0", etc.
   • macOS: "/dev/cu.usbserial-xxx"

Finding Your USB Port

Windows:

# In Device Manager, look under "Ports (COM & LPT)"
# Or use PowerShell:
Get-WmiObject -Class Win32_SerialPort | Select-Object DeviceID,Description

Linux:

# List USB devices
lsusb
# List serial ports
ls /dev/tty*
# Check for new devices after connecting Android
dmesg | tail

macOS:

# List serial ports
ls /dev/cu.*
# Or use system_profiler
system_profiler SPUSBDataType

Running the Receiver

python usb_receiver.py

Protocol Specification

Data Packet Format

Each packet contains:

1. Header (1 byte): Frame type identifier
   • 0x01: Camera 1 frame
   • 0x02: Camera 2 frame
   • 0x03: Audio data
2. Timestamp (8 bytes): Unix timestamp in milliseconds (big-endian)
3. Data Size (4 bytes): Size of payload in bytes (big-endian)
4. Data: Raw camera/audio data
5. End Marker (1 byte): 0xFF

Camera Data Format

• Resolution: 1280x720
• Format: YUV_420_888
• Data: Raw image buffer from Android Camera2 API

Audio Data Format

• Sample Rate: 44,100 Hz
• Channels: Stereo (2 channels)
• Bit Depth: 16-bit signed PCM
• Byte Order: Little-endian

Depth Perception Processing

The dual camera streams can be used for depth perception:

1. Calibration: Calibrate the dual cameras to find intrinsic and extrinsic parameters
2. Stereo Matching: Use algorithms like SGBM or block matching
3. Disparity Map: Generate disparity maps from the stereo pairs
4. Depth Calculation: Convert disparity to actual depth values

Example OpenCV Integration

import cv2
import numpy as np

# Load stereo calibration parameters (you need to calibrate first)
# stereo = cv2.StereoBM_create(numDisparities=64, blockSize=15)

def process_stereo_frame(left_frame, right_frame):
    # Convert YUV to grayscale or RGB
    # Apply stereo matching
    # disparity = stereo.compute(left_gray, right_gray)
    # return disparity
    pass

Troubleshooting

Android App Issues

• Camera not working: Check if device has dual rear cameras
• Permission denied: Grant Camera and Microphone permissions
• USB not connected: Check USB cable and enable USB debugging

PC Receiver Issues

• Port not found: Update COM_PORT variable
• Permission denied: On Linux, add user to dialout group: sudo usermod -a -G dialout $USER
• Data corruption: Check USB cable quality and connection stability

Performance Tips

• Use a high-quality USB cable
• Close other apps using camera/microphone
• Monitor CPU usage on both devices
• Consider reducing frame rate if needed

File Structure

├── app/src/main/java/com/de
│   ├── MainActivity.kt                    # Main activity
│   ├── camera/DualCameraManager.kt       # Camera management
│   ├── audio/AudioCaptureManager.kt      # Audio capture
│   ├── usb/UsbCommunicationManager.kt    # USB communication
│   ├── viewmodel/StreamingViewModel.kt   # App state management
│   └── ui/components/                     # UI components
├── pc_receiver/
│   ├── usb_receiver.py                   # PC-side receiver
│   └── requirements.txt                  # Python dependencies
└── README.md                            # This file

License

This project is for educational and research purposes. Ensure compliance with local laws and regulations when capturing audio/video data.