Machine vision integration for ArduPilot drones and planes
Real-time object detection and counting from aerial platforms
ArduPilot Vision is a complete machine vision system designed to run on a Raspberry Pi Zero W2 with the Google AIY Vision Hat. It integrates with ArduPilot flight controllers to enable autonomous vision-based missions such as:
- Livestock counting - Count sheep, cattle, or other animals in a geofenced area
- Area surveying - Systematic coverage with object detection at each waypoint
- Change detection - Compare images over time to detect changes
- Object monitoring - Continuous monitoring for specific object types
The system uses TensorFlow Lite with Edge TPU acceleration for real-time inference, achieving detection speeds suitable for aerial survey operations.
| Feature | Description |
|---|---|
| Edge TPU Acceleration | Leverages Google's Edge TPU for fast, low-power inference |
| MAVLink Integration | Full bidirectional communication with ArduPilot |
| Geofence Support | Reads fence from flight controller, generates survey patterns |
| Object Tracking | Maintains object identity across frames for accurate counting |
| REST API | HTTP API for remote monitoring and control |
| Modular Design | Easy to extend with new tasks and detection models |
| Simulation Mode | Test and develop without hardware |
| Component | Specification |
|---|---|
| Computer | Raspberry Pi Zero W2 |
| Vision | Google AIY Vision Kit (with Edge TPU) |
| Camera | Raspberry Pi Camera Module v2 |
| Flight Controller | Any ArduPilot-compatible (Pixhawk, Cube, etc.) |
| Connection | Serial (UART) or UDP via telemetry radio |
# Clone the repository
git clone https://github.com/yourusername/ardupilot-vision.git
cd ardupilot-vision
# Create virtual environment
python3 -m venv venv
source venv/bin/activate
# Install the package
pip install -e .# Run with default settings
ardupilot-vision
# Run in simulation mode (no hardware required)
ardupilot-vision --simulation
# Run with custom configuration
ardupilot-vision --config /path/to/config.yaml
# Enable verbose logging
ardupilot-vision --verboseCopy and customize the default configuration:
cp ardupilot_vision/config/default_config.yaml config.yamlKey configuration options:
# MAVLink connection
mavlink:
connection_type: serial # or 'udp'
serial:
port: /dev/ttyAMA0
baudrate: 57600
# Detection settings
vision:
detector:
confidence_threshold: 0.5
use_edge_tpu: true
target_classes:
- 18 # sheep
- 19 # cow
- 0 # person
# API server
api:
enabled: true
port: 5000┌─────────────────────────────────────────────────────────────┐
│ ArduPilot Vision │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌───────────────────┐ │
│ │ MAVLink │ │ Vision │ │ Mission │ │
│ │ Interface │◄─┤ Engine │◄─┤ Controller │ │
│ └──────┬──────┘ └──────┬──────┘ └─────────┬─────────┘ │
│ │ │ │ │
│ ┌──────▼──────┐ ┌──────▼──────┐ ┌─────────▼─────────┐ │
│ │ Geofence │ │ Object │ │ Task │ │
│ │ Manager │ │ Tracker │ │ Scheduler │ │
│ └─────────────┘ └─────────────┘ └───────────────────┘ │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌───────────────────┐ │
│ │ Config │ │ Logger │ │ REST API │ │
│ │ Manager │ │ │ │ Server │ │
│ └─────────────┘ └─────────────┘ └───────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
ardupilot-vision/
├── ardupilot_vision/
│ ├── __init__.py
│ ├── main.py # Application entry point
│ ├── config/
│ │ ├── settings.py # Configuration management
│ │ └── default_config.yaml
│ ├── mavlink/
│ │ ├── connection.py # MAVLink communication
│ │ └── geofence.py # Geofence operations
│ ├── vision/
│ │ ├── camera.py # Camera abstraction
│ │ ├── detector.py # Object detection
│ │ └── tracker.py # Object tracking
│ ├── mission/
│ │ ├── controller.py # Mission state machine
│ │ └── tasks.py # Vision task definitions
│ ├── api/
│ │ ├── server.py # Flask server
│ │ └── routes.py # API endpoints
│ └── utils/
│ ├── logger.py # Logging utilities
│ └── geo.py # Geographic calculations
├── docs/
│ ├── architecture.md
│ ├── setup.md
│ ├── api.md
│ └── functions.md
├── tests/
├── requirements.txt
├── setup.py
└── README.md
The REST API runs on port 5000 by default. Full documentation is available in docs/api.md.
# Check system status
curl http://localhost:5000/api/status
# Get vehicle telemetry
curl http://localhost:5000/api/vehicle
# Start a mission
curl -X POST http://localhost:5000/api/mission/start
# Get detection counts
curl http://localhost:5000/api/detectionsimport requests
API_URL = "http://raspberrypi.local:5000"
# Get current detection counts
response = requests.get(f"{API_URL}/api/detections")
data = response.json()
print(f"Total objects counted: {data['total_count']}")
for class_name, count in data['class_counts'].items():
print(f" {class_name}: {count}")- Setup Geofence - Define survey area in Mission Planner or QGroundControl
- Configure Detection - Set target classes and confidence threshold
- Deploy - Mount Pi + camera on aircraft
- Execute Mission - Start survey via API or automatic trigger
- Review Results - Access counts and detections via API
from ardupilot_vision import ArduPilotVision
# Initialize the system
app = ArduPilotVision(config_path="config.yaml")
# Generate survey waypoints from geofence
app.geofence.download_fence()
waypoints = app.geofence.generate_survey_waypoints(
spacing_m=50,
altitude_m=30,
overlap_percent=20
)
# Start mission
app.mission.set_waypoints(waypoints)
app.mission.start()
# Wait for completion
while app.mission.is_running:
status = app.mission.get_status()
print(f"Progress: {status['progress']:.1f}%")
time.sleep(5)
# Get results
result = app.mission.get_result()
print(f"Counted {result.total_detections} objects")Using the COCO dataset, the following 80 classes are supported:
Click to expand class list
| ID | Class | ID | Class | ID | Class |
|---|---|---|---|---|---|
| 0 | person | 27 | tie | 54 | donut |
| 1 | bicycle | 28 | suitcase | 55 | cake |
| 2 | car | 29 | frisbee | 56 | chair |
| 3 | motorcycle | 30 | skis | 57 | couch |
| 4 | airplane | 31 | snowboard | 58 | potted plant |
| 5 | bus | 32 | sports ball | 59 | bed |
| 6 | train | 33 | kite | 60 | dining table |
| 7 | truck | 34 | baseball bat | 61 | toilet |
| 8 | boat | 35 | baseball glove | 62 | tv |
| 14 | bird | 36 | skateboard | 63 | laptop |
| 15 | cat | 37 | surfboard | 64 | mouse |
| 16 | dog | 38 | tennis racket | 65 | remote |
| 17 | horse | 39 | bottle | 66 | keyboard |
| 18 | sheep | 40 | wine glass | 67 | cell phone |
| 19 | cow | 41 | cup | 68 | microwave |
| 20 | elephant | 42 | fork | 69 | oven |
| 21 | bear | 43 | knife | 70 | toaster |
| 22 | zebra | 44 | spoon | 71 | sink |
| 23 | giraffe | 45 | bowl | 72 | refrigerator |
For agricultural use cases, classes 18 (sheep) and 19 (cow) are particularly relevant.
Tested on Raspberry Pi Zero W2 with Edge TPU:
| Metric | Value |
|---|---|
| Inference time | ~25-40ms per frame |
| Frame rate | ~10-15 FPS |
| Detection accuracy | ~85% mAP (COCO) |
| Power consumption | ~2W |
| Document | Description |
|---|---|
| Architecture | System design and component overview |
| Setup Guide | Detailed installation instructions |
| API Reference | Complete REST API documentation |
| Functions Reference | Detailed function documentation |
Contributions are welcome! Please read our contributing guidelines before submitting pull requests.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
- ArduPilot - Open source autopilot
- Google AIY Vision Kit - Edge TPU hardware
- TensorFlow Lite - ML inference framework
- pymavlink - MAVLink Python library
Made with ❤️ for the ArduPilot community