Computer Vision–Based Autonomous Checkout System

PROJECT OVERVIEW

The Smart Cart System is an automated product recognition and checkout system designed for Raspberry Pi. It uses computer vision to identify products placed under a camera, automatically adds them to a digital cart, and provides a complete payment interface. The system consists of a Python backend for image processing and a web-based frontend for user interaction.

SYSTEM ARCHITECTURE

Hardware Components

1. Raspberry Pi - Main processing unit (Model 3B+ or higher recommended)
2. USB Camera - For product image capture (Logitech C920 or similar recommended)
3. Power Supply - 5V/3A for stable operation
4. Display - Optional HDMI monitor or touchscreen
5. LED strip light - Image recognition support
6. Network - Ethernet or Wi-Fi for web interface access

Software Stack

• Backend: Python 3.11 with OpenCV, TensorFlow Lite, WebSockets
• Frontend: HTML5, CSS3, JavaScript with WebSocket API
• Model: TensorFlow Lite INT8 quantized (96x96x3 input)
• Communication: WebSocket protocol on port 8765
• Web Server: Python HTTP server on port 8000

HARDWARE SETUP

Minimum Requirements

• Raspberry Pi with 2GB RAM
• 16GB microSD card (Class 10 or higher)
• USB Webcam (720p minimum, 1080p recommended)
• Stable 5V/3A power supply
• LED strip light
• Active cooling (recommended for sustained use)

Camera Positioning Guidelines

1. Height: 30-40cm above product surface
2. Angle: 90 degrees (directly overhead)
3. Lighting: Even, diffused lighting without shadows
4. Background: Consistent, non-reflective surface
5. Focus: Fixed focus set to product distance

Power Considerations

• Use official Raspberry Pi power supply
• Avoid powering from computer USB ports
• Consider UPS for retail environments

SOFTWARE INSTALLATION

Step-by-Step Installation

1. Update System

sudo apt update
sudo apt upgrade -y

2. Install System Dependencies

sudo apt install -y python3-pip python3-venv
sudo apt install -y libatlas-base-dev libhdf5-dev libhdf5-serial-dev
sudo apt install -y libopenjp2-7 libilmbase25 libopenexr25

3. Create Virtual Environment

cd ~/smart_cart
python3 -m venv smartcart_env
source smartcart_env/bin/activate

4. Install Python Packages

pip install opencv-python numpy websockets
pip install "numpy<2"  # Required for TensorFlow Lite compatibility
pip install tflite-runtime  # Or use: pip install tensorflow

5. Verify Installation

python3 -c "import cv2; print('OpenCV:', cv2.__version__)"
python3 -c "import numpy; print('NumPy:', numpy.__version__)"
python3 -c "import websockets; print('WebSockets OK')"

MODEL PREPARATION

Model Specifications

• Model: Mobilenetv2
• Format: TensorFlow Lite (.tflite)
• Quantization: INT8 (optimized for Raspberry Pi)
• Input Size: 96x96 pixels
• Channels: 3 (RGB)
• Classes: 4 (bru, cake, dairymilk, lays) (scale depending upon your usecase)
• File Size: ~2-5MB typical

Model Placement

Place your trained model at:

/home/project/smart_cart/model.tflite

Model Training Guidelines

1. Dataset: Minimum 100 images per class
2. Lighting: Consistent with deployment environment
3. Angles: Multiple product orientations
4. Background: Consistent plain background
5. Augmentation: Rotation, scaling, brightness variation

CONFIGURATION

Backend Configuration (backend.py)

# Essential configuration section
MODEL_PATH = "/home/project/smart_cart/model.tflite"
CLASS_NAMES = ["bru", "cake", "dairymilk", "lays"]
CAM_INDEX = 0  # USB camera index
WEBSOCKET_PORT = 8765

# Camera settings (optional optimization)
CAPTURE_DELAY = 0.3  # Seconds for camera adjustment
BUFFER_CLEAR_COUNT = 5  # Frames to discard for fresh capture

Frontend Configuration (index.html)

1. WebSocket Connection

// Line 383 in JavaScript section
ws = new WebSocket('ws://localhost:8765');
// Change to Raspberry Pi IP if accessing from another device:
// ws = new WebSocket('ws://192.168.1.100:8765');

2. Product Database

// Line 257 in JavaScript section
const productDatabase = [
    { id: "bru", name: "Bru", price: 10.00, image: "bru.jpg" },
    { id: "cake", name: "Cake", price: 10.00, image: "cake.jpg" },
    { id: "dairymilk", name: "Dairy Milk", price: 10.00, image: "dairymilk.jpg" },
    { id: "lays", name: "Lays", price: 10.00, image: "lays.jpg" }
];

DEPLOYMENT

Starting the System

Terminal 1 - Backend

cd ~/smart_cart
source smartcart_env/bin/activate
python3 backend.py

Terminal 2 - Frontend

cd ~/smart_cart/frontend
python3 -m http.server 8000

NETWORK ACCESS

Local Access

• URL: http://localhost:8000
• Device: Same Raspberry Pi

Network Access

1. Find Raspberry Pi IP:

hostname -I

2. Access from other devices:

http://[raspberry-pi-ip]:8000

Port Configuration

• Frontend: Port 8000 (HTTP)
• Backend: Port 8765 (WebSocket)

If ports conflict:

# Change in backend.py
WEBSOCKET_PORT = 8766

# Change in frontend JavaScript
ws = new WebSocket('ws://localhost:8766');

OPERATION GUIDE

Starting the System

1. Power on Raspberry Pi
2. Connect camera and ensure proper positioning
3. Open terminal and start backend
4. Open second terminal and start frontend server
5. Open browser to access interface

Product Scanning Workflow

1. Place Product: Position product centered under camera
2. Click Scan: Press "Scan Product" button
3. Wait Processing: System captures and classifies (2-3 seconds)
4. Review Result: Check product name and confidence score
5. Add to Cart: Click "Add to Cart" or scan next product

Cart Management

• Add Items: Scan products or click from product list
• Adjust Quantity: Use +/- buttons for each item
• Remove Items: Click trash icon
• View Total: Check bottom of cart section

Checkout Process

1. Proceed to Payment: Click when cart has items
2. Select Method: Choose card, wallet, or cash
3. Enter Details: Fill payment form
4. Confirm: Complete payment
5. Print Receipt: Optional print function
6. New Trip: Start fresh shopping session

Model Accuracy Improvement

If Model Misclassifies

1. Retrain with more varied data
2. Adjust camera lighting
3. Improve product positioning
4. Add data augmentation during training
5. Increase training epochs

Confidence Threshold

Adjust in frontend if needed:

// Current: 85% default
const confidence = data.confidence || 0.85;
// Change to: 0.90 for stricter matching

MAINTENANCE

Daily Checks

1. Camera Cleanliness: Lens free from dust/smudges
2. Lighting Consistency: Check for shadows/changes
3. Network Connectivity: Verify WebSocket connection
4. System Temperature: Monitor Pi temperature
5. Storage Space: Check available disk space

Weekly Tasks

1. Update Software: Security patches and updates
2. Backup Configuration: Save model and code
3. Clean Logs: Remove old log files
4. Test All Functions: Full system test
5. Check Hardware: Camera mount stability

Monthly Maintenance

1. Model Retraining: If accuracy decreases
2. System Reboot: Clear memory and cache
3. Hardware Inspection: Check cables and connections
4. Performance Review: Check inference times
5. Backup Creation: Full system backup

SECURITY CONSIDERATIONS

Current Limitations

1. No Authentication: Anyone on network can access
2. HTTP Only: No encryption for data transfer
3. Local Network: Not designed for internet access
4. No User Accounts: Single shared cart

For Production Use

1. Add HTTPS: Use SSL certificates
2. Implement Authentication: User login system
3. Network Segmentation: Isolate on separate VLAN
4. Regular Updates: Security patch management
5. Firewall Rules: Restrict access to necessary ports

SCALING AND EXTENSIONS

Adding More Products

1. Retrain Model: Include new classes
2. Update CLASS_NAMES: Add to backend list
3. Update productDatabase: Add to frontend
4. Test Thoroughly: Verify all products work

Multiple Camera Support

# For multiple checkout lanes
CAM_INDEX = 0  # Lane 1
# Additional backend instances on different ports

Database Integration

Potential extensions:

1. Inventory Management: Track stock levels
2. Sales Analytics: Collect purchase data
3. User Profiles: Save shopping preferences
4. Receipt Database: Store transaction history

PERFORMANCE METRICS

Expected Performance

• Inference Time: 1-3 seconds per product
• Accuracy: >95% with good training data
• Memory Usage: 200-300MB for backend
• CPU Usage: 50-70% during operation
• Network Latency: <100ms for WebSocket

Monitoring Commands

# Check system resources
htop
# Monitor temperature
vcgencmd measure_temp
# Check network
iftop
# Monitor processes
top -u $USER

System Version: 1.0.0 Last Updated: November 2025 Compatibility: Raspberry Pi OS Bookworm/Bullseye Required Skills: Basic Linux, Python, Web Development