NeuroDraw

Hand Gesture Drawing (HCI Experiment)

An HTML5 web app for touch-free drawing using the front camera, real-time hand tracking, and simple, non-conflicting gestures. This project explores human–computer interaction (HCI) and future interface design through natural hand movements.

Why this project

Pointing, pinching, and posing are innate human actions. Translating them directly into a digital canvas can reduce friction and suggest a vision for post-touch interfaces. This prototype investigates:

• Natural mapping from hand motion to inking
• Minimal gesture vocabulary with low conflict
• Always-on visual feedback to improve user confidence

Features

• Front camera access with mirrored view for intuitive alignment
• Real-time hand tracking (21 landmarks per hand)
• Always-on overlay: hand skeleton, joints, and labeled fingertips
• Pinch-to-draw with brush and eraser tools
• Gesture controls (non-conflicting):
  • Pinch (thumb + index): Draw while held
  • Thumbs-up: Toggle brush/eraser
  • Pinch (thumb + pinky): Cycle brush size presets
• Toolbar: size slider, undo, clear, save (PNG)

Gestures (final)

• Pinch (thumb + index) to draw while held
• Thumbs-up (hold ~300ms) to toggle tool (brush/eraser)
• Pinky pinch to change brush size

Note: The color-change gesture was intentionally removed to avoid conflicts and simplify the vocabulary.

Run locally

Serve over HTTP(S) to enable camera access in the browser.

# Python
py -m http.server 5500
# or Node
npx serve -p 5500

Open http://localhost:5500 and allow camera permission.

Technical overview

• Camera: MediaDevices.getUserMedia with facingMode: user for front camera and a mirrored transform for video and canvas.
• Tracking: MediaPipe Hands (on-device ML) provides 21 3D landmarks per hand in real time.
• Drawing: HTML5 Canvas with round caps/joins; eraser uses destination-out blending.
• Gestures:
  • Pinch detection uses thumb-to-fingertip distance vs. palm width threshold.
  • Thumbs-up uses a simple heuristic: thumb extended and above the wrist, other fingers folded.
• Overlay: renders skeleton, joints, and fingertip labels for immediate feedback.

Design rationale

• Keep the gesture set small and orthogonal to minimize misfires.
• Require a short hold for thumbs-up and gate it when pinching to disambiguate from drawing.
• Prefer hold-to-act (pinch-draw) for continuous operations; use pose (thumbs-up) for mode toggle.
• Provide constant visual feedback to build trust and aid discoverability.

Research context and references

• Gesture recognition overview explains interpreting human hand gestures via algorithms and ML, including applications across interfaces and accessibility. See Wikipedia: Gesture recognition (https://en.wikipedia.org/wiki/Gesture_recognition).
• On-device real‑time hand gesture systems demonstrate feasibility for interactive apps without specialized hardware; see “On-device Real-time Hand Gesture Recognition” on arXiv (https://arxiv.org/abs/2111.00038).
• MediaPipe Hands (Google Research) describes a real-time, cross‑platform hand and finger tracking solution suitable for web and mobile HCI prototypes.
• Camera access and front-camera selection: see MDN for MediaDevices.getUserMedia and facingMode constraints (https://developer.mozilla.org/docs/Web/API/MediaDevices/getUserMedia).

This prototype aligns with HCI goals of natural, low-latency interaction. The simplified gesture vocabulary is designed to reduce ambiguity and cognitive load while maintaining expressiveness for drawing.

Files

• index.html – layout, toolbars, script includes
• styles.css – minimal, high-contrast UI with glass panel toolbar
• app.js – camera init, hand tracking, gesture parsing, drawing engine, overlay

Notes

• Best in Chrome. Use good lighting and a clear background for higher tracking confidence.
• HTTPS is required for camera on most mobile browsers.
• Tune sensitivity in app.js by adjusting pinchThreshold and the thumbs‑up heuristic.

License

MIT