🂡 AceSense: Wearable Agent for Augmented Card Gameplay with Mentra Live Glasses

📍 Mentra Live Hackathon @ Y Combinator in San Francisco — July 12–13, 2025
🥇 Best Project Use of Roboflow
🥇 Best Project Use of ElevenLabs

AceSense is an augmented reality (AR) card game agent built for Mentra Live's smartglasses via MentraOS. AceSense serves as a wearable coach for augmented interaction during gameplay. It detects your cards, calculates win probability using GPT‑3, and speaks strategic tips aloud via ElevenLabs—keeping your hands free and your eyes on the game.

⚙️ Tools & Technologies

Platform & Runtime

• MentraOS SDK — Provides camera control and AR UI on Mentra Live smart glasses
• Bun — Lightweight backend runtime for low-latency processing
• ngrok — Tunnels local backend to public URL for device integration

ML & AI

• Roboflow (YOLOv11 → RF-DETR) — Initialized with a pretrained YOLOv11 model on the Playing Cards Dataset
  • Applied ±45° shear augmentations to account for viewing angle distortions, doubling dataset to 20k images.
  • Fine-tuned using RF-DETR (a DETR-based transformer model) for improved bounding box accuracy under angular variation
• OpenAI GPT-3.5 (o3-mini) — Performs reasoning to estimate win probability and generate concise strategy tips
• ElevenLabs — Converts tips into natural-sounding audio, spoken aloud through the headset

Design & Planning

• Figma Board — Ideation and UI planning

🚀 How It Works

1. MentraOS captures an image of the player's hand
2. Image is sent to Roboflow for card detection (via YOLOv11 or RF-DETR)
3. Detected card values are passed to OpenAI's o3-mini for odds estimation and tip generation
4. ElevenLabs synthesizes the tip into David Attenborough's voice
5. Output is delivered directly through the Mentra Live on-ear speakers in real time.

🛠 Local Dev

# Start backend server
bun run dev

# Expose to MentraOS device
ngrok http --url=grouse-next-especially.ngrok-free.app 3000

You'll need API keys for:

• Roboflow
• OpenAI
• ElevenLabs
• MentraOS