feat: Watch gateway relay, genUI v0.9 fix, multi-device comms#1
Open
RoyGuanyu wants to merge 24 commits intodart-technologies:mainfrom
Open
feat: Watch gateway relay, genUI v0.9 fix, multi-device comms#1RoyGuanyu wants to merge 24 commits intodart-technologies:mainfrom
RoyGuanyu wants to merge 24 commits intodart-technologies:mainfrom
Conversation
- Enhanced WatchCompanion with dictation input and voice command sending
- Added SpeechRecognizer stub for watchOS (uses system dictation)
- Updated ConnectivityProvider with sendMessage + transferUserInfo fallback
- Updated AppDelegate to forward text-based voice commands to Flutter
- Enhanced WatchBridge dart class to support both file and text events
- Fixed pbxproj: product type, product name, group path for Xcode 26
- Removed legacy WKWatchKitApp plist key (keep WKApplication only)
- Relaxed SDK constraints for Flutter 3.35.2 / Dart 3.9.0 compatibility
Message format: {type: voice_command, text: ..., timestamp: ...}
Supports sendMessage (realtime) with transferUserInfo (background) fallback.
Watch -> iPhone -> Flutter via MethodChannel art.dart.clawfree/watch
Flutter -> Watch via sendReply for AI response display.
…stry - Add PressTalkButton: long-press to record, release to send (Telegram-style) - Add AudioRecorderService: m4a recording via record package (16kHz mono) - Add OpenClawClient: HTTP client for gateway STT, text chat, device registration - Add DeviceRegistry: device discovery/sync via gateway polling (no Firebase) - Add DeviceChips: visual display of connected devices - Integrate press-to-talk into ChatInputBar alongside existing STT - All platforms: iOS, iPad, macOS (Web falls back to STT) - flutter analyze: 0 issues
Phase 1 - Voice → genUI flow: - ChatScreen now uses VoiceController instead of raw SttService - STT results feed into ChatSession.sendMessage() → AI → A2UI → genUI - TTS readback already in A2uiStreamProcessor, now reflected in UI Phase 2 - Continuous interaction: - VoiceOrb shows speaking state (tertiary color + volume icon) - TTS completion auto-restarts listening when hands-free enabled - _toggleHandsFree wires VoiceController.setHandsFreeMode with wake word - Poll TTS.isSpeaking every 200ms to drive VoiceOrb animation Phase 3 - Watch voice relay: - Listen to WatchBridge.onVoiceReceived for text commands from Watch - Forward Watch voice text → ChatSession.sendMessage() → genUI - Send AI text replies back to Watch via WatchBridge.sendReplyToWatch Files changed: - chat_screen.dart: VoiceController integration, TTS polling, Watch bridge - phone_layout.dart: pass isSpeaking to VoiceOrb - voice_orb.dart: speaking state animation (pulse, glow, icon)
- Theme: Lobster Orange (#FF6B35) primary, Teal (#00BFA5) accent, dark backgrounds (#1A1A1A/#242424/#2A2A2A), default dark mode - Phone: suggestion chips on home dashboard empty state, teal accent for home session mode - Tablet: hover states on sidebar items for macOS, sidebar background color, animated transitions - QR scanner: replaced mobile_scanner with URL-input fallback to fix simulator build (MLKit lacks arm64 sim slices) - iOS: platform minimum bumped to 16.0, WatchCompanion target removed from scheme to unblock simulator builds - Podfile: post_install fixes EXCLUDED_ARCHS for Apple Silicon sims - Build verified: flutter analyze (0 issues) + iOS simulator build OK
- record ^5.1.0 → ^6.0.0（修復 record_linux 相容性問題） - 移除 record_linux dependency_overrides（不再需要） - 加回 WatchCompanion native target 到 project.pbxproj - watchOS 10.0+, Bundle ID: art.dart.clawfree.watchkitapp - 修正 WatchCompanion.xcscheme BlueprintIdentifier - macOS build ✓ / Watch build ✓ / flutter analyze 0 issues
- Theme: rich dark surface layers, Google Fonts (Space Grotesk + Inter), WCAG AA contrast, glow helpers - VoiceOrb: glassmorphism backdrop, breathing idle animation, wave bars visualization, 3-layer pulse rings - Chat bubbles: gradient AI backgrounds, TypingIndicator widget, subtle borders - Quick actions: press scale animation with glow effect - Connectivity bar: frosted glass backdrop blur - Pulsing dot: glow shadow effect
P0: Watch ↔ Gateway round-trip 已在 feat/voice-input 完成，本次確認完整。 P1: DeviceRegistry mock fallback - Gateway /v1/devices 回 404 時自動切換 local-only 模式 - local-only 模式下透過 WatchBridge.isWatchReachable 偵測 Watch - 自動新增 Apple Watch 裝置到 registry P2: 裝置管理 UI 頁面 - 新增 lib/src/ui/settings/devices_page.dart - 列出所有 ConnectedDevice（名稱/類型/狀態/最後上線時間） - dark theme 風格，狀態指示燈 - PhoneLayout 新增 Devices quick action 進入 - ChatScreen 初始化 DeviceRegistry 並連接導航
⌚ Watch 端新增： - WatchTTSService：AVSpeechSynthesizer 語音合成，自動偵測中/英文 - 收到 AI 回應後自動 TTS 念出 - PulseMonitorView 改為對話式 UI： - AI 回覆氣泡顯示 - Reply 按鈕（繼續 dictation 對話） - Replay 按鈕（重播 TTS） - 說話中顯示 speaker icon - 點擊 ring 可停止 TTS 或開始 dictation 📱 iPhone 端確認： - Watch 語音訊息已顯示在 chat UI（透過 _send） - Gateway 回應同時顯示在 chat + 轉發 Watch 分工：Watch = 語音 I/O + 文字 | iPhone = genUI 視覺顯示
支持 iOS 18+ 自動切換 Dark/Light/Tinted 模式： ✨ 新增功能： - AppIcon-Dark: 深色背景 (#1A1A1A) + 原龍蝦 + 微妙 orange glow - AppIcon (Light): 淺色背景 (#F5F5F5) + 龍蝦 + 立體陰影 - AppIcon-Tinted: 單色剪影 (brand orange #FF6B35) + 深色背景 🛠️ 技術實作： - Python PIL 處理圖片（白底移除、背景替換、濾鏡效果） - 生成所有 iOS 需要的尺寸 (20-1024pt, @1x-3x) - 更新 Xcode Assets Catalog 配置 © 2026 Rollbytes Inc. All rights reserved.
- messageStream → incomingMessages - textStream → incomingText - pubspec.yaml: pin genui ref to feature/v0.9-migration branch genUI v0.9 API 重命名 + 追蹤 migration 分支 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Stop current speech before starting new utterance - Add generation counter to prevent stale async handlers - Prevent isSpeaking flag flip after stop() is called 修復 TTS 競態條件，加入世代計數器防止狀態錯亂 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add WatchTTSService.swift to WatchCompanion Sources build phase - Add Embed Watch Content copy files phase (before Thin Binary) - Add Runner → WatchCompanion target dependency - Fix plist key: WKAppCompanionId → WKCompanionAppBundleIdentifier - Update add_watch_target.rb with embed phase and dependency logic 修復 Watch build：加入 TTS 檔案、嵌入階段、target 依賴、plist key Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add /watch/relay SSE endpoint to gateway (broadcast hub) - Upgrade WatchBridge: iPhone uses WCSession, iPad/macOS uses relay - iPhone listens to relay for iPad→Watch forwarding - ChatScreen configures WatchBridge with gateway URL on init - Add WATCH-COMMUNICATION.md architecture docs (EN + 繁中摘要) Watch 閘道中繼 + 多裝置通訊架構（iPhone↔iPad↔Watch） Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace sheet-based dictation with inline TextField so the watch app no longer dismisses after each send - Add Clawfree branding + CFBundleDisplayName to watch app - Add Assets.xcassets with AppIcon for watchOS launch icon - Buffer WCSession events in WatchEventStreamHandler when Flutter eventSink is nil (app backgrounded), flush on reconnect - Add didReceiveMessage fallback (no replyHandler) + unified handler - Keep keyboard focus after send in ChatInputBar Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Redesign Watch PulseMonitorView for voice-first interaction (one-tap dictation, fullScreenCover, auto re-trigger) - Add InputCoordinator for phone/watch input coordination (first-come-first-served + queuing) - Add WatchManagementPanel bottom sheet UI - Add Watch debug logging (WatchBridge, ConnectivityProvider, AppDelegate) - Add pingWatch, isWatchReachable method channel handlers - Add WatchState tracking in DeviceRegistry - Integrate watch status indicators in PhoneLayout - Add STT availability check before recording Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace TextField+dictation multi-step flow with single giant mic button - Main screen: 60%+ mic circle with pulsing animation (brand colors) - Tap → dictation sheet → auto-send on submit - Visual states: idle/recording/sending/reply with distinct colors - Reply bubble with tap-to-replay TTS - Brand colors: Lobster Orange #FF6B35, Teal #00BFA5, Dark bg #1A1A1A fix(a2ui): align surface manager with genUI transport adapter API - incomingMessages → messageStream - incomingText → textStream
- Add LocalSyncServer (port 8765) for iPhone→iPad/macOS broadcast - Add LocalSyncClient with exponential backoff reconnect - Add DeviceRoleDetector (iPhone=host, iPad/macOS=client) - Integrate sync into ChatScreen with device count badge - Add trip planning demo responses: create agent, 3-day Tokyo itinerary, sushi class modification, Tokyo weather card - Add NSLocalNetworkUsageDescription + Bonjour to Info.plist - All builds pass: analyze, iOS, macOS, watchOS
- WatchVoiceEvent: voice_command parsing, isTextCommand, toJson roundtrip - DemoCacheAiClient: trip planning, trip agent, sushi class responses - Self-correction guard, JSON single-chunk delivery - All 25 new tests pass, flutter analyze clean
新增兩個 Watch 互動場景： Scene A — 建 Agent： - 模型選擇（TabView 滑選：Opus 4.6 / Sonnet 4.5 / Gemini Pro） - 技能多選（Checkbox：機票/飯店/行程/天氣/餐廳） - 確認建立 → 傳送指令到 iPhone Scene B — 規劃旅行： - 目的地圖片橫滑（5 城市：Tokyo/Kyoto/Osaka/Seoul/Bangkok） - 天數選擇（3/5/7 天按鈕） - 景點 Checkbox 多選 - 摘要確認 → 傳送指令到 iPhone 兩個流程同時支援觸控和語音觸發（說 create agent / plan trip） 主畫面新增快捷按鈕：建 Agent + 規劃旅行
Watch 每步操作（選模型/選技能/選城市/選天數/勾景點） 即時透過 WCSession → iPhone → WebSocket 廣播到所有裝置。 iPhone/iPad/macOS 顯示 Watch 互動流程覆蓋層： - 進度條 + 當前步驟高亮 - 即時顯示 Watch 的選擇結果 - 建 Agent 和規劃旅行兩種流程 移除 AUTO_DEMO 依賴，改為手動操作觸發。
- 全英文介面轉換（Watch SwiftUI + Flutter Dart） - Watch TTS 語音引導（進入流程時自動提示） - Watch 語音→iPhone 輸入欄顯示並自動發送 - 四設備同步 E2E 測試（4/6 通過） Changes: - ios/WatchCompanion/*.swift: 中文→English UI - lib/src/ui/*.dart: 中文→English UI - test/integration/sync_e2e_test.dart: 新增同步測試
- Add isAutoDemoMode flag to PulseMonitorView.swift - Auto-play 3 demo scripts on Watch launch (no interaction needed) - Chain animations: recording pulse → typewriter → send → next - Add WatchCompanionUITests target scaffold - Auto-demo cycles: Create agent → Plan Tokyo trip → Add sushi class
royai-tw
force-pushed
the
feat/watch-gateway-roundtrip
branch
from
45e5d90 to
2f9c7be
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
messageStream→incomingMessages,textStream→incomingText; pin pubspec tofeature/v0.9-migrationbranchisSpeakingstateWatchTTSService.swiftto Xcode project, add Embed Watch Content phase + Runner→WatchCompanion dependency, fixWKCompanionAppBundleIdentifierplist key/watch/relayendpoint enables iPhone ↔ iPad ↔ Watch communication via gateway;WatchBridgeauto-detects WCSession (iPhone) vs relay (iPad/macOS)Commits (4)
fix: genUI v0.9 API rename + track migration branchfix: TTS race condition with generation counterfix: Watch build — target dependency, embed phase, Info.plist keyfeat: Watch gateway relay + multi-device communicationTest plan
🤖 Generated with Claude Code