feat(templates): Implement Gemini Computer Use templates for TypeScript and Python

[dprevoznik]

feat(templates): Implement Gemini Computer Use templates for TypeScript and Python

Summary

This PR introduces full-fledged Gemini Computer Use templates for both TypeScript and Python. The templates implement Google's Gemini 2.5 Computer Use model using Kernel's Computer Controls API for browser automation.

Changes

New Templates

• TypeScript: Complete rewrite of gemini-computer-use template with native Kernel integration
• Python: New gemini-computer-use template following the same architecture

Architecture

Both templates follow a modular design based on Google's computer-use-preview reference:

File	Description
index.ts / main.py	Entry point with Kernel app registration and action handler
loop.ts / loop.py	Gemini sampling loop - orchestrates model calls and action execution
session.ts / session.py	Browser session management with optional replay recording
tools/computer.ts / tools/computer.py	Maps Gemini actions to Kernel's Computer Controls API
tools/types/	Type definitions for Gemini function calls

Supported Actions

Action	Description
click_at	Click at coordinates (x, y)
hover_at	Move mouse to coordinates
type_text_at	Click and type text at coordinates
scroll_document	Scroll page (up/down/left/right)
scroll_at	Scroll at specific coordinates
navigate	Navigate to a URL
go_back / go_forward	Browser history navigation
key_combination	Press key combinations (e.g., "ctrl+c")
drag_and_drop	Drag from one point to another
wait_5_seconds	Wait action

Key Features

• Native Kernel Integration: Uses Kernel's Computer Controls API directly instead of third-party browser automation libraries
• Replay Recording: Optional video replay recording for debugging (paid plans only)
• Context Management: Intelligently manages screenshot history to stay within context limits
• Coordinate Normalization: Handles Gemini's normalized coordinates (0-1000) to actual screen dimensions
• Local Development: Both templates support local execution via npx tsx index.ts / direct Python execution

CLI Updates

• Added TemplateGeminiComputerUse constant and template metadata in pkg/create/templates.go
• Templates prioritized in selection list alongside Anthropic and OpenAI computer use templates

Usage

# Create a new project
kernel create my-app --language typescript --template gemini-computer-use

# Deploy
kernel deploy index.ts --env-file .env

# Invoke
kernel invoke ts-gemini-cua cua-task --payload '{"query": "Navigate to https://example.com and describe the page"}'

# With replay recording
kernel invoke ts-gemini-cua cua-task --payload '{"query": "...", "record_replay": true}'

Requirements

• GOOGLE_API_KEY - Google AI Studio

Testing

Templates have been manually tested with various browser automation tasks.

Related

• Closes KERNEL-870

---

Note

Adds native Gemini Computer Use support across both languages and aligns docs/tests/CLI metadata.

• New Python template python/gemini-computer-use with main.py, loop.py, session.py, and tools/ implementing Kernel Computer Controls and Gemini sampling loop
• TypeScript template rewrite typescript/gemini-computer-use: replaces Stagehand with native Kernel APIs; adds loop.ts, session.ts, tools/; updates index.ts, README, and dependencies
• CLI templates registry: marks GeminiComputerUse as available for Python and TypeScript; sets concrete invoke commands for both languages
• QA docs: adds py-gemini-cua to matrix, create/deploy steps, invoke commands; updates totals (18 apps/21 tests)
• Tests: removes "gemini-computer-use not available for python" cases and related unavailability checks

^{Written by Cursor Bugbot for commit 5e859dc. This will update automatically on new commits. Configure here.}

[dprevoznik]

🔧 CI Fix Available

I've pushed a fix for the CI failure. The tests were expecting gemini-computer-use to not be available for Python, but the template now supports both TypeScript and Python.

👉 Click here to create a PR with the fix

Handled in this commit 08ff1aa

[dprevoznik]

Conducted multiple agent reviews + deslop commands + bugbot fixes.

[dprevoznik]

Both templates for Gemini CUA worked very well in testing.

replays.9.mp4

[Sayan-]

overall lgtm. my comments aren't hard blocking but would prefer to address now.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}