Skip to content

feat: Add "Neon Sketch" drawing tool and improve UI #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Josephaswisher wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: Add "Neon Sketch" drawing tool and improve UI #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions runtime/backend/gemini-agent.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,12 @@ CRITICAL CONSTRAINTS:
4. Never remove, hide, or disable the command input text box rendered by CommandInput.
5. When embedding multi-line ASCII art or code, wrap it in a <pre> element and call String.raw on the string literal so the JSX remains valid without escape errors.

TOOLS:
- You have a special "drawing" tool available.
- If the user asks to draw something, use the drawing tool.
- The tool takes a list of drawing commands.
- Schema: { tool: 'drawing', commands: [{ shape: 'line' | 'circle' | 'rectangle', color: string (hex), positions: [x, y, x2, y2, ...] }] }

REQUEST:
The user said: "${userCommand}".
Translate this into a compelling end product in GeneratedContent.tsx only. Remember you can create anything that can be rendered in a browser!
Expand Down Expand Up @@ -325,6 +331,8 @@ Please apply the requested update and include DONE at the very end.`;
promptWithContext,
'--output-format',
'json',
'--tools',
'drawing',
];

return new Promise((resolve, reject) => {
Expand Down
52 changes: 52 additions & 0 deletions runtime/frontend/src/App.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,3 +92,55 @@ body, html {
.video-window__caption a:focus-visible {
text-decoration: underline;
}

.error-window {
padding: 16px;
}

.error-window h3 {
color: #c93434;
margin: 0 0 8px;
font-size: 16px;
}

.error-window p {
color: #444;
font-size: 13px;
line-height: 1.5;
margin: 0;
}

.error-window-details {
background: rgba(201, 52, 52, 0.12);
border: 1px solid rgba(201, 52, 52, 0.2);
padding: 12px;
border-radius: 8px;
margin-top: 12px;
font-size: 12px;
line-height: 1.4;
color: #333;
}

.error-window-details strong {
display: block;
font-weight: 600;
margin-bottom: 4px;
}

.error-window-details p {
font-size: 12px;
}

.error-window-details p:not(:last-child) {
margin-bottom: 8px;
}

.error-window-details code {
font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier,
monospace;
background: rgba(201, 52, 52, 0.1);
padding: 2px 4px;
border-radius: 4px;
white-space: pre-wrap;
word-break: break-word;
}
85 changes: 59 additions & 26 deletions runtime/frontend/src/App.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import type { WindowData } from './components/Desktop';
import CommandInput from './components/CommandInput';
import GeneratedContent from './components/GeneratedContent';
import GeminiStatsWindow from './components/GeminiStatsWindow';
import DrawingPadApp from './components/DrawingPadApp';
import DrawingPadApp, { type DrawingCommand } from './components/DrawingPadApp';
import './App.css';

const API_BASE_URL = 'http://localhost:3001';
Expand Down Expand Up @@ -97,21 +97,32 @@ function App() {
return () => clearInterval(interval);
}, [refreshStatus]);

const createCommandWindow = useCallback((request: string) => {
const newWindow: WindowData = {
id: `window-${Date.now()}`,
title: `Request: ${truncate(request, 24)}`,
content: (
<div>
<h3>Preview</h3>
<p>Request: {request}</p>
<p>Gemini will replace this window with a tailored experience.</p>
</div>
),
};

setWindows((prev) => [...prev, newWindow]);
}, []);
const createErrorWindow = useCallback(
(request: string, errorMessage: string) => {
const newWindow: WindowData = {
id: `error-window-${Date.now()}`,
title: `Error: ${truncate(request, 24)}`,
content: (
<div className="error-window">
<h3>Agent Request Failed</h3>
<p>Your request could not be completed. See details below.</p>
<div className="error-window-details">
<strong>Command:</strong>
<p>
<code>{request}</code>
</p>
<strong>Error:</strong>
<p>
<code>{errorMessage}</code>
</p>
</div>
</div>
),
};
setWindows((prev) => [...prev, newWindow]);
},
[],
);

const handleCloseWindow = useCallback((id: string) => {
setWindows((prev) => prev.filter((window) => window.id !== id));
Expand Down Expand Up @@ -187,14 +198,17 @@ function App() {
});
}, [openStaticWindow]);

const handleOpenDrawingPad = useCallback(() => {
openStaticWindow({
id: 'drawing-pad',
title: 'Neon Sketch',
content: <DrawingPadApp />,
position: { x: 520, y: 160 },
});
}, [openStaticWindow]);
const handleOpenDrawingPad = useCallback(
(initialCommands?: DrawingCommand[]) => {
openStaticWindow({
id: 'drawing-pad',
title: 'Neon Sketch',
content: <DrawingPadApp initialCommands={initialCommands} />,
position: { x: 520, y: 160 },
});
},
[openStaticWindow],
);

const handleOpenUsageStats = useCallback(() => {
openStaticWindow({
Expand Down Expand Up @@ -236,6 +250,20 @@ function App() {
const mode =
(data.mode as AgentMode) ?? agentStatus.agentMode ?? 'UNKNOWN';

if (data.result?.output) {
try {
const agentJson = JSON.parse(data.result.output);
if (
agentJson.tool === 'drawing' &&
Array.isArray(agentJson.commands)
) {
handleOpenDrawingPad(agentJson.commands);
}
} catch (e) {
console.warn('Could not parse agent JSON output', e);
}
}

setLastActivity({
command,
message: data.message || 'Command processed.',
Expand All @@ -259,14 +287,19 @@ function App() {
: null,
timestamp: Date.now(),
});
createCommandWindow(command);
createErrorWindow(command, message);
} finally {
setIsProcessing(false);
setPendingCommand(null);
refreshStatus();
}
},
[agentStatus.agentMode, createCommandWindow, refreshStatus],
[
agentStatus.agentMode,
createErrorWindow,
handleOpenDrawingPad,
refreshStatus,
],
);

const commandHint = useMemo(() => {
Expand Down
12 changes: 12 additions & 0 deletions runtime/frontend/src/components/CommandInput.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,18 @@
margin: 0;
}

.command-input-status.pending .loading-indicator {
display: inline-block;
width: 12px;
height: 12px;
border: 2px solid rgba(255, 255, 255, 0.4);
border-top-color: #fff;
border-radius: 50%;
animation: spin 1s linear infinite;
margin-right: 8px;
vertical-align: middle;
}

@keyframes command-submit-spin {
to {
transform: rotate(360deg);
Expand Down
1 change: 1 addition & 0 deletions runtime/frontend/src/components/CommandInput.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ export default function CommandInput({
<div
className={`command-status-label command-status-label--${statusTone}`}
>
{statusTone === 'pending' && <div className="loading-indicator" />}
<span>{statusMessage}</span>
{authError && (
<span className="command-status-label__error">{authError}</span>
Expand Down
136 changes: 48 additions & 88 deletions runtime/frontend/src/components/DrawingPadApp.css
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,112 +1,72 @@
.drawing-pad {
/**
* @license
* Copyright 2025 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/

.drawing-pad-app {
display: flex;
flex-direction: column;
gap: 16px;
min-width: 420px;
max-width: 680px;
}

.drawing-pad__header h3 {
margin: 0 0 4px;
font-size: 20px;
background-color: #222;
border-radius: 8px;
padding: 16px;
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.5);
}

.drawing-pad__header p {
margin: 0;
color: #4e4c63;
font-size: 14px;
.drawing-canvas {
background-color: #000;
border-radius: 4px;
cursor: crosshair;
touch-action: none;
}

.drawing-pad__toolbar {
.drawing-controls {
display: flex;
flex-wrap: wrap;
justify-content: space-between;
align-items: center;
gap: 14px;
padding: 12px 16px;
border: 1px solid #dad6ff;
border-radius: 12px;
background: linear-gradient(135deg, rgba(102, 126, 234, 0.1), rgba(118, 75, 162, 0.1));
box-shadow: inset 0 0 0 1px rgba(255, 255, 255, 0.5);
margin-top: 12px;
}

.drawing-pad__tool {
.color-palette {
display: flex;
align-items: center;
gap: 8px;
font-size: 12px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.08em;
color: #2d1b69;
}

.drawing-pad__tool input[type='range'] {
accent-color: #704bff;
width: 130px;
}

.drawing-pad__tool input[type='color'] {
width: 42px;
.color-swatch {
width: 28px;
height: 28px;
border: none;
border-radius: 6px;
padding: 0;
background: transparent;
}

.drawing-pad__value {
padding: 2px 8px;
border-radius: 999px;
background: rgba(45, 27, 105, 0.08);
color: #2d1b69;
font-size: 12px;
}

.drawing-pad__clear {
margin-left: auto;
padding: 8px 14px;
border-radius: 999px;
border: 1px solid rgba(45, 27, 105, 0.25);
background: rgba(255, 255, 255, 0.9);
color: #2d1b69;
font-weight: 600;
letter-spacing: 0.05em;
border-radius: 50%;
border: 2px solid #fff;
cursor: pointer;
transition: transform 0.1s ease, box-shadow 0.1s ease;
transition:
transform 0.2s,
box-shadow 0.2s;
}

.drawing-pad__clear:hover {
transform: translateY(-1px);
box-shadow: 0 4px 8px rgba(45, 27, 105, 0.1);
.color-swatch:hover {
transform: scale(1.1);
}

.drawing-pad__canvas {
width: 100%;
height: 420px;
border-radius: 14px;
border: 2px solid rgba(45, 27, 105, 0.25);
background: repeating-linear-gradient(
0deg,
rgba(126, 107, 197, 0.04),
rgba(126, 107, 197, 0.04) 24px,
rgba(255, 255, 255, 0.4) 24px,
rgba(255, 255, 255, 0.4) 48px
),
#ffffff;
cursor: crosshair;
.color-swatch.active {
box-shadow: 0 0 8px 2px var(--swatch-color);
transform: scale(1.15);
}

@media (max-width: 600px) {
.drawing-pad {
min-width: 0;
}

.drawing-pad__toolbar {
flex-direction: column;
align-items: flex-start;
}
.clear-button {
background-color: #444;
color: #fff;
border: 1px solid #666;
border-radius: 4px;
padding: 8px 16px;
font-family: 'VT323', monospace;
font-size: 16px;
cursor: pointer;
transition:
background-color 0.2s,
box-shadow 0.2s;
}

.drawing-pad__clear {
margin-left: 0;
}
.clear-button:hover {
background-color: #555;
box-shadow: 0 0 8px rgba(255, 255, 255, 0.3);
}
Loading