Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ MimiClaw turns a tiny ESP32-S3 board into a personal AI assistant. Plug it into

![](assets/mimiclaw.png)

You send a message on Telegram. The ESP32-S3 picks it up over WiFi, feeds it into an agent loop — the LLM thinks, calls tools, reads memory — and sends the reply back. Supports both **Anthropic (Claude)** and **OpenAI (GPT)** as providers, switchable at runtime. Everything runs on a single $5 chip with all your data stored locally on flash.
You send a message on Telegram. The ESP32-S3 picks it up over WiFi, feeds it into an agent loop — the LLM thinks, calls tools, reads memory — and sends the reply back. Supports **Anthropic (Claude)**, **OpenAI (GPT)**, **OpenRouter**, and **NVIDIA NIM** as providers, switchable at runtime. Everything runs on a single $5 chip with all your data stored locally on flash.

## Quick Start

Expand All @@ -40,7 +40,7 @@ You send a message on Telegram. The ESP32-S3 picks it up over WiFi, feeds it int
- An **ESP32-S3 dev board** with 16 MB flash and 8 MB PSRAM (e.g. Xiaozhi AI board, ~$10)
- A **USB Type-C cable**
- A **Telegram bot token** — talk to [@BotFather](https://t.me/BotFather) on Telegram to create one
- An **Anthropic API key** — from [console.anthropic.com](https://console.anthropic.com), or an **OpenAI API key** — from [platform.openai.com](https://platform.openai.com)
- An **API key** — from [Anthropic](https://console.anthropic.com), [OpenAI](https://platform.openai.com), [OpenRouter](https://openrouter.ai), or [NVIDIA NIM](https://org.ngc.nvidia.com/setup/api-keys)

### Install

Expand Down Expand Up @@ -128,7 +128,7 @@ Edit `main/mimi_secrets.h`:
#define MIMI_SECRET_WIFI_PASS "YourWiFiPassword"
#define MIMI_SECRET_TG_TOKEN "123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11"
#define MIMI_SECRET_API_KEY "sk-ant-api03-xxxxx"
#define MIMI_SECRET_MODEL_PROVIDER "anthropic" // "anthropic" or "openai"
#define MIMI_SECRET_MODEL_PROVIDER "anthropic" // "anthropic", "openai", "openrouter", or "nvidia"
#define MIMI_SECRET_SEARCH_KEY "" // optional: Brave Search API key
#define MIMI_SECRET_TAVILY_KEY "" // optional: Tavily API key (preferred)
#define MIMI_SECRET_PROXY_HOST "" // optional: e.g. "10.0.0.1"
Expand Down Expand Up @@ -168,8 +168,8 @@ Connect via serial to configure or debug. **Config commands** let you change set
```
mimi> wifi_set MySSID MyPassword # change WiFi network
mimi> set_tg_token 123456:ABC... # change Telegram bot token
mimi> set_api_key sk-ant-api03-... # change API key (Anthropic or OpenAI)
mimi> set_model_provider openai # switch provider (anthropic|openai)
mimi> set_api_key sk-ant-api03-... # change API key (Anthropic, OpenAI, OpenRouter, or NVIDIA NIM)
mimi> set_model_provider openai # switch provider (anthropic|openai|openrouter|nvidia)
mimi> set_model gpt-4o # change LLM model
mimi> set_proxy 127.0.0.1 7897 # set HTTP proxy
mimi> clear_proxy # remove proxy
Expand Down Expand Up @@ -250,7 +250,7 @@ MimiClaw stores everything as plain text files you can read and edit:

## Tools

MimiClaw supports tool calling for both Anthropic and OpenAI — the LLM can call tools during a conversation and loop until the task is done (ReAct pattern).
MimiClaw supports tool calling for Anthropic, OpenAI, OpenRouter, and NVIDIA NIM — the LLM can call tools during a conversation and loop until the task is done (ReAct pattern).

| Tool | Description |
|------|-------------|
Expand Down Expand Up @@ -280,10 +280,10 @@ This turns MimiClaw into a proactive assistant — write tasks to `HEARTBEAT.md`
- **OTA updates** — flash new firmware over WiFi, no USB needed
- **Dual-core** — network I/O and AI processing run on separate CPU cores
- **HTTP proxy** — CONNECT tunnel support for restricted networks
- **Multi-provider** — supports both Anthropic (Claude) and OpenAI (GPT), switchable at runtime
- **Multi-provider** — supports Anthropic (Claude), OpenAI (GPT), OpenRouter, and NVIDIA NIM, switchable at runtime
- **Cron scheduler** — the AI can schedule its own recurring and one-shot tasks, persisted across reboots
- **Heartbeat** — periodically checks a task file and prompts the AI to act autonomously
- **Tool use** — ReAct agent loop with tool calling for both providers
- **Tool use** — ReAct agent loop with tool calling for all providers

## For Developers

Expand Down
27 changes: 18 additions & 9 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,12 +54,13 @@ Telegram App (User)
│ └──────────────────────────────────────────┘ │
└───────────────────────────────────────────────────┘
│ Anthropic Messages API (HTTPS)
│ Anthropic/OpenAI/OpenRouter/NVIDIA NIM API (HTTPS)
│ + Brave Search API (HTTPS)
┌───────────┐ ┌──────────────┐
│ Claude API │ │ Brave Search │
└───────────┘ └──────────────┘
┌──────────────┐ ┌──────────────┐
│ LLM Provider │ │ Brave Search │
│ (API Gateway)│ │ │
└──────────────┘ └──────────────┘
```

---
Expand All @@ -75,7 +76,7 @@ Telegram App (User)
b. Build system prompt (SOUL.md + USER.md + MEMORY.md + recent notes + tool guidance)
c. Build cJSON messages array (history + current message)
d. ReAct loop (max 10 iterations):
i. Call Claude API via HTTPS (non-streaming, with tools array)
i. Call LLM provider API via HTTPS (non-streaming, with tools array)
ii. Parse JSON response → text blocks + tool_use blocks
iii. If stop_reason == "tool_use":
- Execute each tool (e.g. web_search → Brave Search API)
Expand Down Expand Up @@ -158,7 +159,7 @@ main/
| Task | Core | Priority | Stack | Description |
|--------------------|------|----------|--------|--------------------------------------|
| `tg_poll` | 0 | 5 | 12 KB | Telegram long polling (30s timeout) |
| `agent_loop` | 1 | 6 | 12 KB | Message processing + Claude API call |
| `agent_loop` | 1 | 6 | 12 KB | Message processing + LLM provider API call |
| `outbound` | 0 | 5 | 8 KB | Route responses to Telegram / WS |
| `serial_cli` | 0 | 3 | 4 KB | USB serial console REPL |
| httpd (internal) | 0 | 5 | — | WebSocket server (esp_http_server) |
Expand Down Expand Up @@ -232,7 +233,7 @@ All configuration is done exclusively through `mimi_secrets.h` at build time. Th
| `MIMI_SECRET_WIFI_SSID` | WiFi SSID |
| `MIMI_SECRET_WIFI_PASS` | WiFi password |
| `MIMI_SECRET_TG_TOKEN` | Telegram Bot API token |
| `MIMI_SECRET_API_KEY` | Anthropic API key |
| `MIMI_SECRET_API_KEY` | LLM API key (Anthropic, OpenAI, OpenRouter, or NVIDIA NIM) |
| `MIMI_SECRET_MODEL` | Model ID (default: claude-opus-4-6) |
Comment on lines +236 to 237
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Configuration section still conflicts with runtime override behavior.

Around this section, Line 229 states there is no runtime configuration, but current behavior uses CLI + NVS overrides. This will mislead operators.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/ARCHITECTURE.md` around lines 236 - 237, The docs claim there is "no
runtime configuration" but the code actually supports runtime overrides via CLI
and NVS; update the ARCHITECTURE.md text around the configuration section so it
no longer asserts "no runtime configuration" and instead documents that
environment variables like MIMI_SECRET_API_KEY and MIMI_SECRET_MODEL can be
overridden at runtime using the CLI and NVS (include brief notes on precedence
and default model value claude-opus-4-6); ensure the table entries for
MIMI_SECRET_API_KEY and MIMI_SECRET_MODEL remain but add a sentence referencing
the supported runtime overrides (CLI/NVS) and the override precedence.

| `MIMI_SECRET_PROXY_HOST` | HTTP proxy hostname/IP (optional) |
| `MIMI_SECRET_PROXY_PORT` | HTTP proxy port (optional) |
Expand Down Expand Up @@ -278,7 +279,15 @@ Client `chat_id` is auto-assigned on connection (`ws_<fd>`) but can be overridde

---

## Claude API Integration
## LLM Provider Integration

MimiClaw supports multiple LLM providers via a unified API proxy:
- **Anthropic** (`anthropic`) - Native Messages API
- **OpenAI** (`openai`) - Chat Completions API
- **OpenRouter** (`openrouter`) - OpenAI-compatible endpoint
- **NVIDIA NIM** (`nvidia`) - OpenAI-compatible endpoint

### Anthropic API Format

Endpoint: `POST https://api.anthropic.com/v1/messages`

Expand Down Expand Up @@ -388,7 +397,7 @@ The CLI provides debug and maintenance commands only. All configuration is done
| `session/manager.py` | `memory/session_mgr.c` | JSONL per chat, ring buffer |
| `channels/telegram.py` | `telegram/telegram_bot.c` | Raw HTTP, no python-telegram-bot |
| `bus/events.py` + `queue.py`| `bus/message_bus.c` | FreeRTOS queues vs asyncio |
| `providers/litellm_provider.py` | `llm/llm_proxy.c` | Direct Anthropic API only |
| `providers/litellm_provider.py` | `llm/llm_proxy.c` | Multi-provider LLM API support |
| `config/schema.py` | `mimi_config.h` + `mimi_secrets.h` | Build-time secrets only |
| `cli/commands.py` | `cli/serial_cli.c` | esp_console REPL |
| `agent/tools/*` | `tools/tool_registry.c` + `tool_web_search.c` | web_search via Brave API |
Expand Down
2 changes: 1 addition & 1 deletion main/cli/serial_cli.c
Original file line number Diff line number Diff line change
Expand Up @@ -887,7 +887,7 @@ esp_err_t serial_cli_init(void)
esp_console_cmd_register(&model_cmd);

/* set_model_provider */
provider_args.provider = arg_str1(NULL, NULL, "<provider>", "Model provider (anthropic|openai)");
provider_args.provider = arg_str1(NULL, NULL, "<provider>", "Model provider (anthropic|openai|openrouter|nvidia)");
provider_args.end = arg_end(1);
esp_console_cmd_t provider_cmd = {
.command = "set_model_provider",
Expand Down
44 changes: 36 additions & 8 deletions main/llm/llm_proxy.c
Original file line number Diff line number Diff line change
Expand Up @@ -187,19 +187,47 @@ static bool provider_is_openai(void)
return strcmp(s_provider, "openai") == 0;
}

static bool provider_is_openrouter(void)
{
return strcmp(s_provider, "openrouter") == 0;
}

static bool provider_is_nvidia(void)
{
return strcmp(s_provider, "nvidia") == 0;
}
Comment on lines +195 to +198
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Preserve backward compatibility for prior nvidia_nim provider values.

Line 149 only matches "nvidia". If NVS/build-time config still contains "nvidia_nim", routing falls back to Anthropic URL/headers and requests fail.

🔧 Suggested fix
 static bool provider_is_nvidia(void)
 {
-    return strcmp(s_provider, "nvidia") == 0;
+    return strcmp(s_provider, "nvidia") == 0 ||
+           strcmp(s_provider, "nvidia_nim") == 0; /* backward-compatible alias */
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
static bool provider_is_nvidia(void)
{
return strcmp(s_provider, "nvidia") == 0;
}
static bool provider_is_nvidia(void)
{
return strcmp(s_provider, "nvidia") == 0 ||
strcmp(s_provider, "nvidia_nim") == 0; /* backward-compatible alias */
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@main/llm/llm_proxy.c` around lines 147 - 150, The provider_is_nvidia function
only checks for "nvidia" and breaks compatibility with legacy "nvidia_nim";
update provider_is_nvidia to treat both "nvidia" and "nvidia_nim" as NVIDIA
providers (e.g., check strcmp(s_provider, "nvidia")==0 || strcmp(s_provider,
"nvidia_nim")==0 or test a prefix like strncmp(s_provider, "nvidia", 6)==0) so
routing uses the NVIDIA URL/headers when s_provider holds either value; modify
the provider_is_nvidia function accordingly.


/** Return the full API URL for the active provider. */
static const char *llm_api_url(void)
{
return provider_is_openai() ? MIMI_OPENAI_API_URL : MIMI_LLM_API_URL;
if (provider_is_openai()) return MIMI_OPENAI_API_URL;
if (provider_is_openrouter()) return MIMI_OPENROUTER_API_URL;
if (provider_is_nvidia()) return MIMI_NVIDIA_NIM_API_URL;
return MIMI_LLM_API_URL;
}

/** Return the HTTP Host header value for the active provider. */
static const char *llm_api_host(void)
{
return provider_is_openai() ? "api.openai.com" : "api.anthropic.com";
/* TODO(PR#40): derive host from configured API URL once custom provider URLs are supported. */
if (provider_is_openai()) return "api.openai.com";
if (provider_is_openrouter()) return "openrouter.ai";
if (provider_is_nvidia()) return "integrate.api.nvidia.com";
return "api.anthropic.com";
}

/** Return the HTTP request path for the active provider. */
static const char *llm_api_path(void)
{
return provider_is_openai() ? "/v1/chat/completions" : "/v1/messages";
if (provider_is_openrouter()) return "/api/v1/chat/completions";
if (provider_is_openai() || provider_is_nvidia()) return "/v1/chat/completions";
return "/v1/messages";
}

/** Return true for OpenAI-compatible providers (OpenAI, OpenRouter, NVIDIA NIM). */
static bool provider_is_openai_compat(void)
{
return provider_is_openai() || provider_is_openrouter() || provider_is_nvidia();
}

/* ── Init ─────────────────────────────────────────────────────── */
Expand Down Expand Up @@ -265,7 +293,7 @@ static esp_err_t llm_http_direct(const char *post_data, resp_buf_t *rb, int *out

esp_http_client_set_method(client, HTTP_METHOD_POST);
esp_http_client_set_header(client, "Content-Type", "application/json");
if (provider_is_openai()) {
if (provider_is_openai_compat()) {
if (s_api_key[0]) {
char auth[LLM_API_KEY_MAX_LEN + 16];
snprintf(auth, sizeof(auth), "Bearer %s", s_api_key);
Expand Down Expand Up @@ -293,7 +321,7 @@ static esp_err_t llm_http_via_proxy(const char *post_data, resp_buf_t *rb, int *
int body_len = strlen(post_data);
char header[1024];
int hlen = 0;
if (provider_is_openai()) {
if (provider_is_openai_compat()) {
hlen = snprintf(header, sizeof(header),
"POST %s HTTP/1.1\r\n"
"Host: %s\r\n"
Expand Down Expand Up @@ -559,13 +587,13 @@ esp_err_t llm_chat_tools(const char *system_prompt,
/* Build request body (non-streaming) */
cJSON *body = cJSON_CreateObject();
cJSON_AddStringToObject(body, "model", s_model);
if (provider_is_openai()) {
if (provider_is_openai_compat()) {
cJSON_AddNumberToObject(body, "max_completion_tokens", MIMI_LLM_MAX_TOKENS);
} else {
cJSON_AddNumberToObject(body, "max_tokens", MIMI_LLM_MAX_TOKENS);
}

if (provider_is_openai()) {
if (provider_is_openai_compat()) {
cJSON *openai_msgs = convert_messages_openai(system_prompt, messages);
cJSON_AddItemToObject(body, "messages", openai_msgs);

Expand Down Expand Up @@ -635,7 +663,7 @@ esp_err_t llm_chat_tools(const char *system_prompt,
return ESP_FAIL;
}

if (provider_is_openai()) {
if (provider_is_openai_compat()) {
cJSON *choices = cJSON_GetObjectItem(root, "choices");
cJSON *choice0 = choices && cJSON_IsArray(choices) ? cJSON_GetArrayItem(choices, 0) : NULL;
if (choice0) {
Expand Down
2 changes: 1 addition & 1 deletion main/llm/llm_proxy.h
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ esp_err_t llm_proxy_init(void);
esp_err_t llm_set_api_key(const char *api_key);

/**
* Save the LLM provider to NVS. (e.g. "anthropic", "openai")
* Save the LLM provider to NVS. (e.g. "anthropic", "openai", "openrouter", "nvidia")
*/
esp_err_t llm_set_provider(const char *provider);

Expand Down
2 changes: 2 additions & 0 deletions main/mimi_config.h
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@
#define MIMI_LLM_MAX_TOKENS 4096
#define MIMI_LLM_API_URL "https://api.anthropic.com/v1/messages"
#define MIMI_OPENAI_API_URL "https://api.openai.com/v1/chat/completions"
#define MIMI_OPENROUTER_API_URL "https://openrouter.ai/api/v1/chat/completions"
#define MIMI_NVIDIA_NIM_API_URL "https://integrate.api.nvidia.com/v1/chat/completions"
#define MIMI_LLM_API_VERSION "2023-06-01"
#define MIMI_LLM_STREAM_BUF_SIZE (32 * 1024)
#define MIMI_LLM_LOG_VERBOSE_PAYLOAD 0
Expand Down
2 changes: 1 addition & 1 deletion main/mimi_secrets.h.example
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
#define MIMI_SECRET_FEISHU_APP_ID ""
#define MIMI_SECRET_FEISHU_APP_SECRET ""

/* Anthropic API */
/* LLM API (anthropic, openai, openrouter, or nvidia) */
#define MIMI_SECRET_API_KEY ""
#define MIMI_SECRET_MODEL ""
#define MIMI_SECRET_MODEL_PROVIDER "anthropic"
Expand Down