From 7e62aa630a906d814dfdf684ca693541598b33ae Mon Sep 17 00:00:00 2001 From: ispyisail Date: Sun, 15 Mar 2026 23:41:44 +1300 Subject: [PATCH] docs: Add ESP32 integration guide Covers TLS certificate pinning, RAM management (sequential memory usage), JSON deserialization filters, pagination, retry strategy, NVS caching, heap monitoring, and troubleshooting. Based on production ESP32 badminton timer that polls Hello Club for event bookings. Co-Authored-By: Claude Opus 4.6 --- README.md | 1 + docs/esp32-guide.md | 324 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 325 insertions(+) create mode 100644 docs/esp32-guide.md diff --git a/README.md b/README.md index 784d7e2..a46f799 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,7 @@ See [Authentication](docs/authentication.md) for how to get your API key from th | [Rate Limiting](docs/rate-limiting.md) | 30 req/min limit, 429 handling, retry strategy | | [V2 Migration](docs/v2-migration.md) | V2 transition guide with side-by-side comparison | | [Gotchas](docs/gotchas.md) | Broken endpoints, spec vs reality, known issues | +| [ESP32 Guide](docs/esp32-guide.md) | TLS certs, RAM management, and working example for embedded use | ## Field Reference diff --git a/docs/esp32-guide.md b/docs/esp32-guide.md new file mode 100644 index 0000000..eff1051 --- /dev/null +++ b/docs/esp32-guide.md @@ -0,0 +1,324 @@ +# ESP32 Integration Guide + +Using the Hello Club API from an ESP32 microcontroller. Covers TLS certificates, RAM management, and a working example. + +> Based on a production ESP32 badminton court timer that polls Hello Club for bookings and auto-starts game timers. + +## The Challenge + +The ESP32-WROOM-32 has **~60KB usable RAM** after WiFi and FreeRTOS overhead. A single HTTPS connection uses **~40-50KB** for the TLS handshake (mbedTLS). That leaves almost nothing for parsing JSON responses. Getting HTTPS API calls to work reliably requires careful memory management at every step. + +## Prerequisites + +- **Arduino framework** with ESP32 board support +- **Libraries:** [ArduinoJson](https://arduinojson.org/) v7+, WiFiClientSecure (built-in) + +```ini +# platformio.ini +[env:esp32dev] +platform = espressif32 +board = esp32dev +framework = arduino +lib_deps = + bblanchon/ArduinoJson +``` + +## 1. TLS Certificates + +`WiFiClientSecure` requires a root CA certificate to verify the API server identity. You have three options: + +### Option A: Pin the Root CA (recommended) + +Pin only the root CA(s) that sign `api.helloclub.com`. This uses minimal RAM and still validates the connection. + +```cpp +#include + +// Google Trust Services Root R4 (ECDSA) - signs api.helloclub.com +// Valid until 2036-06-22 +const char* rootCA = +"-----BEGIN CERTIFICATE-----\n" +"MIICCTCCAY6gAwIBAgINAgPlwGjvYxqccpBQUjAKBggqhkjOPQQDAzBHMQswCQYD\n" +"VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIG\n" +"A1UEAxMLR1RTIFJvb3QgUjQwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAwMDAw\n" +"WjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2Vz\n" +"IExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjQwdjAQBgcqhkjOPQIBBgUrgQQAIgNi\n" +"AATzdHOnaItgrkO4NcWBMHtLSZ37wWHO5t5GvWvVYRg1rkDdc/eJkTBa6zzuhXyi\n" +"QHY7qca4R9gq55KRanPpsXI5nymfopjTX15YhmUPoYRlBtHci8nHc8iMai/lxKvR\n" +"HYqjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW\n" +"BBSATNbrdP9JNqPV2Py1PsVq8JQdjDAKBggqhkjOPQQDAwNpADBmAjEA6ED/g94D\n" +"9J+uHXqnLrmvT/aDHQ4thQEd0dlq7A/Cr8deVl5c1RxYIigL9zC2L7F8AjEA8GE8\n" +"p/SgguMh1YQdc4acLa/KNJvxn7kjNuK8YAOdgLOaVsjh4rsUecrNIdSUtUlD\n" +"-----END CERTIFICATE-----\n"; +``` + +**How to find the right certificate:** +1. Run `openssl s_client -connect api.helloclub.com:443 -showcerts` +2. Copy the last certificate in the chain (the root CA) +3. Or check in a browser: click the padlock, view certificate, certification path, export root + +**Tip:** Include a second root CA (e.g. Let's Encrypt ISRG Root X1) as a fallback. If Hello Club changes hosting provider, your device won't break until you can push a firmware update. + +### Option B: setInsecure() - development only + +```cpp +WiFiClientSecure client; +client.setInsecure(); // Skips certificate verification entirely +``` + +Works but provides **no protection against MITM attacks**. Your API key is sent in plaintext to whoever intercepts the connection. Never use this in production. + +### Option C: ESP32 built-in CA bundle + +```cpp +#include + +WiFiClientSecure client; +client.setCACertBundle(esp_crt_bundle_attach); +``` + +Uses Mozilla's CA bundle stored in flash. Convenient but uses more flash (~200KB) and may be outdated on older ESP-IDF versions. + +### Certificate Expiry + +Root CAs have long lifetimes (10-20 years), but they do expire. The current root for `api.helloclub.com` (GTS Root R4) expires **2036-06-22**. Consider implementing OTA firmware updates so you can rotate certificates without physical access to the device. + +## 2. RAM Management: The Core Problem + +Here is why naive HTTPS + JSON parsing crashes the ESP32: + +``` +Total usable RAM: ~60 KB +TLS connection: -45 KB (mbedTLS handshake buffers) +JSON response: -20 KB (even a small /event response) + ------- +Remaining: -5 KB <- CRASH (heap exhaustion) +``` + +### Solution: Sequential memory usage + +The key insight is that you don't need TLS and JSON memory **at the same time**. Read the response as a String, close the TLS connection (freeing ~45KB), then parse the JSON: + +```cpp +bool fetchEvents(const String& apiKey) { + WiFiClientSecure client; + client.setCACert(rootCA); + + HTTPClient http; + http.begin(client, "https://api.helloclub.com/event?limit=5&sort=startDate"); + http.addHeader("X-Api-Key", apiKey); + http.addHeader("Accept", "application/json"); + http.setTimeout(10000); + + int httpCode = http.GET(); + if (httpCode != HTTP_CODE_OK) { + http.end(); + return false; + } + + // Step 1: Read the full response as a String + String payload = http.getString(); + + // Step 2: FREE the TLS connection BEFORE parsing JSON + http.end(); + client.stop(); + // ~45KB of RAM is now available again + + // Step 3: Parse JSON with the freed memory + DynamicJsonDocument doc(8192); + deserializeJson(doc, payload); + + // Step 4: Free the payload String too + payload = String(); + + // Process events... + JsonArray events = doc["events"].as(); + for (JsonObject event : events) { + Serial.println(event["name"].as()); + } + + return true; +} +``` + +### Don't keep persistent TLS clients + +It's tempting to keep a `WiFiClientSecure` as a class member to reuse the TLS session. Don't - it holds ~45KB the entire time. Create it on the stack inside your request function so it's freed automatically when the function returns. + +```cpp +// BAD - holds 45KB permanently +class ApiClient { + WiFiClientSecure client; // Allocated for the lifetime of the object +}; + +// GOOD - freed after each request +bool makeRequest() { + WiFiClientSecure client; // Stack-allocated, freed when function exits + client.setCACert(rootCA); + // ... make request ... +} // client destructor frees TLS memory here +``` + +## 3. JSON Deserialization Filters + +The Hello Club API returns large objects with many fields you probably don't need. ArduinoJson's `Filter` feature lets you specify which fields to keep during deserialization, reducing memory usage by up to 90%. + +```cpp +// Define a filter - only keep these 5 fields per event +StaticJsonDocument<256> filter; +filter["events"][0]["id"] = true; +filter["events"][0]["name"] = true; +filter["events"][0]["description"] = true; +filter["events"][0]["startDate"] = true; +filter["events"][0]["endDate"] = true; + +// Parse with filter - everything else is discarded +DynamicJsonDocument doc(8192); // 8KB is enough for ~20 filtered events +deserializeJson(doc, payload, DeserializationOption::Filter(filter)); +``` + +Without the filter, you'd need a `DynamicJsonDocument(65536)` or larger - more than the ESP32's entire free heap. + +## 4. Pagination + +A 7-day lookahead can return dozens of events. Fetching them all at once would require a huge response buffer. Instead, paginate with small pages: + +```cpp +const int PAGE_SIZE = 5; + +for (int offset = 0; offset < 100; offset += PAGE_SIZE) { + String params = "fromDate=" + fromDate; + params += "&toDate=" + toDate; + params += "&sort=startDate"; + params += "&limit=" + String(PAGE_SIZE); + params += "&offset=" + String(offset); + + DynamicJsonDocument doc(8192); + if (!fetchPage("/event", params, doc)) break; + + JsonArray events = doc["events"].as(); + if (events.size() == 0) break; // No more events + + for (JsonObject event : events) { + // Cache only the events you care about + } + + if (events.size() < PAGE_SIZE) break; // Last page +} +``` + +**Why 5?** Each page creates a TLS connection (~45KB), parses response (~8KB), processes, then frees everything. A page of 5 events with filtered JSON fits comfortably in 8KB. Larger pages risk OOM. + +## 5. Retry Strategy + +The ESP32's WiFi stack is less reliable than a desktop HTTP client. Network glitches, DNS failures, and TLS handshake timeouts are common. But keep retries short - `delay()` blocks the entire main loop. + +```cpp +const int MAX_RETRIES = 2; +const int RETRY_DELAYS[] = {500, 1000}; // milliseconds + +for (int attempt = 0; attempt < MAX_RETRIES; attempt++) { + // ... make request ... + if (httpCode == HTTP_CODE_OK) return true; + + bool shouldRetry = (httpCode == 429 || httpCode == 503 || httpCode == 504 || httpCode < 0); + if (httpCode == 401) shouldRetry = false; // Bad API key - don't retry + + if (!shouldRetry || attempt == MAX_RETRIES - 1) return false; + + delay(RETRY_DELAYS[attempt]); +} +``` + +**Key points:** +- **Max 2 attempts** - more just wastes time and blocks your app +- **Short delays** (500ms, 1s) - the ESP32 can't do anything else during `delay()` +- **Don't retry 401** - the API key is wrong, retrying won't help +- **Do retry 429, 503, 504** - transient errors that may resolve +- **Retry on `httpCode < 0`** - means network/TLS failure (DNS, timeout, etc.) + +For polling-based applications, also implement a **backoff at the poll level**: poll every hour normally, but switch to every 5 minutes after a failure. + +## 6. NVS Caching (Offline Resilience) + +ESP32's NVS (Non-Volatile Storage) lets you persist API data across reboots. If your device loses WiFi or the API is down, it can still work from cached data. + +```cpp +#include + +void saveEventsToNVS(const std::vector& events) { + DynamicJsonDocument doc(4096); + JsonArray arr = doc.to(); + + for (const auto& evt : events) { + JsonObject obj = arr.createNestedObject(); + obj["i"] = evt.id; // Short keys save NVS space + obj["n"] = evt.name; + obj["s"] = (long)evt.startTime; + obj["e"] = (long)evt.endTime; + } + + String json; + serializeJson(doc, json); + + Preferences prefs; + prefs.begin("helloclub", false); + prefs.putString("events", json); + prefs.end(); +} + +void loadEventsFromNVS() { + Preferences prefs; + prefs.begin("helloclub", true); // true = read-only + String json = prefs.getString("events", "[]"); + prefs.end(); + + // Parse and populate your event cache... +} +``` + +**Tips:** +- Use **abbreviated key names** (`i`, `n`, `s` instead of `id`, `name`, `startTime`) - NVS has size limits +- **Truncate strings** - event IDs to 12 chars, names to 40 chars +- Call `loadFromNVS()` on boot **before** the first API poll +- NVS has limited write cycles (~100K) - don't write on every loop iteration + +## 7. Heap Monitoring + +Always log free heap before and after API calls during development. This is how you catch memory leaks: + +```cpp +Serial.printf("Before request - free heap: %d bytes\n", ESP.getFreeHeap()); + +// ... make API request ... + +Serial.printf("After request - free heap: %d bytes\n", ESP.getFreeHeap()); +``` + +If free heap keeps decreasing across multiple API calls, you have a memory leak (usually an unclosed HTTPClient or WiFiClientSecure that wasn't properly destructed). + +## Memory Budget Reference + +Typical heap usage measured on ESP32-WROOM-32 (4MB flash, 520KB SRAM): + +| Component | RAM Usage | Notes | +|-----------|----------|-------| +| WiFi + FreeRTOS | ~200 KB | Always allocated after `WiFi.begin()` | +| Free heap (typical) | ~50-70 KB | What's left for your code | +| TLS handshake (mbedTLS) | ~40-50 KB | Allocated during HTTPS connection | +| `DynamicJsonDocument(8192)` | 8 KB | Enough for ~20 filtered events | +| `String` response body | 2-10 KB | Depends on response size and page limit | +| ArduinoJson filter | < 1 KB | `StaticJsonDocument<256>` | + +**Critical threshold:** If `ESP.getFreeHeap()` drops below **10KB**, you're at risk of crashes from heap fragmentation. Monitor this in development. + +## Troubleshooting + +| Symptom | Cause | Fix | +|---------|-------|-----| +| `ESP.getFreeHeap()` shows 15KB+ free but JSON parse fails | Heap fragmentation - no single contiguous block large enough | Free TLS before parsing (Section 2) | +| TLS handshake timeout (httpCode = -1) | Weak WiFi signal, underpowered USB cable, or NTP not synced | Check WiFi RSSI, use good USB cable, sync NTP before first request | +| `deserializeJson` returns `NoMemory` | Response too large for document buffer | Use JSON filter (Section 3) and pagination (Section 4) | +| Works once then crashes on second request | `WiFiClientSecure` or `HTTPClient` not properly freed | Always call `http.end()` and `client.stop()` | +| Certificate verification fails after months | Server changed intermediate CA | Pin the **root** CA (not intermediate), or include fallback roots | +| `WiFi.begin()` hangs or connects then drops | SSID is case-sensitive on ESP32 | Double-check exact case: "MyNetwork" != "mynetwork" |