Sporadic 500 errors on /statement route in production

[anthonybailey]

Problem

Sporadic 500 Internal Server Error responses for the /statement route in production, despite the route working correctly most of the time.

Investigation Findings

Local Reproduction Attempt

• Environment: Main branch, en-only mode (PARAGLIDE_LOCALES=en)
• Test method: 30+ concurrent requests to /statement route via netlify serve
• Result: All requests returned 200 status codes

Root Cause Analysis

The /statement route has graceful error handling that masks most API failures:

1. API Endpoint Behavior (/api/signatories):

   try {
     // Fetch from Airtable API
     const records = await fetchAllPages(fetch, url)
     // Process and return real data
   } catch (e) {
     console.error('Error fetching signatories:', e)
     // Always returns HTTP 200 with fallback data
     return json({
       signatories: fallbackSignatories,
       totalCount: 0
     })
   }

2. Page Loader (+page.ts):

   const response = await fetch('api/signatories');
   // No error handling - assumes API always returns 200
   const { signatories, totalCount } = await response.json();

Key Insight

Why we see 200s despite errors: The API endpoint catches Airtable failures and returns HTTP 200 with placeholder data ("Error", "This should be" signatories) instead of throwing 500 errors.

Performance Observations

• Without Airtable API key: Sub-second responses (fallback data)
• With Airtable API key: 1-4 second responses (real API calls)
• Implication: Airtable API latency could contribute to timeout-related 500s

Likely Causes of Production 500s

Since our local testing with graceful degradation didn't reproduce 500s, the production failures likely occur from:

1. Infrastructure-level failures:

   • Edge function timeouts before reaching API endpoint
   • Netlify platform issues
   • Memory/resource exhaustion

2. Race conditions under high load:

   • Concurrent requests overwhelming edge function
   • Airtable API rate limiting causing different error paths

3. Unhandled JavaScript errors:

   • Runtime errors that bypass the try/catch in API endpoint
   • Edge function bundling/execution issues

4. Network/timeout issues:

   • Airtable API taking >4 seconds (production timeout threshold)
   • DNS resolution failures

Next Steps (Claude's suggestions - not yet endorsed)

1. Add error handling to page loader:

   try {
     const response = await fetch('api/signatories');
     if (\!response.ok) throw new Error(`API error: ${response.status}`);
     const data = await response.json();
   } catch (e) {
     // Handle API failures gracefully
   }

2. Monitor edge function logs in production for specific error patterns

3. Consider caching strategy for signatory data to reduce Airtable API dependency

4. Add timeout handling for slow Airtable API responses

Environment Details

• Branch tested: main
• Local tool: netlify serve
• Configuration: English-only mode
• Airtable access: Read-only API key configured

[anthonybailey]

OK. So maybe errors manifesting as 500 in production could be its own separate thing. I haven't played locally with CI or other env variables it might set differently.

But hitting Airtable on every render is rather demanding and might well explain a sporadic error. It isn't a fast call. I don't know if we let Airtable cache it. I don't know if the page render fetches every signature even when it only shows the first few.

Fundamentally, we have a statement we want hundreds more to sign, so without sane design this is going to be brittle. (It's almost like we should have bought not built the petition hosting service. No! Bad Anthony! Cassandra has to wait!)

[anthonybailey]

Dumb copy paste of Discord as we discover stuff:

Nils | dasSpieler — 17:38
uhh that's very true, the /people page was cached, will do the same for /statement
[17:40]
ok it already is cached phew

anthonybailey.net — 17:41
Cached in Airtable with what ttl?

@anthonybailey.net
Cached in Airtable with what ttl?

Nils | dasSpieler — 17:41
no in netlify, via cache-control header

anthonybailey.net — 17:42
same ask re ttl

Nils | dasSpieler — 17:42
1h

anthonybailey.net — 17:44
Hmmm. Maybe it doesn't manifest under local netlify serve, but I don't see that happening back here.
[17:46]
If it's current and is ultimately down to an unreliable Airtable call, we could just pre-render for now and require deploy to update.

Nils | dasSpieler — 17:46
in production i get cache-status "Netlify Edge"; hit most of the time

[Wituareard]

We have identified a call to setLocale in +layout.ts as a possible cause which has been removed and the page is also pre-rendered now.

[anthonybailey]

I am still as yet unable to reproduce locally with unfixed code and netlify serve, although that is not definitely at odds with Nils setLocale theory.

For sakeof info: I have sometimes gotten Airtable to return a 427 with sufficiently aggressive parallel requests to our local server. But the load involved is not something I expect we saw during trouble times in production.

     [render] Error fetching signatories: Error: Failed to fetch data from Airtable: Too Many Requests. Details:
     {"errors":[{"error":"RATE_LIMIT_REACHED","message":"Rate limit exceeded. Please try again later"}]}
         at p33 (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:33828:15)
         at eventLoopTick (ext:core/01_core.js:177:7)
         at async u27 (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:33883:17)
         at async Er (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:40082:17)
         at async $10 (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:41773:63)
         at async Object.run (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:468:14)
         at async m2 (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:455:10)
         at async Yt (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:41734:17)
         at async fetch (file:///home/anthony/repos/pauseai-l10n/notes/references/pauseai-website/.netlify/edge-functions/render.js:41624:19)

Edge functions are documented as having a ten seconds timeout. We know that is false for us: the limit we saw when developing the write endpoint api callbacks was thirty seconds. But edge function timeout can be disregarded given when we reproduced the production results in browser, I saw the sporadic 500 within a second.

Dumping other ideas (Claude summary because I'm lazy):
"""

1. Sporadic edge function cold start failures - The edge function fails to initialize properly on some instances
2. Race condition in paraglide runtime - The mock AsyncLocalStorage or locale handling fails under specific timing conditions
3. Memory spike during concurrent requests - Multiple simultaneous requests cause immediate memory exhaustion
4. Bundling/compilation issue - The edge function bundle has an intermittent problem that only manifests under certain conditions

Since we can't reproduce it locally, it's likely an edge-runtime-specific issue. The fact that it's swift and sporadic suggests it's either:

• A race condition that only occurs with specific request timing
• An initialization issue that affects only some edge function instances
• A resource allocation failure that happens before the request is processed
  """

[Wituareard]

If it is setLocale it might also have had something to do with headers, cookies etc

[anthonybailey]

Yeah. The sporadic nature across same client browser makes me suspect the first option though: some edge functions initialize correctly, some don't.

[seankessinger]

Did this get fixed?

@anthonybailey ran into the same problem (#538) and said it matched this issue. He tracked down the cause and fixed it in PR #548.

[anthonybailey]

I mean... It was never easy to reproduce. We could safely close this.

…

On Fri, 19 Dec 2025, 22:09 seankessinger, ***@***.***> wrote: *seankessinger* left a comment (PauseAI/pauseai-website#370) <#370 (comment)> Did this get fixed? @anthonybailey <https://github.com/anthonybailey> ran into the same problem (#538 <#538>) and said it matched this issue. He tracked down the cause and fixed it in PR #548 <#548>. — Reply to this email directly, view it on GitHub <#370 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAXXDJSUA4OYRYBTVUMAKT4CRZRDAVCNFSM6AAAAAB7NZK7HGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNZWHAYTMMRYGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Wituareard]

Agreed, and we didn't get any more reports which happened when this was written iirc