Skip to content

Conversation

@ZL-Asica
Copy link
Member

🐛 Bug Fix

📌 Description

This PR fixes two related problems that were causing excessive Airtable API usage and unstable image delivery:

  1. Quota pressure on Airtable: Frequent ISR revalidation, incorrect use of Nextjs cache API, and uncached attachment image requests led to unnecessary .select().all() calls and repeated downloads of signed Airtable URLs.
  2. Image caching churn: Attachment URLs changed over time, so browser/proxy caches were ineffective and images were re-fetched.

What changed

  • Implemented a two-tier cache:

    • Data: Refactored the unstable_cache with per-table tags and a longer default REVALIDATE_TIME (1h) so cache misses are the only times the Airtable client runs.
    • Images: Added a local /api/images route that downloads an Airtable attachment once, transcodes to WebP, stores it in Cloudflare R2, and serves future hits from R2 with long-lived immutable caching. Cache keys are stable via Airtable attachment.id.
  • Added a tag-based GC (garbage collector) for R2 objects that haven’t been accessed recently, with a throttled ISR entry point.

  • Introduced production-only structured logging to measure true cache misses and end-to-end latency (for future Airtable quota checking purpose).

🔍 Feature Changes

  • API update — new GET /api/images/{attId}/{variant}/{filename}?src=... route
  • New component or page
  • UI/UX improvement
  • Other: Infra for image caching (Cloudflare R2), WebP transcoding, and cache GC

Highlights

  • src/app/api/images/[...path]/route.ts:
    • On hit: stream WebP from R2 with Cache-Control: public, max-age=31536000, immutable.
    • On miss: fetch Airtable signed src, transcode to WebP (sharp), store to R2, return WebP.
    • Tag objects on access: last-access=YYYY-MM-DD for GC.
  • src/utils/image.ts: image URLs now point to the local API and are keyed by attachment.id, not a rotating signed URL.

🔄 Refactor Changes

  • Code optimization — increased revalidate windows to reduce ISR churn (People/Projects → 4h, Project detail → 6h).
  • Improve maintainability — split out R2 client helpers, logging utilities, image conversion utilities, and GC logic with clear boundaries.
  • Remove unnecessary code
  • Other: ESLint tweak to ignore **.md (keeps lint signal clean for code files)

🛠 Bug Fixes

  • Fixed issue: Excessive Airtable quota usage due to frequent cache misses, revalidation, and rotating attachment URLs.
  • Improved error handling — safer fallbacks around R2 head/get/put, robust conversions, and API responses
  • Performance fix — immutable long-term image caching + WebP reduces bandwidth and repeated downloads
  • Other: —

✅ Checklist

  1. Feature Updates:

    • Code follows project coding style.
    • Tested in a Next.js environment.
    • Relevant documentation is updated (README: env vars + caching architecture).
  2. Bug Fixes:

    • Bug has been reproduced and verified (quota pressure from short revalidate + rotating image URLs).
    • Fix does not introduce new issues (basic manual checks done; add automated tests where useful).
    • Added necessary tests.
  3. Code Refactor:

    • No breaking changes introduced (APIs added; existing consumers unaffected).
    • Performance improvement validated (design-level; logs in prod will quantify).
    • Documentation updated if necessary.
  4. Dependency Updates:

    • Verified functionality after update (manual).
    • Checked for security vulnerabilities using npm audit / pnpm audit.

📜 Dependency Changes

Dependency Name Old Version New Version Reason
@aws-sdk/client-s3 (added) R2 SDK compatibility for object storage operations
sharp (added) Efficient server-side image decode/resize/transcode to WebP

📝 Steps to Reproduce (before fix)

  1. Visit pages with many Airtable-backed images (e.g., People or Projects).
  2. Observe network: images use signed Airtable URLs that rotate; repeat visits or ISR revalidations refetch images and data.
  3. With short revalidate (e.g., 60s), ISR causes frequent server fetches → Airtable quota usage climbs and images are re-downloaded.
  4. Over time, the combination of rotating URLs and short revalidation produces cache churn and avoids effective CDN/browser caching.

💬 Additional Notes

Env vars

# Data cache
REVALIDATE_TIME="3600"        # default to 1h

# Optional production logs (structured NDJSON for Airtable fetches)
AIRTABLE_LOG="1"

# Cloudflare R2
R2_ENDPOINT="https://<account-id>.r2.cloudflarestorage.com"
R2_ACCESS_KEY_ID="..."
R2_SECRET_ACCESS_KEY="..."
R2_BUCKET="..."
R2_CLEANUP_MAX_AGE_DAYS="45"  # default 45d if unset

GC behavior

  • R2 objects are tagged with last-access=YYYY-MM-DD.
  • maybeRunR2CleanupFromISR runs at most once per 24h, deleting objects older than 60d in the current page config (default is 45d if only the env var is used).

Operational notes

  • If R2 credentials are missing in non-prod environments, the server prints a warning; the route will return a clear 500 when invoked.
  • Logs in production (AIRTABLE_LOG=1) provide:
    • airtable.fetch.start/done with table, rows, duration, and cache tag.
    • Use these to confirm lower miss rates after deployment.

Files touched (high level)

  • API: src/app/api/images/[...path]/route.ts (new)
  • R2 client & GC: src/lib/r2.ts, src/lib/r2-gc.ts (new)
  • Image utils: src/utils/image-convert.ts (new), src/utils/image.ts (updated)
  • Airtable cache & logs: src/lib/airtable.ts, src/lib/logger.ts (new)
  • ISR windows: people/projects pages updated
  • Misc: eslint.config.mjs, README.md, minor loading shimmer class rename

…hing

- Add `/api/images/[...path]` to fetch→transcode (WebP)→cache in Cloudflare R2;
  serve hits with `Cache-Control: public, max-age=31536000, immutable`
- Key image cache by Airtable `attachment.id`; update `getImgUrlFromAttachmentObj`
- Add R2 client helpers and tag-based GC with throttled ISR runner
- Increase revalidation windows (people/projects: 4h; project detail: 6h);
  default `REVALIDATE_TIME` to 3600s to cut cache misses
- Add production-only structured logging to measure Airtable cache misses
- Docs: env vars + caching architecture; eslint ignore `**.md`; minor UI class rename

ENV:
REVALIDATE_TIME=3600, AIRTABLE_LOG=1,
R2_ENDPOINT, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET,
R2_CLEANUP_MAX_AGE_DAYS=45
@ZL-Asica ZL-Asica requested a review from Copilot October 20, 2025 23:51
@ZL-Asica ZL-Asica self-assigned this Oct 20, 2025
@ZL-Asica ZL-Asica added bug general issues with the website enhancement general tag for new features to add labels Oct 20, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR reduces Airtable API quota pressure by implementing a two-tier caching system for data and images. The changes address frequent ISR revalidation and rotating Airtable attachment URLs that previously caused cache churn.

Key changes:

  • Implemented R2-backed image caching with WebP transcoding and stable cache keys
  • Extended ISR revalidation windows (People/Projects: 4h, Project detail: 6h) and increased default REVALIDATE_TIME from 60s to 1h
  • Added tag-based garbage collection for R2 objects and structured production logging for cache metrics

Reviewed Changes

Copilot reviewed 19 out of 22 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/app/api/images/[...path]/route.ts New API route that caches Airtable images in R2 with WebP transcoding
src/utils/image-convert.ts WebP transcoding utility using sharp with HDR→SDR conversion
src/lib/r2.ts R2 client wrapper with helper functions for object operations
src/lib/r2-gc.ts Garbage collector for removing old R2 objects based on access tags
src/lib/logger.ts Structured NDJSON logging for production Airtable fetch metrics
src/utils/image.ts Updated to generate stable local API URLs instead of rotating Airtable URLs
src/lib/consts.ts Added R2 configuration constants and increased default revalidate time
src/lib/airtable.ts Added cache tags and structured logging to track fetch operations
src/app/people/page.tsx Increased revalidation to 4h and integrated R2 cleanup
src/app/projects/page.tsx Increased revalidation to 4h
src/app/projects/[id]/page.tsx Increased revalidation to 6h
package.json Added @aws-sdk/client-s3 and sharp dependencies
README.md Documented new environment variables and caching architecture

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@github-actions
Copy link

Build Failed!

🎨 Lint Check

Lint: No linting issues found!

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@github-actions
Copy link

Build Failed!

🎨 Lint Check

Lint: No linting issues found!

@ZL-Asica ZL-Asica marked this pull request as ready for review October 21, 2025 00:05
@ZL-Asica ZL-Asica merged commit 67d9e71 into main Oct 21, 2025
1 of 2 checks passed
@ZL-Asica ZL-Asica deleted the fix/airtable-api branch October 21, 2025 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug general issues with the website enhancement general tag for new features to add size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants