Cache is not shared across app nodes, leading to stale data with per-node Memcached usage

[tagliala]

Problem statement

When multiple servers run the same Hawk-based application, updates to resources (e.g., contacts) in one server aren’t reflected across all nodes because each is configured to use a local Memcached instance (localhost:11211). As a result, cache invalidation only affects the node that performed the update. Other app nodes continue to return stale data from their independent caches until TTL expiry or manual refresh.

Observations

• Logs show Dalli connecting to localhost:11211 on each server, indicating a per-node cache.
• Server logs ("cache HIT/MISS") highlight that the HTTP cache for resource requests is local; a PUT invalidates only locally.
• Users routed to different app nodes can see outdated resource data, e.g., after a profile/settings update.
• Risk: Outdated data, privacy/compliance issues, and inconsistent user experience.

Root cause

Hawk (or libraries using it) default to a local cache configuration (localhost:11211). Cache invalidation is per-node, so changes aren’t propagated across the cluster.

Recommended approaches

1. Configure shared cache backend:
   • Use a shared Memcached cluster (all nodes point to same endpoints, not localhost) or switch to Redis (shared configuration).
   • Example (Memcached): ['memc-1:11211', 'memc-2:11211']
   • Example (Redis): redis://cache-host:6379
   • Ensure the Hawk client/library is configured to use the shared cache (not local Dalli instance).
2. If you must keep per-node caches:
   • Publish cache-bust events (e.g., with Redis Pub/Sub) after PUT/POST to invalidate across all nodes. (Adds complexity.)
3. As a short-term workaround:
   • Lower cache TTL. This reduces the window of data staleness, but does not eliminate it and can increase backend load.

Steps to verify remediation

• Update cache configs; warm cache on node A, update value on node B, immediately fetch from A.
• Expect a cache MISS or new value if cache is shared and invalidation propagates.

Action items

• Review and document current cache adapter configuration for Hawk in staging/prod.
• Move to a shared cache backend (Memcached cluster or Redis) for all Hawk caches.
• Deploy and validate using test scenarios across multiple nodes.

Open questions

• Do we already have a shared Redis or Memcached instance?
• Is Hawk configured to use its own cache client or Rails.cache?
• What is the current resource cache TTL, and is there support for cache versioning per record?

Bottom line

Hawk’s default cache setup can cause data drift when used with per-node cache servers. Ensure a shared cache backend for cross-node consistency.

---

This issue is based on analysis of a real-world problem but reworded to apply generically to any Hawk-based app using local (per-node) Memcached for cache.