Scale down web app instances during rolling deploy to avoid memory quota exceeded #1948
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses memory quota issues during Cloud Foundry rolling deployments by implementing a scale-down-deploy-scale-up pattern. The changes prevent deployment failures caused by insufficient memory quota when Cloud Foundry needs to simultaneously stage the application and run both old and new instances.
Changes:
- Modified
.circleci/deploy.shto capture current instance count before deployment - Implemented automatic scale-down to 1 instance before deployment if multiple instances are running
- Added scale-up logic to restore original instance count after successful deployment
- Added scale-up logic to restore service even when deployment fails
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ota exceeded Similar to the sidekiq worker fix in commit 48b4e42, scale down the main touchpoints web app to 1 instance before deploying, then scale back up after successful deployment. This prevents 'organization's memory limit exceeded' errors during rolling deployments when the org doesn't have enough quota for multiple instances during the staging/deployment process.
rileyseaburg
force-pushed
the
fix-web-deploy-memory-quota
branch
from
930ce30 to
6250c86
Compare
Sidekiq workers don't need the Rust widget renderer (it's only used for rendering widgets in the web app). Without SKIP_WIDGET_RENDERER=true, the app crashes on startup because the native library isn't found. This was causing the sidekiq worker to crash with: 'WidgetRenderer native library not found' Now the deploy script sets this env var before pushing.
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Production deployments are failing with:
Root Cause
During rolling deployments, Cloud Foundry needs memory for:
Even with
--strategy rolling, if the org doesn't have enough total memory quota, staging fails before rolling deployment even starts.Solution
This PR applies the same fix that was implemented for the sidekiq worker in commit 48b4e42:
Changes
.circleci/deploy.shto:Testing
This approach has been proven to work for the sidekiq worker deployment since December 2025 (commit 48b4e42).
Related