Skip to content

Scale down web app instances during rolling deploy to avoid memory quota exceeded #1948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rileyseaburg wants to merge 2 commits into
base: production
Choose a base branch
from
Open

Scale down web app instances during rolling deploy to avoid memory quota exceeded #1948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6250c86
Scale down web app instances during rolling deploy to avoid memory qu…
rileyseaburg Jan 13, 2026
988249d
Set SKIP_WIDGET_RENDERER for sidekiq workers to prevent crash
rileyseaburg Jan 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .circleci/deploy-sidekiq.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,11 @@ cf_push_with_retry() {
cf stop "$app_name" || true
sleep 5

# Set SKIP_WIDGET_RENDERER to prevent crash from missing Rust native library
# Sidekiq doesn't need the widget renderer - it's only used for rendering widgets in the web app
echo "Setting SKIP_WIDGET_RENDERER=true for $app_name..."
cf set-env "$app_name" SKIP_WIDGET_RENDERER "true" > /dev/null 2>&1 || true

# Push without rolling strategy (direct replacement since we stopped it)
# Let CF auto-detect buildpacks to avoid re-running supply phase (Rust already built in CircleCI)
if cf push "$app_name" \
Expand Down
31 changes: 31 additions & 0 deletions .circleci/deploy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,17 @@ cf_push_with_retry() {
# Wait for any in-progress deployment
wait_for_deployment "$app_name"

# Get current instance count and scale down to 1 to avoid memory quota issues during rolling deploy
echo "Checking current instance count for $app_name..."
local current_instances=$(cf app "$app_name" 2>/dev/null | grep "^instances:" | awk '{print $2}' | cut -d'/' -f2 || echo "1")
echo "Current instances: $current_instances"

if [ "$current_instances" -gt 1 ]; then
echo "Scaling down to 1 instance to free memory for rolling deploy..."
cf scale "$app_name" -i 1 || true
sleep 5
fi

for i in $(seq 1 $max_retries); do
echo "Attempt $i of $max_retries to push $app_name..."
local exit_code=0
Expand All @@ -239,6 +250,13 @@ cf_push_with_retry() {
echo "Push initiated successfully, waiting for full deployment to complete..."
if wait_for_deployment_complete "$app_name"; then
echo "Successfully deployed $app_name"

# Scale back up to original instance count
if [ "$current_instances" -gt 1 ]; then
echo "Scaling back up to $current_instances instances..."
cf scale "$app_name" -i "$current_instances" || true
fi

release_deploy_lock "$app_name"
trap - EXIT # Clear the trap
return 0
Expand All @@ -260,6 +278,13 @@ cf_push_with_retry() {
echo "Active deployment detected, waiting for it to complete instead of retrying..."
if wait_for_deployment_complete "$app_name"; then
echo "Existing deployment completed successfully"

# Scale back up to original instance count
if [ "$current_instances" -gt 1 ]; then
echo "Scaling back up to $current_instances instances..."
cf scale "$app_name" -i "$current_instances" || true
fi

release_deploy_lock "$app_name"
trap - EXIT
return 0
Expand All @@ -275,6 +300,12 @@ cf_push_with_retry() {
fi
done

# If we failed, try to scale back up anyway
if [ "$current_instances" -gt 1 ]; then
echo "Deploy failed, attempting to scale back up to $current_instances instances..."
cf scale "$app_name" -i "$current_instances" || true
fi

release_deploy_lock "$app_name"
trap - EXIT # Clear the trap
echo "Failed to push $app_name after $max_retries attempts"
Expand Down