[JENKINS-76327] EC2 Plugin blocks Jenkins queue lock for up to 25 seconds for each orphan instance during startup

[jenkins-infra-bot]

Controller startup after outage/crash with orphaned agents leads to build queue completely blocked.
 
The impact scales linearly with the number of orphaned agents: 10 agents: +4 minutes (25x10)

Steps to Reproduce

1. Create many EC2 agents via EC2 plugin
2. Stop Jenkins controller
3. Manually delete the EC2 instances from AWS console (instances no longer exist in AWS)
4. Start Jenkins controller

Actual Result

• All Queue operations blocked for: 25 seconds × number of orphaned agents
• Some liveness probes may check the queue status, and time out because any queue operation is blocked. This will be interpreted as the instance being unhealthy, potentially causing automation to restart the instance

Expected Result

• Queue operations continue normally (negligible held time)
• Orphaned agents are handled asynchronously or skipped

Root Cause

EC2RetentionStrategy.internalCheck() calls CloudHelper.getInstanceWithRetry() which holds +25s in case the instance if not found (orphan).

Call chain:

ComputerRetentionWork.doAperiodicRun  runs every 1 minute
Queue.withLock()      [LOCK]
EC2RetentionStrategy.check()
EC2RetentionStrategy.internalCheck()
EC2Computer.getState()for each instance
CloudHelper.getInstanceWithRetry()
Thread.sleep(5000)    × 5 retries = 25 seconds MAX

 

Issue probably introduced in JENKINS-54071 , PR . As it started using retries in retention checks.

 

Workarround

1. Manually delete `$JENKINS_HOME/nodes/` before startup after outages
2. Enable `cleanUpOrphanedNodes: true` in EC2 cloud config (only available in latest versions), it will reduces frequency but doesn't prevent blocking

---

Originally reported by apuig, imported from: EC2 Plugin blocks Jenkins queue lock for up to 25 seconds for each orphan instance during startup

• assignee: thoulen
• status: Open
• priority: Major
• component(s): ec2-plugin
• resolution: Unresolved
• votes: 0
• watchers: 2
• imported: 2025-12-06

Raw content of original issue

Controller startup after outage/crash with orphaned agents leads to build queue completely blocked. 
 
The impact scales linearly with the number of orphaned agents: 10 agents: +4 minutes (25x10)
Steps to Reproduce

1. Create many EC2 agents via EC2 plugin
2. Stop Jenkins controller
3. Manually delete the EC2 instances from AWS console (instances no longer exist in AWS)
4. Start Jenkins controller
Actual Result

	
All Queue operations blocked for: 25 seconds × number of orphaned agents
	
Some liveness probes may check the queue status, and time out because any queue operation is blocked. This will be interpreted as the instance being unhealthy, potentially causing automation to restart the instance



Expected Result

	
Queue operations continue normally (negligible held time)
	
Orphaned agents are handled asynchronously or skipped



Root Cause

EC2RetentionStrategy.internalCheck() calls CloudHelper.getInstanceWithRetry() which holds +25s in case the instance if not found (orphan).

Call chain:

ComputerRetentionWork.doAperiodicRun  runs every 1 minute
Queue.withLock()                      [LOCK]
EC2RetentionStrategy.check()
EC2RetentionStrategy.internalCheck()
EC2Computer.getState()                for each instance
CloudHelper.getInstanceWithRetry()
Thread.sleep(5000)                    × 5 retries = 25 seconds MAX

 
Issue probably introduced in JENKINS-54071 , PR . As it started using retries in retention checks.
 
Workarround
1. Manually delete `$JENKINS_HOME/nodes/` before startup after outages
2. Enable `cleanUpOrphanedNodes: true` in EC2 cloud config (only available in latest versions), it will reduces frequency but doesn't prevent blocking

• environment: EC2 plugin 2043.v483cc2854116

[jenkins-infra-bot]

• Remote link associated with this issue:

  • CloudBees-internal issue

[jenkins-infra-bot]

apuig:

• Comment link
• Original comment link (no redirect)
• Raw content of original comment:

  I'm thinking the proper fix might involve a periodic job to keep a state cache, so the retention check isn’t blocking the queue at all.
  
  Alternatively, maybe simply skipping withretry for getState during retention checks could be enough for reasonable scale. WDYT?
  

I'm thinking the proper fix might involve a periodic job to keep a state cache, so the retention check isn’t blocking the queue at all.

Alternatively, maybe simply skipping withretry for getState during retention checks could be enough for reasonable scale. WDYT?

[jenkins-infra-bot]

jglick:

• Comment link
• Original comment link (no redirect)
• Raw content of original comment:

  skipping withretry for getState during retention checks could be enough
  
  That would fix the reported issue I think, and would certainly be an improvement. Just review the logic to see what will happen. There are two cases to consider: the bug scenario here (where the instance is really deleted); and the scenario motivating JENKINS-54071, which IIUC related to laggy results from AWS, which may not be relevant for this call site of getState anyway.
  
  There is still a potential problem in case the AWS API call hangs due to network infrastructure issues. I am not sure if the SDK client has a built-in timeout on API operations, and if so, how long; in CloudBees CI we generally consider any blockage of Queue.lock for more than a few seconds to be a problem.
  

skipping withretry for getState during retention checks could be enough

That would fix the reported issue I think, and would certainly be an improvement. Just review the logic to see what will happen. There are two cases to consider: the bug scenario here (where the instance is really deleted); and the scenario motivating JENKINS-54071, which IIUC related to laggy results from AWS, which may not be relevant for this call site of getState anyway.

There is still a potential problem in case the AWS API call hangs due to network infrastructure issues. I am not sure if the SDK client has a built-in timeout on API operations, and if so, how long; in CloudBees CI we generally consider any blockage of Queue.lock for more than a few seconds to be a problem.