-
Notifications
You must be signed in to change notification settings - Fork 11
Support retrying interrupted FARGATE task #53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds support for automatically retrying FARGATE_SPOT tasks that are interrupted by AWS. The implementation detects when a task is interrupted by matching the "Your Spot Task was interrupted" message in the task's stopped reason and triggers a retry mechanism.
- Adds detection for FARGATE_SPOT task interruptions using regex pattern matching
- Implements retry logic for interrupted spot tasks similar to existing host termination handling
- Updates CI configuration to test against newer Ruby versions (3.3, 3.4) while removing older versions (2.7, 3.0, 3.1)
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| lib/wrapbox/runner/ecs.rb | Adds regex constant and retry logic for detecting and handling interrupted FARGATE_SPOT tasks |
| .github/workflows/main.yml | Updates Ruby version matrix for CI testing to include newer versions |
Comments suppressed due to low confidence (1)
lib/wrapbox/runner/ecs.rb:31
- [nitpick] The constant name SPOT_TASK_INTERRUPTED should include 'REGEXP' suffix to match the naming pattern of HOST_TERMINATED_REASON_REGEXP for consistency.
SPOT_TASK_INTERRUPTED = /Your Spot Task was interrupted/
| WAIT_DELAY = 5 | ||
| TERM_TIMEOUT = 120 | ||
| HOST_TERMINATED_REASON_REGEXP = /Host EC2.*terminated/ | ||
| SPOT_TASK_INTERRUPTED = /Your Spot Task was interrupted/ |
Copilot
AI
Jul 18, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The regex pattern should end with a word boundary or be more specific to avoid potential false matches if AWS changes the message format slightly. Consider using /Your Spot Task was interrupted\b/ or anchoring the pattern.
| SPOT_TASK_INTERRUPTED = /Your Spot Task was interrupted/ | |
| SPOT_TASK_INTERRUPTED = /Your Spot Task was interrupted\b/ |
|
@joker1007 @akihiro17 Can you take a look? |
joker1007
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. LGTM!
Tasks executed with FARGATE_SPOT may be interrupted. When interrupted, the following log is output.
Handling this log, the Spot Task is retried in the wrapbox. This