Allow circuit to close even when open with failures < threshold#7
Open
njbennett wants to merge 1 commit intoahawkins:masterfrom
Open
Allow circuit to close even when open with failures < threshold#7njbennett wants to merge 1 commit intoahawkins:masterfrom
njbennett wants to merge 1 commit intoahawkins:masterfrom
Conversation
- We observed an issue recently where the circuit breakers for our application got stuck open and had to be manually closed by editing the fuse database. - We believe that the issue was that multiple threads moving through the breaker simultaneously triggered a race condition, where the breaker recorded a failure as it was also opening the fuse. This caused the fuse to get into a state where it was open with failures below the failure threshhold. - At this point (as demonstrated by the test we've added) the circuit will stay closed forever: 'tripped' will always return 'false' because failure count is below threshold, so it will never enter half-open state and allow a successful test request to close the circuit. - By also sending test requests when the circuit is open but not tripped (which we think should only ever happen in this error state) the circuits will be able to close again once the system that they guard against returns to normal, even if request volume during an outage is high enough to put them into this state. Signed-off-by: Natalie Bennett <nbennett@pivotal.io> Signed-off-by: Tom Viehman <tviehman@pivotal.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
application got stuck open and had to be manually closed by editing
the fuse database.
breaker simultaneously triggered a race condition, where the breaker
recorded a failure as it was also opening the fuse. This caused the fuse
to get into a state where it was open with failures below the failure
threshhold.
will stay closed forever: 'tripped' will always return 'false'
because failure count is below threshold, so it will never enter
half-open state and allow a successful test request to close the circuit.
(which we think should only ever happen in this error state) the
circuits will be able to close again once the system that they guard
against returns to normal, even if request volume during an outage is
high enough to put them into this state.
Signed-off-by: Natalie Bennett nbennett@pivotal.io
Signed-off-by: Tom Viehman tviehman@pivotal.io