Fix rotating secret with TTL=0 to use exponential backoff retry instead of 1 sec by aswanidutt · Pull Request #2134 · hashicorp/consul-template

aswanidutt · 2026-02-11T21:52:43Z

issue: When a rotating secret that has rotation_period but ttl=0, it should not be treated as a rotating secret. Instead, it should wait and retry exponentially until original LeaseDuration with non-renewable lease logic (80-95% threshold)

This prevents the bug where sleep time would be 0 seconds ( +1 second of cushion) and constant polling every second to rotate

Changes:

Added maxJitter constant (50%)
Applied jitter to exponential backoff for rotating secrets with ttl=0:
Exponential backoff: 2s, 4s, 8s, 16s, 32s, 64s...Max backoff: 300 seconds (when VaultDefaultLeaseDuration not configured)
Jitter applied after capping to prevent thundering herd
Updated test to validate jittered values are within 90-100% of expected

Fixes: VAULT-38829-https://hashicorp.atlassian.net/browse/VAULT-38829?

before this change: checking to rotate every second

after the code change : checking every Exponential backoff: 2s, 4s, 8s, 16s, 32s, 64s, the max is VaultDefaultLeaseDuration set on consul-template

- When a rotating secret has rotation_period but ttl=0, it should not be treated as a rotating secret - Instead, it should fall back to using the original LeaseDuration with non-renewable lease logic (80-95% threshold) - This prevents the bug where sleep time would be 0 seconds Changes: - vault_common.go: Only set rotatingSecret=true when ttl > 0 - vault_common_test.go: Add test case for rotating secret with ttl=0 Fixes: VAULT-38829

tvoran · 2026-02-12T23:10:42Z

Can you separate the unrelated fixes into a separate PR? Like the race condition, CWE-338, etc.

…own pr

aswanidutt · 2026-02-13T18:19:51Z

need to merge #2135 first before this PR

aswanidutt · 2026-02-13T18:20:26Z

Can you separate the unrelated fixes into a separate PR? Like the race condition, CWE-338, etc.

#2135 added for the above unrelated fixes

tvoran

Just a couple questions. It's also a little strange to me that the TestFileQuery_Fetch/fires_changes test is failing on this PR, when it seems to have passed for recent commits in this repo.

tvoran · 2026-02-14T00:43:17Z

dependency/vault_common.go

+					// FIX: When TTL is 0, implement exponential backoff with jitter
+					// Start at 5s, double each time, max is VaultDefaultLeaseDuration or 300s


It would be good to mention something to the effect that Vault responds with a TTL of 0 when there's been an error rotating the secret on the Vault side, so this is effectively retrying an error condition.

added relavent comment

tvoran · 2026-02-14T00:44:21Z

dependency/vault_common.go

+					backoff := 5 * (1 << uint(retryCount)) // 5s, 10s, 20s, 40s, 80s...
+					maxBackoff := int(VaultDefaultLeaseDuration.Seconds())
+					if maxBackoff <= 0 {
+						maxBackoff = 300 // Default max of 300 seconds if not configured
+					}
+					if backoff > maxBackoff {
+						backoff = maxBackoff
+					}
+					backoffDur := time.Duration(backoff) * time.Second


Have you considered using the cenkalti/backoff library here? I'm not sure it'll fit this exact use case, but it's pretty widely used in Vault and it's already an indirect import in this project.

there are multiple libraries that are being used for the same use case, I have changed this to cenkalti-v4

tvoran · 2026-02-14T00:50:52Z

dependency/vault_common.go

+				} else if ttlData == 0 {
+					// FIX: When TTL is 0, implement exponential backoff with jitter
+					// Start at 5s, double each time, max is VaultDefaultLeaseDuration or 300s
+					backoff := 5 * (1 << uint(retryCount)) // 5s, 10s, 20s, 40s, 80s...


How was it decided to start at 5s for the first retry? Should it retry quicker than that?

changed to default 2 series instead of 5 from other references in the repo

tvoran · 2026-02-14T01:01:49Z

dependency/vault_common.go

+				if ttlData > 0 {
+					log.Printf("[DEBUG] Found rotation_period and set lease duration to %d seconds", ttlData)
+					// Add a second for cushion
+					base = int(ttlData) + 1
+					rotatingSecret = true


Feels like the retryCount should be reset when this condition is hit? Since we're treating ttl=0 as an error to retry, and ttl > 0 as a successful fetch?

good catch, I made that change. done

tvoran · 2026-02-18T07:21:34Z

dependency/vault_common.go

+					// Add a second for cushion
+					base = int(ttlData) + 1
+					rotatingSecret = true
+				} else if ttlData == 0 {


So in looking at consul-template a bit more it looks like there's already a notion of retries with backoff that is set in the RetryConfig:

consul-template/config/vault.go

Line 66 in dd7e8f0

Retry *RetryConfig `mapstructure:"retry"`

I'm wondering if we've tried just returning an error from VaultReadQuery.Fetch() and VaultWriteQuery.Fetch() in the ttl=0 case. Would that trigger the existing retry mechanism?

@tvoran thanks for that input, thats alot easier than I thought - here is the PR - #2136
but the only caveat is that there is default max 12 retries and maxbackoff of 1 min for retry.go. So, I had to use cenkaltiv4 in view.go to customize the retry backoff with max backoff of 5m and unlimited retries.

aswanidutt and others added 8 commits February 11, 2026 15:49

adding exponential back off retry for ttl 0

b0fc949

fixing CWE-338 Use of Cryptographically Secure Pseudo-Random Number

0b584b7

fixing cwe-338 on other places to address race test failure

d482dc5

reverting back to mathrand

6532fac

fixing race condition

7df06e8

fixing linting issue

d7246f0

fixing cwe-338

dbd6de2

removing the cwe-338 and race condition changes as they are in their …

f6ce3df

…own pr

aswanidutt marked this pull request as ready for review February 13, 2026 18:19

aswanidutt requested review from a team as code owners February 13, 2026 18:19

aswanidutt changed the title ~~Fix rotating secret with TTL=0 to use lease duration~~ Fix rotating secret with TTL=0 to use exponential back retry instead of 1 sec Feb 13, 2026

aswanidutt changed the title ~~Fix rotating secret with TTL=0 to use exponential back retry instead of 1 sec~~ Fix rotating secret with TTL=0 to use exponential backoff retry instead of 1 sec Feb 13, 2026

aswanidutt added 2 commits February 13, 2026 12:32

adding comments to address the CWE-338 later

92cd7f9

putting the race condition file back

8f245e3

tvoran reviewed Feb 14, 2026

View reviewed changes

hashicorp deleted a comment from ibm-mend-app bot Feb 16, 2026

aswanidutt added 3 commits February 16, 2026 11:18

using existing backoff and other pr review changes

68c34d8

go linter fix

d7a66ff

adding as direct dependecy

dd7e8f0

aswanidutt requested a review from tvoran February 16, 2026 17:33

tvoran reviewed Feb 18, 2026

View reviewed changes

aswanidutt closed this Feb 25, 2026

		// FIX: When TTL is 0, implement exponential backoff with jitter
		// Start at 5s, double each time, max is VaultDefaultLeaseDuration or 300s

Conversation

aswanidutt commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

before this change: checking to rotate every second

after the code change : checking every Exponential backoff: 2s, 4s, 8s, 16s, 32s, 64s, the max is VaultDefaultLeaseDuration set on consul-template

Uh oh!

tvoran commented Feb 12, 2026

Uh oh!

aswanidutt commented Feb 13, 2026

Uh oh!

aswanidutt commented Feb 13, 2026

Uh oh!

tvoran left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aswanidutt Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aswanidutt commented Feb 11, 2026 •

edited

Loading

aswanidutt Feb 19, 2026 •

edited

Loading