Skip to content

Conversation

@TheSafo
Copy link
Contributor

@TheSafo TheSafo commented Dec 30, 2025

What does this PR do?

Enables configuring byte-based throttlers in generators to be based per block (e.g. 10 blocks per second instead of 100 MB/s).

Impl. details:

  • Adds a wrapper for generators around lading throttle that's based around throttling blocks at a time (which aligns with how the throttler is currently used)
  • Adds some config tests to verify that both the new format and old format is still supported

Motivation

When using payload generators where blocks are intentionally not the same size/close to maximum bytes, it can be helpful to reason about throughput per block. This allows us to do things like:

  • Define a payload where each block is 1 second worth of a system's logs
  • Set the throttle rate at 5 blocks per second
    to simulate generating logs at 5x standard playback speed.

Related issues

A follow up pr (#1683) adds some flushing logic to BufWriter based generators that enables more regular flushing. A separate PR will introduce a new payload format to utilize block-based throttling.

@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 3 times, most recently from c4199fc to 6ed7dca Compare December 31, 2025 16:20
@TheSafo TheSafo changed the title blocks [WIP] Update generator throttler to support block-based strategy Dec 31, 2025
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 11 times, most recently from a5dc28f to 10ba2f9 Compare January 5, 2026 14:50
@TheSafo TheSafo changed the base branch from main to graphite-base/1675 January 5, 2026 16:00
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 10ba2f9 to d724a4c Compare January 5, 2026 16:00
@TheSafo TheSafo changed the base branch from graphite-base/1675 to jsaf/throttle-error-change January 5, 2026 16:00
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 4 times, most recently from 74c2f05 to 9ae8ce6 Compare January 6, 2026 18:11
@TheSafo TheSafo marked this pull request as ready for review January 6, 2026 19:15
@TheSafo TheSafo requested a review from a team as a code owner January 6, 2026 19:15
@TheSafo TheSafo marked this pull request as draft January 6, 2026 19:25
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 2 times, most recently from 3c1df18 to 01c4cd3 Compare January 6, 2026 21:03
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from a636e4c to 661d654 Compare January 6, 2026 21:58
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from af8d207 to cf2d888 Compare January 6, 2026 22:13
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 3 times, most recently from e1d8abc to e73bb21 Compare January 6, 2026 22:31
@TheSafo TheSafo changed the title [WIP] Update generator throttler to support block-based strategy Update generator throttler to support block-based strategy Jan 7, 2026
@TheSafo TheSafo mentioned this pull request Jan 7, 2026
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from cf2d888 to e564637 Compare January 15, 2026 21:34
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from e73bb21 to 57bce08 Compare January 15, 2026 21:34
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from e564637 to a1be97c Compare January 15, 2026 21:35
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 57bce08 to a47d241 Compare January 15, 2026 21:35
@TheSafo TheSafo marked this pull request as ready for review January 16, 2026 18:13
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from a47d241 to 88ac56e Compare January 21, 2026 15:48
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from a1be97c to 47e1784 Compare January 21, 2026 15:48
Base automatically changed from jsaf/add-config-tests to main January 21, 2026 16:22
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 88ac56e to 1692f26 Compare January 21, 2026 18:03
@TheSafo
Copy link
Contributor Author

TheSafo commented Jan 21, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link

gh-worker-devflow-routing-ef8351 bot commented Jan 21, 2026

View all feedbacks in Devflow UI.

2026-01-21 18:04:28 UTC ℹ️ Start processing command /merge


2026-01-21 18:04:38 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2026-01-21 18:14:11 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 25m (p90).


2026-01-21 18:32:01 UTC ℹ️ MergeQueue: This merge request was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants