Summary
Add batching and rate limiting to the webhook notification pipeline to handle high-volume repos gracefully. When many events arrive in quick succession (e.g., a CI run creating multiple check events, or a batch of issues being filed), collapse them into summary messages instead of flooding agents.
RFC: #177 (GitHub Webhook to Swarm Notification Bridge)
Design
Cooldown / Batching
- Track events per repo within a rolling time window
- If 5+ events from the same repo arrive within 60 seconds, batch them into a single summary message
- Summary format:
[GitHub] 7 events on finml-sage/repo in the last 60s: 3 issue comments, 2 PR reviews, 2 issues opened
- After the batch is sent, reset the counter for that repo
- Use an in-memory buffer (dict of repo -> event list with timestamps)
Rate Limiting
- Global rate limit: maximum N webhook-triggered swarm messages per minute (configurable, default 30)
- Per-repo rate limit: maximum M messages per minute per repo (configurable, default 10)
- When rate limited, queue events and send a summary when the window resets
- Prevents webhook abuse (accidental or intentional flood of events)
Configuration
- Sensible defaults that work without configuration
- Optional env var overrides for tuning:
GITHUB_WEBHOOK_BATCH_THRESHOLD (default: 5 events)
GITHUB_WEBHOOK_BATCH_WINDOW (default: 60 seconds)
GITHUB_WEBHOOK_RATE_LIMIT (default: 30 messages/minute global)
Implementation Notes
- Use
asyncio primitives for the batching timer (not threads)
- The batch buffer should be cleaned up periodically to prevent memory leaks from repos that go quiet
- Batch summary should still include the
[GitHub] prefix for consistency
- Individual high-priority events (e.g., new external contributor's first issue) could bypass batching in a future enhancement
Acceptance Criteria
Dependencies
Depends on #180 (event formatting -- batching wraps around the formatted messages).
Summary
Add batching and rate limiting to the webhook notification pipeline to handle high-volume repos gracefully. When many events arrive in quick succession (e.g., a CI run creating multiple check events, or a batch of issues being filed), collapse them into summary messages instead of flooding agents.
RFC: #177 (GitHub Webhook to Swarm Notification Bridge)
Design
Cooldown / Batching
[GitHub] 7 events on finml-sage/repo in the last 60s: 3 issue comments, 2 PR reviews, 2 issues openedRate Limiting
Configuration
GITHUB_WEBHOOK_BATCH_THRESHOLD(default: 5 events)GITHUB_WEBHOOK_BATCH_WINDOW(default: 60 seconds)GITHUB_WEBHOOK_RATE_LIMIT(default: 30 messages/minute global)Implementation Notes
asyncioprimitives for the batching timer (not threads)[GitHub]prefix for consistencyAcceptance Criteria
[GitHub]prefixDependencies
Depends on #180 (event formatting -- batching wraps around the formatted messages).