Enhancement: Support Redis Consumer Groups on Data Streams

[hthuwal]

Summary

Add support for the standard Redis consumer group scaling pattern, where multiple consumers process messages from a single shared stream in parallel. Currently the library is limited to exclusive single-consumer ownership of entire streams and cannot be used for this use case.

Background

Redis Streams natively support a consumer group model where multiple consumers belonging to the same group each receive a different subset of messages from a single stream. This is a standard horizontal scaling pattern:

Stream: click-events
  └── Consumer Group: click-processors
        ├── Consumer 1 → receives click1, click4, click7...
        ├── Consumer 2 → receives click2, click5, click8...
        └── Consumer 3 → receives click3, click6, click9...

Each consumer calls XREADGROUP on the same stream, and Redis ensures each message is delivered to exactly one consumer in the group.

Current Limitation

The library is built around a model of exclusive ownership of entire streams — each entry in the Load Balancer Stream (LBS) assigns a whole data stream to a single consumer, protected by a distributed redsync mutex. This makes it architecturally incompatible with the consumer group pattern:

1. Exclusive lock on stream assignment — When an LBS message for click-events is consumed, the receiving consumer acquires an exclusive distributed mutex on that stream name. No other consumer can take ownership.
2. LBS message delivered to exactly one consumer — The LBS uses XREADGROUP with ">", so the assignment is delivered to one consumer instance only.
3. DoneStream permanently removes the LBS entry — The LBS message is XACK'd and XDEL'd on completion, so no other consumer can ever receive it.

There is currently no workaround within the library's API. Users who need this pattern must bypass the library entirely and use go-redis XREADGROUP directly, losing all recovery guarantees.

Proposed Enhancement

1. Document the unsupported paradigm explicitly

Update the README and relevant docs to clearly state that the library does not support the native Redis consumer group parallel processing pattern (multiple consumers on a single shared stream). This prevents users from spending time trying to make it work. Link to this issue for context and to track progress on full support.

2. Add consumer group mode

Introduce a consumer group mode where the library manages failure recovery for the native XREADGROUP / XAUTOCLAIM pattern on a single shared data stream, without the exclusive ownership model. This would complement the existing LBS-based model rather than replace it.

At a high level, this mode would need to handle:

• Reading from a shared stream via XREADGROUP across N consumer instances
• Detecting and recovering stuck/idle messages via XAUTOCLAIM (native Redis recovery)
• Notifying consumers when they should re-process a message claimed from a failed peer

Use Case

Any high-throughput, single-stream workload where horizontal scaling via consumer groups is the right model — e.g., event pipelines, click tracking, audit logs — and where automatic recovery from consumer failure is still desired.

References

• Redis Streams - Consumer Groups
• XREADGROUP command
• XAUTOCLAIM command

[bburli-craft]

Thanks for the thorough writeup. I want to push back on the framing, because I think there's a categorical distinction worth making explicit — and I'll fold it into the README either way.

This library's notion of recovery is about state handoff, not message redelivery. The workloads it targets are ones where the consumer accumulates meaningful in-memory state tied to a stream — a session, a buffer, an open context, a running aggregation — and the stream is just the transport. When that consumer dies, Redis still has the messages; the hard part is that the processing state died with the process, and a successor needs a structured way to take over ownership and resume. That's what the LBS + mutex + fencing machinery exists for, and Redis has no native primitive for it.

The consumer-group load-sharing pattern you describe is, by construction, stateless at the worker level. That's not a coincidence — it's a precondition. N peers can only round-robin messages from one stream if any peer can handle any message interchangeably; the moment a worker accumulates state tied to a specific key or session, you can't load-share anymore and you're back to sticky routing / exclusive ownership. So in the shared pattern, the entire processing state of a message is either contained in the message or already externalized. When a worker dies, there is no state to hand off — XAUTOCLAIM returns the pending entry, another worker processes it from scratch, and correctness is preserved.

That's why I don't think a "consumer group mode" belongs here. It's not that the use case is invalid (it's very real) or that Redis already covers most of it (a convenience wrapper could still exist) — it's that the specific problem this library exists to solve, state handoff on owner death, has no referent in a pattern that is stateless by construction. There is nothing to hand off.

I'll take point #1 — updating the README to state the scope explicitly in these terms, so users evaluating the library can tell immediately whether their workload is the stateful-ownership kind it targets.

Happy to keep discussing if you think the distinction I'm drawing doesn't hold.

[a3y3]

I've read through both these messages and I find myself confused. I think it'll be worth it to get on a quick call to hash this out.

[bburli-craft]

Set one up Monday 20/04.