Redis Streams: Consumer Groups, Retries and Memory, Tested

Redis Streams are an append-only log inside Redis: producers add entries, and consumers read them at their own pace, pick up where they left off after a restart, and acknowledge what they have processed. They fill the gap between Pub/Sub, which forgets every message, and a full message broker. This guide is for developers who need events that survive a crash: order events, activity feeds, audit trails, webhooks to process. It covers the commands you actually use, and the numbers we measured on memory, trimming, crash recovery and read speed.

On this page

• What a stream is
• The six commands you need
• Consumer groups: sharing the work
• Tested: an offline consumer catches up
• Tested: a consumer crashes mid-job
• Tested: memory per entry and trimming
• Tested: read one at a time, or in batches?
• Streams, Pub/Sub, lists or Kafka?
• A worker loop you can copy
• FAQ

What a stream is

A stream is a key holding an ordered list of entries. Each entry has an ID, made of a millisecond timestamp and a sequence number such as 1762500000000-0, and a small set of field-value pairs. New entries go on the end. Entries stay until you trim them, so any consumer can read from any point.

The six commands you need

XGROUP CREATE orders billing 0 MKSTREAM creates the group reading from the oldest entry (and the stream, if it does not exist; use $ instead of 0 to start from new entries only), and XRANGE reads a range of history without a group. The full list is in the Redis Streams documentation.

Consumer groups: sharing the work

A consumer group is a named reader of the stream with its own position. Two things follow:

• Inside one group, each entry goes to one consumer. Run three billing workers and they split the entries between them, like a job queue.
• Each group sees every entry. A billing group and an analytics group both receive every order, independently, like Pub/Sub that remembers.

When a consumer reads an entry, it moves to that group’s pending list until the consumer acknowledges it with XACK. That pending list is what makes crash recovery possible.

Tested: an offline consumer catches up

Setup for all the tests: Redis 8.10.3 in a Linux virtual machine (WSL2) on a laptop, with a Node.js 24 client using ioredis on the same laptop. Default configuration. Measured 7 November 2026.

We created a group, added 100 entries while no consumer was running, then started a consumer.

Compare that with Pub/Sub, where the same test delivered 0 of 100 (see Redis Pub/Sub, tested). This is the main reason to choose Streams: a deploy or a crash does not lose events.

Tested: a consumer crashes mid-job

Consumer c1 read 10 entries and then “crashed” without acknowledging them.

Nothing was lost and nothing needed a manual fix. In production you would set the idle time to longer than your slowest job, for example 60 seconds, and run XAUTOCLAIM from every consumer on a timer.

Two things to plan for:

• An entry can be processed twice. If c1 finished the work but crashed before XACK, c2 repeats it. Make handlers idempotent: check whether the order was already charged before charging it.
• Poison entries. An entry that crashes every consumer is claimed again and again. The extended form XPENDING orders billing - + 100 shows a delivery count per entry; after, say, five attempts, copy it to a dead-letter stream and acknowledge it.

Tested: memory per entry and trimming

We added 100,000 entries with two short fields (user and action), then compared that with a plain list holding the same data as strings.

Streams are compact: entries that share the same field names are packed together, so the field names are not stored again for every entry. But a stream never shrinks on its own. XACK does not delete anything; acknowledged entries stay until you trim. An event stream with no trimming will eventually fill your store.

Trim on every write with MAXLEN ~ N (keep about N entries) or MINID ~ <id> (keep entries newer than a time). The ~ lets Redis trim in efficient chunks rather than to the exact number, which is much cheaper. In our test it kept 10,000 entries in 0.20 MB while 100,000 went through; usually ~ leaves slightly more than N, because it trims whole internal blocks.

At about 21 bytes per small entry, a 128 MB store holds a few million entries before anything else. Larger entries cost more; measure your own with MEMORY USAGE.

Tested: read one at a time, or in batches?

Reading in batches was about 100 times faster than reading one entry at a time, with Redis on the same laptop. Each single read costs a full round trip, so over a real network the gap only widens: at 9 ms per round trip (Mumbai to Mumbai, as measured in our free Redis hosting comparison) a one-at-a-time consumer would manage roughly 50 entries a second. Use COUNT of 50 to 500 and acknowledge the whole batch in one XACK.

Streams, Pub/Sub, lists or Kafka?

Pick Streams when you already run Redis and need durable events with a few consumer groups. Pick a dedicated broker such as Kafka when the event log is the centre of your system, needs to be kept for weeks and is read by many teams. For background jobs with retries and delays, a queue library is less work; see our Redis job queue guide.

A worker loop you can copy

import Redis from 'ioredis';
const r = new Redis(process.env.REDIS_URL);
const STREAM = 'orders', GROUP = 'billing', ME = `c-${process.pid}`;

// '0' = start from the oldest entry, so nothing written before the first worker is missed
await r.xgroup('CREATE', STREAM, GROUP, '0', 'MKSTREAM').catch((e) => {
  if (!String(e.message).includes('BUSYGROUP')) throw e;           // only "group exists" is fine
});

async function handle(id, fields) { /* your idempotent work */ }

for (;;) {
  // 1. take over anything a crashed consumer left behind for over 60 s
  const [, stale] = await r.xautoclaim(STREAM, GROUP, ME, 60000, '0', 'COUNT', 100);
  // 2. read new entries in a batch, waiting up to 5 s
  const res = await r.xreadgroup('GROUP', GROUP, ME, 'COUNT', 200, 'BLOCK', 5000, 'STREAMS', STREAM, '>');
  const entries = [...stale, ...(res ? res[0][1] : [])];
  // (production: check delivery counts with XPENDING STREAM GROUP - + 100 and move entries
  //  delivered 5+ times to a dead-letter stream before handling them)
  for (const [id, fields] of entries) await handle(id, fields);
  if (entries.length) await r.xack(STREAM, GROUP, ...entries.map(([id]) => id));
}

Producers trim as they write:

await r.xadd('orders', 'MAXLEN', '~', 100000, '*', 'id', '42', 'total', '499');

Run the loop as a long-running process, separate from your web server. On Voroa that is a background worker, which never sleeps, next to a key value store in the same Mumbai region; the store’s connection URL goes in an environment variable. Store details are in the Key Value guide.

Run Streams next to your workers

Voroa runs a Redis-compatible key value store in Mumbai beside the services and background workers you deploy, with data written to disk and a free size to start. See the sizes on the managed Redis page.

FAQ

What is the difference between Redis Streams and Pub/Sub? Streams store entries, so a consumer that was offline can catch up, and entries can be acknowledged and retried. Pub/Sub stores nothing. In our tests, Streams delivered 100 of 100 entries to a late consumer; Pub/Sub delivered 0.

Does XACK delete the entry? No. It only removes the entry from the group’s pending list. Trim the stream with MAXLEN or MINID to free memory.

How many consumers can a group have? As many as you like. Each entry goes to one consumer in the group, so adding consumers spreads the load.

Are Redis Streams durable? They are as durable as your Redis persistence. With data written to disk, entries survive a restart, except possibly the last moments before a crash; on a store with no persistence, a restart loses the stream.

Can Redis Streams replace Kafka? For modest volumes and a handful of consumer groups, often yes, and you avoid running another system. For long retention, very high throughput or many teams reading the same log, a dedicated broker fits better.

Do Streams work in Valkey? Yes. Streams are part of the core that Valkey kept from Redis 7.2. See Valkey vs Redis for where the two differ.

Tested 7 November 2026 with Redis 8.10.3 in WSL2 on a laptop. Found something out of date? Write to [email protected].