Redis Data Structures, in depth: choosing types, internal encodings, queues and streams, big keys and cluster limits

The model: keys, types and a single execution thread

A Redis database is a dictionary from a key to one typed value. The type is fixed when the key is created, and a command for the wrong type fails with WRONGTYPE. Commands execute one at a time on the main thread, so each command is atomic and there are no partial reads, but a slow command also blocks every other client. Since Redis 6, I/O threads can help read and write sockets; command execution itself still runs on one thread.

That makes complexity the most important number in the documentation. HGET is O(1), ZADD is O(log n), and SMEMBERS, HGETALL and LRANGE 0 -1 are O(n) in the size of the value. An O(n) command on a key with a million elements pauses the whole instance for tens of milliseconds or more, and that pause shows up as latency for unrelated keys.

Encodings: why small values are cheap

Each logical type has a compact encoding for small values and a general one for large values. A listpack is a single contiguous block of bytes holding entries back to back, each with its own length prefix. It has no per-element pointers, so it is several times smaller than a hash table, but lookups scan it linearly. Redis keeps small hashes, sets, sorted sets and lists in listpacks, and converts a value to the general encoding once it has more entries than a threshold or any element longer than a byte limit. Sets of integers use an intset, a sorted array. Sorted sets graduate to a skiplist plus hash table, which gives O(log n) rank queries and O(1) score lookups.

The thresholds are configuration, and their names changed when listpack replaced the older ziplist in Redis 7.0; Redis 7.2 added listpack encoding for small sets of non-integers. Rather than trusting defaults from a blog post, read them from your server and check real keys:

127.0.0.1:6379> CONFIG GET *-max-listpack-*
127.0.0.1:6379> CONFIG GET set-max-intset-entries
127.0.0.1:6379> HSET user:42 name "Asha" plan "pro"
127.0.0.1:6379> OBJECT ENCODING user:42
"listpack"
127.0.0.1:6379> MEMORY USAGE user:42

This matters for memory design. Ten million users stored as one small hash each stay in listpack encoding; add one long free-text field to each and every hash converts to a hash table, and memory can grow several times over. Do not rely on a value converting back when it shrinks. When you measure, use MEMORY USAGE on representative keys rather than guessing from payload sizes.

Choosing a structure

Hash field expiration (HEXPIRE and related commands) arrived in Redis 7.4; before that, a TTL applied only to a whole key. Sorted-set scores are double-precision floats, so integers above 2^53 lose precision, which matters if you pack a timestamp and a score into one number. HyperLogLog has a standard error of about 0.81 percent.

Queues: lists versus streams

A list with LPUSH and BRPOP is the simplest queue, but a popped item exists only in the worker's memory; if the worker crashes, the job is lost. The reliable pattern moves each item atomically into a per-worker processing list with BLMOVE and removes it only after the work succeeds. A recovery job scans processing lists of dead workers and pushes items back.

A stream keeps entries after they are read. Entry IDs are a millisecond timestamp plus a sequence number, consumer groups track which entries each consumer has received, and a pending list records entries delivered but not acknowledged. XAUTOCLAIM transfers entries that have been pending too long to a live consumer. Trim streams with MAXLEN ~ so they cannot grow forever; the tilde lets Redis trim whole internal nodes, which is much cheaper than exact trimming.

# Reliable list queue: the item is never only in flight in a client's memory
BLMOVE jobs:ready jobs:processing:worker-7 RIGHT LEFT 5   # block up to 5 s
# ... do the work, then acknowledge by removing it from the processing list
LREM jobs:processing:worker-7 1 "<job payload>"

# Stream with a consumer group: replay, pending tracking and reclaim built in
XADD orders MAXLEN ~ 1000000 * order_id 9812 total 42.50
XGROUP CREATE orders billing $ MKSTREAM
XREADGROUP GROUP billing worker-7 COUNT 10 BLOCK 5000 STREAMS orders >
XACK orders billing 1759300000000-0
XAUTOCLAIM orders billing worker-8 60000 0-0 COUNT 10   # take over entries idle > 60 s

Both patterns give at-least-once delivery: a job can run twice if a worker finishes the work and crashes before acknowledging. Make handlers idempotent, for example by recording processed job IDs with SET NX. Neither makes Redis a replacement for a replicated log such as Kafka when durability across node loss matters, because replication is asynchronous and an acknowledged write can be lost on failover.

Worked example: a game backend

A match service records results for millions of players. Each player's profile is a small hash, the season leaderboard is one sorted set, daily active users are a HyperLogLog per day, a bitmap marks who has played this season, and each player's recent activity is a stream capped at 200 entries. One pipelined transaction writes all of it in a single round trip.

import time
import redis

r = redis.Redis(decode_responses=True)

def record_match(player_id, season, points, match_id):
    now_ms = int(time.time() * 1000)
    p = r.pipeline(transaction=True)                      # MULTI/EXEC: all or nothing
    p.hset(f"player:{player_id}", mapping={"last_match": match_id})
    p.hincrby(f"player:{player_id}", "matches", 1)
    p.zincrby(f"lb:{season}", points, player_id)          # leaderboard
    p.pfadd(f"active:{time.strftime('%Y-%m-%d')}", player_id)   # daily uniques
    p.setbit(f"played:{season}", int(player_id), 1)        # dense numeric ids only
    p.xadd(f"feed:{player_id}", {"match": match_id, "pts": points},
           maxlen=200, approximate=True)                   # bounded activity feed
    p.execute()

def top10(season):
    return r.zrange(f"lb:{season}", 0, 9, desc=True, withscores=True)

def rank_of(player_id, season):
    return r.zrevrank(f"lb:{season}", player_id)          # 0-based, None if absent

Walk through the costs. The hash writes are O(1) and the profile stays in listpack encoding because it has a handful of short fields. ZINCRBY is O(log n); with ten million players that is a few dozen pointer hops, microseconds of work. The top-10 query is O(log n + 10). The daily HyperLogLog is at most about 12 KB however many players are active, and PFMERGE across seven days gives weekly actives. The bitmap costs one bit per possible player ID, so ten million IDs cost about 1.2 MB, but a single ID of four billion would allocate about 512 MB; only use bitmaps for dense IDs.

What would break it: an unbounded stream per player, a leaderboard shared by several seasons that is never deleted, or an admin page calling ZRANGE lb:2026 0 -1 on ten million members. In Redis Cluster the transaction above also fails, because its keys hash to different slots. Tag the player's hash and feed as {player_id} and write them in one MULTI, then send the season-wide updates separately, guarded by a SET NX on the match ID, because a retried ZINCRBY would count the points twice.

Big keys, O(n) commands and safe iteration

Find big keys before they find you. redis-cli --bigkeys and redis-cli --memkeys sample the keyspace with SCAN and report the largest keys per type. The latency monitor and SLOWLOG GET show which commands actually took time.

• Never run KEYS in production; use SCAN, and HSCAN, SSCAN and ZSCAN inside a value. Scans are cursor based and may return an element more than once, so make callers tolerate duplicates.
• Delete large values with UNLINK, which frees memory in a background thread, rather than DEL, which frees it inline. Enabling the lazyfree options makes eviction and expiry of big keys non-blocking too.
• Read large collections in pages: ZRANGE key 0 99, LRANGE key 0 99, HSCAN with COUNT.
• Split values that grow without bound, for example a per-day key instead of one ever-growing set, and set a TTL on every key that is not permanent.

Cluster, memory and eviction

Redis Cluster splits the keyspace into 16,384 hash slots, and a multi-key command or transaction only works when all its keys are in one slot. A hash tag controls this: only the part of the key inside the first pair of braces is hashed, so player:{42}:profile and feed:{42} land together. Use tags to group one entity's keys, never to put a whole feature in one slot, or that slot's node becomes a hot spot.

When memory reaches maxmemory, the maxmemory-policy decides what happens. noeviction rejects writes, which is right when Redis holds data you cannot rebuild. The allkeys-lru and allkeys-lfu policies suit a pure cache. The volatile-* policies only evict keys with a TTL, which surprises teams whose keys have none. Eviction removes whole keys, so a huge sorted set is evicted all at once or not at all.

Failure modes and trade-offs

• Encoding cliffs. One long field converts a compact value to a hash table and memory jumps. Keep long text in its own string key.
• Blocking commands. O(n) reads, big deletes and long Lua scripts stall every client. Page, use UNLINK and keep scripts short.
• Hot keys. One global leaderboard or counter concentrates load on one thread and, in a cluster, one node. Shard the key and merge on read.
• Failover data loss. Replication is asynchronous, so the last writes can vanish on promotion. WAIT reduces but does not remove the window.
• Double processing. List and stream queues are at-least-once. Handlers must be idempotent.
• Precision. Sorted-set scores are doubles; use a secondary key or member naming for tie-breaks rather than packing more than 53 bits.

Related reading

Sorted sets and single-key scripts power rate limiter architecture; hot-key splitting is covered in hot-key mitigation architecture; cache invalidation patterns in caching: CDN, application, database and invalidation; and running Redis as a managed service in GCP Memorystore.

What to do next

1. Run redis-cli --bigkeys and --memkeys against a replica and list every key over a size you would not want to read or delete in one call.
2. Read your *-max-listpack-* settings and check OBJECT ENCODING on representative keys of each type.
3. Search your code for KEYS, HGETALL, SMEMBERS and full-range reads on unbounded values, and replace them with scans or pages.
4. Give every list and stream a trim policy and every non-permanent key a TTL.
5. Convert any RPOP-based queue to BLMOVE with a processing list, or to a stream with a consumer group, and make handlers idempotent.
6. Before moving to Redis Cluster, list every multi-key command and design hash tags per entity.
7. Choose a maxmemory-policy deliberately and alert on memory, evictions and slow-log entries.