Loki vs Elastic vs ClickHouse for Logs

Three philosophies of a log index

Every log store makes one central bet: how much of each log line do I index up front, and how much do I scan at query time? Indexing costs write throughput, memory, and disk now, in exchange for fast reads later. Scanning defers that cost to read time. The three systems sit at three points on that spectrum.

Elasticsearch builds an inverted index over every field it’s told to index — by default, a lot of them. Every token in every message becomes a posting list pointing at the documents that contain it. A search for a string is then a dictionary lookup, not a scan, which is why Elastic feels instant for ‘find the request with this trace ID.’ The bill arrives as index size and heap pressure.

Loki takes the opposite bet. It indexes only a small set of labels (service, namespace, level, pod…) and stores the raw log lines as compressed chunks. There is no inverted index over message content at all. A query filters chunks by label, then linearly greps the matching chunks. Cheap to write, cheap to store — you pay at query time, and only for the chunks your labels didn’t already exclude.

ClickHouse is a columnar OLAP database that happens to be extraordinary at logs. Each field becomes a column, stored contiguously and compressed hard. There’s no full inverted index by default; instead it relies on sorting, sparse primary indexes, and optional skip indexes to avoid reading columns it doesn’t need. You query with SQL. It scans, but it scans columns at billions of rows per second, so aggregations that would melt the other two are routine.

Elasticsearch: index everything, search anything

Elastic’s superpower is that any field is a first-class query target. Full-text search on the message body, exact-match on a status code, range on latency, wildcard on a hostname — all fast, because Lucene has already inverted them. For investigation-heavy work — an SRE pivoting from a user ID to a trace ID to an error string at 3 a.m., or a security team hunting an IOC across months — nothing beats having every field pre-indexed.

The cost is proportional. The inverted index, doc values, and (unless you disable it) the stored _source commonly push total on-disk size to roughly 1–3× the raw log volume. Indexing is CPU- and memory-hungry: analysis, tokenization, and segment merges all compete for the same heap that serves queries. And the index is a living thing — segments merge, shards rebalance, and hot data ages into warm and cold tiers under Index Lifecycle Management (ILM). That operational surface is the real Elastic tax, more than the disk.

The instinct to ‘just index it all’ is also where Elastic gets you into trouble, because mapping explosions and high-cardinality fields turn the index from an asset into a liability — the subject of the cardinality section below.

Loki: index the labels, grep the rest

Loki’s design goal was explicit: be to logs what Prometheus is to metrics — cheap, label-driven, and boring to operate. It keeps a small index that maps label sets to chunks of compressed log lines. The lines themselves are never tokenized or inverted; they sit in object storage (S3/GCS) as gzip/snappy blobs.

A LogQL query has two phases. First the label matchers ({service="checkout", level="error"}) select which chunks to open — this part is index-fast. Then any line filter (|= "timeout") is a linear scan over the bytes of those chunks. So Loki is fast exactly when your labels are selective and slow when they aren’t: a filter over a broad label set means grepping gigabytes.

{service="checkout", level="error"}   # index: pick chunks
  |= "payment"                        # scan: grep the bytes
  | json                               # parse per line
  | latency_ms > 500                    # filter on parsed field

Because it indexes so little, Loki’s storage footprint is the smallest of the three — often ~0.3× raw or less after chunk compression — and its write path is light. The flip side: it has no idea what’s inside a line until it reads it, so content-heavy or analytical queries do real I/O every time. Loki rewards teams whose day-to-day question is ‘show me this service’s recent errors,’ not ‘compute p99 latency by route across last month.’

ClickHouse: columns, compression, and SQL

ClickHouse treats a log stream as a wide, append-only table. timestamp, service, level, trace_id, message, and structured fields each become a column. Columns store values of one type contiguously, which is why they compress so brutally well — a level column of ‘INFO’ repeated a million times collapses to almost nothing, and a sorted timestamp column encodes as deltas. Real-world log tables land around 0.1–0.2× raw, the tightest of the three.

Queries are SQL, and that changes what ‘log analysis’ even means. Joins, window functions, GROUP BY, quantiles, and time-bucketed rollups are all native and fast because the engine reads only the columns a query touches, at vectorized, billions-of-rows-per-second speeds.

SELECT route,
       count() AS errors,
       quantile(0.99)(latency_ms) AS p99
FROM logs
WHERE timestamp > now() - INTERVAL 1 HOUR
  AND status >= 500
GROUP BY route
ORDER BY errors DESC;

That query — ‘how many 500s by route in the last hour, with tail latency’ — is where ClickHouse simply outclasses the others. What you give up is turnkey full-text search: finding an arbitrary substring in message is a scan unless you’ve added a token-based skip index or bloom filter, and even then it’s coarser than Lucene’s inverted index. You also inherit a database to schema, partition, and merge — ClickHouse is less ‘drop in logs’ and more ‘design a table.’

Schema-on-write vs schema-on-read

Underneath the indexing question is a structural one: when do you decide what the fields are? This split — schema-on-write versus schema-on-read — shapes how each system copes with the messy reality of logs, which rarely arrive with a clean, stable structure.

Elasticsearch and ClickHouse lean schema-on-write: fields are known at ingest and given a type. Elastic can auto-detect (dynamic mapping) or be told explicitly; ClickHouse wants a declared table. The payoff is fast, typed queries; the price is that a new or malformed field is a schema event — a mapping conflict in Elastic (a string where a number was expected can reject the document), or a column you must ALTER TABLE to add in ClickHouse. Semi-structured escape hatches exist — Elastic’s flattened type, ClickHouse’s Map and JSON columns — but they trade some query speed for flexibility.

Loki is the schema-on-read extreme: it stores raw lines and parses fields at query time with | json or | logfmt. Nothing to migrate, nothing to reject — any log line is welcome — but every query that needs a field pays to parse it, and there’s no type enforcement to catch a field that silently changed shape. If your log formats are volatile, Loki’s tolerance is a feature; if they’re stable and you query the same fields constantly, pre-typing them in Elastic or ClickHouse pays off every single query.

The cardinality story

This is the axis that quietly decides most real deployments, and it bites each system differently. Cardinality is the number of distinct values a field takes — low for level (a handful), astronomically high for trace_id, user_id, or a raw URL with query strings.

Elasticsearch handles high-cardinality values reasonably — that’s what an inverted index is for — but suffers from high-cardinality fields (a mapping explosion). If logs contain dynamic JSON keys, every new key becomes a new field in the mapping; thousands of fields bloat cluster state, balloon memory, and can tip a cluster over. The fix is disciplined mappings, flattened types, or dropping dynamic fields — but the failure mode is real and common.

Loki has the sharpest cliff of all, and it’s the number-one way people misuse it: putting a high-cardinality value into a label. Because each unique label-set combination creates a separate stream (and its own chunks), adding trace_id or user_id as a label can spawn millions of tiny streams — a cardinality explosion that wrecks the index, destroys compression, and can OOM the ingesters. The rule is iron: labels are for low-cardinality dimensions you slice by; everything high-cardinality stays in the log line and is found with a filter or, in newer Loki, a structured-metadata / bloom accelerator. Get this wrong and Loki’s cost advantage evaporates.

ClickHouse is the most tolerant. A high-cardinality column is just a column; it compresses (perhaps less well) and costs storage, but it doesn’t threaten the cluster’s stability the way a Loki label or an Elastic mapping does. You can even keep trace_id in the sort key or a skip index to make point lookups fast. This tolerance is a big part of why ClickHouse scales to very wide, very high-cardinality log schemas that would force painful compromises elsewhere.

Storage and compression economics

Roll the three together and the storage story is stark. For the same ingested volume you can expect, very roughly:

An order of magnitude separates the extremes. At a terabyte a day, that is the difference between a storage bill you barely notice and one that dominates the observability budget. But storage is only half the ledger. Elastic’s cost is also hot resources — RAM and CPU to keep indices queryable — whereas Loki and ClickHouse push cold data to cheap object storage and read it on demand. Two systems can have the same disk footprint and very different monthly bills depending on how much must stay hot. Always compare total cost of ownership, not just gigabytes.

The write path: ingestion and backpressure

Ingestion characteristics differ as much as reads. Elastic’s indexing is the expensive part of its write path: analysis, inversion, and continuous segment merges consume CPU and heap, and a heavy indexing load directly steals resources from queries. Bulk indexing, refresh-interval tuning, and enough hot nodes are the levers; get them wrong and you see rejected bulk requests and rising merge backlogs.

Loki has the lightest write path by design — append lines to chunks, flush to object storage — which is why it ingests high volume cheaply. Its ingestion risk is not CPU but stream count: too many active streams (there’s that cardinality cliff again) blows up ingester memory.

ClickHouse wants large, batched inserts. Each insert creates a data part, and parts merge in the background; a flood of tiny inserts creates too many parts and triggers ‘too many parts’ backpressure. The standard pattern is to buffer through Kafka or an async insert layer and write in chunks of thousands to millions of rows. Done right, a single node ingests enormous throughput; done naively (row-at-a-time), it stalls.

Query languages: Lucene/KQL vs LogQL vs SQL

The query language you’ll live in every day is a real ergonomic difference, not a footnote.

Elastic exposes Lucene/KQL and the JSON Query DSL (plus ES|QL in newer versions). KQL is comfortable for search-style questions — status:500 and service:checkout — and the DSL exposes the full power of aggregations, but the JSON grows verbose fast for anything analytical.

Loki’s LogQL deliberately echoes PromQL: label selectors, line filters, parsers (| json, | logfmt), and metric queries that turn logs into rate/aggregation time series (rate({...} |= "error" [5m])). It’s elegant for label-driven exploration and metrics-from-logs, and clunky for rich relational analysis.

ClickHouse is just SQL — the most widely known query language on earth, with joins, subqueries, window functions, and a huge function library. That is a genuine advantage for analysts and for reusing existing SQL tooling and BI, at the cost of SQL being wordier than a one-line label selector for the simple ‘show me this service’s errors’ case.

Query speed depends on the shape of the query

“Which is fastest?” has no answer without naming the query, because the three systems win different workloads. Split log queries into two archetypes:

Needle-in-a-haystack search — ‘find the one request with this trace ID,’ ‘show lines containing this exact error.’ Here Elastic wins decisively: the inverted index turns it into a lookup. Loki is fine if your labels narrow the haystack first, and painful if they don’t. ClickHouse scans unless you’ve pre-built a skip index for that field.

Aggregation and analytics — ‘count 500s by route,’ ‘p99 latency per endpoint over 24h,’ ‘top talkers by bytes.’ Here ClickHouse wins decisively: columnar scans and vectorized execution chew through billions of rows. Elastic can do it with aggregations but pays in heap and slows under high cardinality; Loki’s metric queries can compute rates but must scan the underlying chunks, so they degrade on large ranges.

The crossover between these two archetypes is the entire decision. A team that mostly searches leans Elastic; a team that mostly aggregates leans ClickHouse; a team that mostly tails and filters by service at minimal cost leans Loki. Most real teams do some of each, which is why hybrid stacks (below) exist.

Operational burden: what breaks and who fixes it

Whatever you pick, you operate it — and the day-two experience varies widely.

Elastic is the heaviest to run: shard sizing, replica counts, ILM policies for hot/warm/cold/frozen tiers, heap tuning, mapping hygiene, and the ever-present risk of a red cluster from a runaway query or a mapping explosion. It rewards a team with real Elastic expertise and punishes one without.

Loki is the lightest conceptually — object storage does the durability, and there’s little index to corrupt — but its microservice-style deployment (distributor, ingester, querier, compactor) and its unforgiving cardinality rules mean the failure modes are subtle: a bad label quietly degrades everything until someone reads the stream-count metrics.

ClickHouse sits in between: it’s a database, so you own schema design, partitioning, TTL-based retention, and merge/parts health, plus replication and sharding if you go distributed. The concepts are familiar to anyone who has run an OLAP store, but they are concepts you must actively manage, not autopilot.

Cost at scale and the hybrid escape hatch

At small volume, any of the three is cheap and the choice is about ergonomics. At scale — terabytes a day, months of retention — the storage and compute multipliers above compound into very different bills, and cost becomes a primary driver. Elastic’s hot-resource appetite is what pushes high-volume teams to look elsewhere for the bulk of their logs.

Which is why many mature stacks don’t pick one — they tier. A common pattern: keep everything cheaply in Loki or ClickHouse for volume and analytics, and index only a high-value subset (errors, security events, audited fields) into Elastic for rich search. Another: ClickHouse as the analytical backbone with Loki for cheap live-tail. Grafana front-ends all three, which makes mixing them operationally palatable. The honest answer to ‘which one’ is often ‘two of them, for different jobs.’

A decision framework

Strip away the tribalism and it comes down to matching the system to your dominant query shape and your budget:

Then sanity-check against the traps: are you tempted to put a high-cardinality value in a Loki label (don’t) or to let Elastic auto-map dynamic JSON (don’t)? Do you have the SQL discipline ClickHouse ingestion wants (batch your inserts)? The right choice is the one whose strengths line up with your most frequent, most painful query — and whose failure mode you can live with.