Why it matters

Bad plans from missing stats are the #1 cause of Impala performance surprises. A query that should take seconds takes hours because Impala shuffled a huge table it thought was small.

Advertisement

The architecture

COMPUTE STATS gathers table-level (row count, size) and column-level (distinct count, null count, min/max) statistics. Results are stored in the metastore and cached by the Catalog service.

COMPUTE INCREMENTAL STATS refreshes stats per partition rather than whole table. Much faster for large partitioned tables where only recent partitions changed.

Stats collection pathsCOMPUTE STATSfull-table scanCOMPUTE INCREMENTALper-partitionQuery plannerconsults statsStats live in metastore, cached by Catalog, consulted by CBO
Two stats collection modes.
Advertisement

How it works end to end

The CBO uses stats to estimate operator output sizes, choose join orders, decide broadcast vs shuffle joins, and estimate memory needs.

Stats can go stale: table grows, distributions shift, new partitions added. Query plans based on stale stats can be arbitrarily wrong.

Automation: schedule COMPUTE STATS after significant data changes. For incremental tables, run COMPUTE INCREMENTAL STATS on affected partitions.