Why it matters

HLL is standard in analytics warehouses. 1.6% error at 1.5KB is a great trade.

Advertisement

The architecture

Hash item; bucket by top bits.

Track max leading zeros per bucket.

Harmonic mean estimate.

HyperLogLogHash + buckettop bitsRegistersmax leading zerosEstimateharmonic meanFlajolet et al. 2007; Redis, BigQuery, Snowflake all use HLL for distinct counts
HyperLogLog.
Advertisement

How it works end to end

Redis PFADD/PFCOUNT.

BigQuery APPROX_COUNT_DISTINCT.

Snowflake HLL functions.