Why it matters

Probabilistic counting scales to billions of distinct items in KB memory. Standard in analytics.

Advertisement

The architecture

Hash items; observe leading zeros.

Estimate count from max.

Merge sketches by max.

Probabilistic countingHash itemsget bitsTrackmax leading zerosEstimatedistinct countFlajolet-Martin 1985; HyperLogLog 2007 gives 1.6% error at 1.5KB
Probabilistic counting.
Advertisement

How it works end to end

Hash to observe leading zeros.

Estimate = 2^max.

Merge sketches by max.