Why it matters
Partitioning is where skew shows up most visibly. If ten percent of your keys are one special value (nulls, defaults, sentinels), HashPartitioner sends all of them to one reducer and every other reducer sits idle waiting for the slow one to finish. Custom partitioning is the standard fix.
Partitioner choice also determines whether the output is globally sorted, which matters for downstream range scans and merge joins. TotalOrderPartitioner enables cheap ordered output at the cost of a sampling step.
The architecture
HashPartitioner uses (key.hashCode() & Integer.MAX_VALUE) % numReducers. It is fast and simple and gives uniform distribution when hashCode is well-distributed. Almost every job uses it unless there is a reason not to.
TotalOrderPartitioner samples the input to compute range boundaries, then routes each key to the reducer whose range covers it. Output across reducers is globally sorted. This is what Hadoop's terasort benchmark uses.
How it works end to end
A partitioner is invoked in the map task after the map function emits a record and before the record lands in the circular buffer. It runs on every emitted key and returns an integer between 0 and numReducers-1. That integer becomes the partition ID that determines both the position in the buffer and eventually which reducer fetches this record.
Custom partitioners can route based on any function of the key. Common patterns include salting hot keys (append a random suffix to distribute them across multiple reducers) and domain-based routing (put all keys with tenant ID X on the same reducer to co-locate joins).
Partitioner choice interacts with combiner. A partitioner that gives a hot key its own reducer means the combiner has to do the heavy lifting for that key, which may or may not be enough.