Core concept

Automatic — no persistent state needed for many ops.

Advertisement

How it works

Long lineage = long recompute. checkpoint() breaks lineage.

Advertisement

Trade-offs + gotchas

Every Spark job. Foundation of resilience.