Core concept

sc.setCheckpointDir("hdfs:///checkpoint")
df.checkpoint()
Advertisement

How it works

Slow (IO). Reliable recovery.

Advertisement

Trade-offs + gotchas

Very long lineage. Streaming checkpointing for state recovery.