Why it matters
Even in the DataFrame era, RDD-level knowledge matters for debugging performance issues, writing custom aggregators, and interoperating with legacy code. Direct RDD usage remains appropriate for cases where you need fine-grained control over partitioning or serialization.
The architecture
An RDD has five key properties: a list of partitions (chunks of data), a function to compute each partition, dependencies on parent RDDs, an optional partitioner (for key-value RDDs), and preferred locations for each partition (for locality-aware scheduling).
Transformations (map, filter, join) produce new RDDs with lineage back to the parent. Actions (count, collect, save) trigger computation and return results to the driver.
How it works end to end
Lazy evaluation: transformations don't run when defined. They accumulate lineage. Only actions force execution. This lets Spark optimize the whole plan before running anything.
Recovery: if a partition is lost (executor death), Spark re-executes the transformations in its lineage on some other executor. No replication needed. Trade-off is longer recovery for long lineages.
Persistence: cache() or persist() stores an RDD in memory or disk after first computation. Subsequent uses skip recomputation. This is what makes iterative algorithms fast.