All 32 articles, sorted alphabetically
Spark Adaptive Query Execution (AQE)
How AQE reoptimizes Spark queries at runtime based on actual data statistics.
Read article →Spark barrier execution mode architecture
Deep-dive on Spark barrier execution mode: gang scheduling that launches all N tasks of a stage at once so they can run MPI-style all-to-all communica…
Read article →Spark runtime bloom-filter join architecture
Deep-dive on Spark's runtime bloom-filter (dynamic filtering) join: why shuffle dominates cost, building a bloom filter from …
Read article →Spark Broadcast Join
How broadcast join avoids shuffle by broadcasting the small side to every executor.
Read article →Spark Connect
3.4+. Thin client model.
Read article →Spark DataSource V2 architecture
Deep-dive on Spark DataSource V2: TableCatalog and DDL, capability sets, the ScanBuilder pushdown negotiation and the unhandled-filter contract, stati…
Read article →Spark DataFrame and Dataset
How DataFrames add schema and columnar optimization to Spark, how Datasets extend DataFrames with type safety, and when to use each.
Read article →Delta Lake
How Delta Lake adds ACID transactions, schema evolution, and time travel to Spark data lakes.
Read article →Spark Dynamic Allocation
How dynamic allocation adds/removes executors based on workload demand.
Read article →Spark Execution Architecture in Depth
A 2500-word walkthrough of Apache Spark's execution architecture: SparkSession, driver, cluster manager, Catalyst, DAG, execu…
Read article →Spark EXPLAIN Plans
Physical plan inspection.
Read article →Spark external shuffle service architecture - decoupling shuffle from executor lifetime
Deep-dive on Spark's external shuffle service: sort-based shuffle file layout and index, Netty zero-copy serving, surviving e…
Read article →Apache Hudi with Spark
How Hudi enables streaming-first data lakes with upserts + incremental queries.
Read article →Apache Iceberg with Spark
How Iceberg brings ACID + time travel + schema evolution to Spark data lakes.
Read article →Spark on Kubernetes architecture
Deep-dive on Spark on Kubernetes: operator, driver/executor pods, K8s scheduler, dynamic allocation, external shuffle, node pools.
Read article →Spark unified memory management architecture
Deep-dive on Spark memory: the reserved/user/unified heap layout, unified execution and storage with borrow-and-evict, the eviction asymmetry, spill-t…
Read article →Apache Spark Overview
What Spark is, how RDDs and DataFrames give in-memory distributed computing, and where Spark fits versus MapReduce, Flink, and modern warehouses.
Read article →Spark PageRank
Classic algorithm on GraphFrames.
Read article →Partition pruning -- skipping data you don't need to read
Deep-dive on Spark partition pruning: avoiding reading data (the fastest scan is the one you never do), static partition pruning (filter on the partit…
Read article →Spark Photon architecture
Deep-dive on Photon: a C++ vectorized execution engine beneath Spark SQL. How the planner splits the physical plan into native and JVM operators, how …
Read article →Spark RDD lineage architecture
Deep-dive on Spark RDD lineage: lazy transformations building a DAG, narrow vs wide dependencies, stage cutting at shuffle boundaries, recomputation o…
Read article →Spark Security
Kerberos + Ranger + TLS + encrypted shuffle.
Read article →Spark Shuffle
How Spark shuffle works internally, why it's usually the slowest phase, and how to minimize shuffle bytes.
Read article →Spark SQL
How Spark SQL provides ANSI-compatible SQL over any DataFrame source (Parquet, Delta, JDBC, Kafka), the query engine architecture, and Thrift server.
Read article →Spark SQL optimizer architecture
Deep-dive on Catalyst optimizer: analyzer, rule-based rewrites, cost model, physical planner, Tungsten codegen, AQE, and extensions.
Read article →Spark Stages and Tasks
How stages are built from transformations, how tasks are the unit of parallelism, and how partition count drives everything.
Read article →Spark Structured Streaming Architecture in Depth
A 2500-word walkthrough of Spark Structured Streaming: sources, streaming DataFrame, watermarks, state store, aggregations, sinks, checkpointing, exac…
Read article →Structured streaming + Kafka architecture
Deep-dive on Spark structured streaming + Kafka: offset tracking, checkpoint, exactly-once, rate limits, schema, restart.
Read article →Structured streaming watermark architecture
Deep-dive on Spark structured streaming watermarks: event time, watermark math, state, trigger, output modes, late data, checkpoint.
Read article →Spark Structured Streaming state
Deep-dive on Spark Structured Streaming state management: keyed state stores, checkpointing for exactly-once recovery, watermarks bounding state and h…
Read article →Spark Tungsten Execution Engine
Memory + CPU optimizations.
Read article →Spark whole-stage code generation architecture
Deep-dive on Spark SQL whole-stage code generation: replacing the Volcano iterator model with a fused Janino-compiled Java loop via the produce/consum…
Read article →