Apache Spark

Spark Accumulators

Write-only counters across executors.

Spark Actions vs Transformations

Lazy transformations. Eager actions.

Spark Adaptive Query Execution (AQE)

How AQE reoptimizes Spark queries at runtime based on actual data statistics.

Spark AQE architecture

Deep-dive on Adaptive Query Execution: query stages at shuffle boundaries, MapOutputStatistics-driven re-planning, partition coalescing and advisory s…

Spark Architecture

The internal architecture of Spark: how the driver orchestrates executors via the cluster manager, and how task scheduling works.

Spark Arrow

How Apache Arrow accelerates Spark's Python + R + native integrations.

Spark + Avro

Row-based with schema evolution.

Spark Broadcast Variables

Ship small data to all executors once.

Spark Broadcast Hints

Force broadcast join.

Spark Broadcast Join

How broadcast join avoids shuffle by broadcasting the small side to every executor.

Spark broadcast joins -- avoiding the shuffle for large-small joins

Deep-dive on Spark broadcast joins: the join/shuffle problem, broadcasting the small side to avoid shuffling the large side, the auto-broadcast thresh…

Spark cache() + persist()

Store in memory for reuse.

Spark Cache Strategy

When to cache + storage level.

Spark Catalyst Optimizer

SQL query optimizer.

Spark Cost-Based Optimizer

Uses stats for plan selection.

Spark Checkpointing

Break lineage. Write to reliable storage.

Spark Cluster Managers

Standalone, YARN, K8s, Mesos.

Spark coalesce vs repartition

Reduce vs shuffle partitions.

Spark Columnar Processing

Vectorized reads + processing.

Spark DataFrame Complex Types

Array, Map, Struct handling.

Spark Connect

3.4+. Thin client model.

Spark Connected Components

Graph component detection.

Spark Cost Optimization

Cluster sizing + spot + Delta.

Spark DAG (Directed Acyclic Graph)

Logical plan of transformations.

Spark Data Skew Handling

AQE + salting + broadcast.

Spark on Databricks

Managed Spark + Delta + notebooks.

Spark DataFrame API

Schema-aware. Catalyst optimization.

Spark DataFrame and Dataset

How DataFrames add schema and columnar optimization to Spark, how Datasets extend DataFrames with type safety, and when to use each.

Spark on GCP Dataproc

Managed Spark on GCP. Fast startup.

Spark Dataset API

Typed DataFrame. Scala only.

Spark Debugging

How to debug Spark: driver logs, executor logs, UI, remote debug.

Delta Lake architecture

Deep-dive on Delta Lake: ACID commit, transaction log, time travel, schema enforcement, Z-order + optimize, vacuum, MERGE + UPDATE.

Delta Lake

How Delta Lake adds ACID transactions, schema evolution, and time travel to Spark data lakes.

Delta Lake Architecture in Depth

A 2500-word walkthrough of Delta Lake: transaction log, Parquet data files, checkpoints, OPTIMIZE + Z-ORDER, MERGE, time travel, streaming, VACUUM.

Spark Driver + Executor

Driver coordinates, executors run tasks.

Spark DStreams (Legacy)

First-gen streaming. Deprecated.

Spark Dynamic Allocation

How dynamic allocation adds/removes executors based on workload demand.

Spark on AWS EMR

Managed Hadoop-Spark on AWS.

Spark Encryption

How to enable encryption for Spark: shuffle files, RPC, at-rest storage.

Spark Execution Architecture in Depth

A 2500-word walkthrough of Apache Spark's execution architecture: SparkSession, driver, cluster manager, Catalyst, DAG, execu…

Spark Execution Model

How Spark turns a user program into jobs, jobs into stages, and stages into tasks. The DAG scheduler and task scheduler working together.

Spark Executor Sizing

Cores + memory per executor.

Spark EXPLAIN Plans

Physical plan inspection.

Spark GC Tuning

G1GC + heap sizing.

Spark GPU Acceleration

How NVIDIA RAPIDS accelerates Spark SQL on GPUs.

Spark GraphFrames

DataFrame-based graph API.

Spark GraphX

How GraphX enables distributed graph algorithms on Spark.

Spark HBase Connector

Read/write HBase from Spark.

Spark History Server

How to use the Spark History Server to analyze finished jobs.

Apache Hudi with Spark

How Hudi enables streaming-first data lakes with upserts + incremental queries.

Spark + Hudi architecture

Deep-dive on Spark with Apache Hudi: write path (upsert/insert), read path (snapshot/incremental/CDC), procedures, timeline, catalog.

Apache Iceberg with Spark

How Iceberg brings ACID + time travel + schema evolution to Spark data lakes.

Spark + Iceberg architecture

Deep-dive on Spark integration with Iceberg: extension, write/read paths, procedures, Catalyst push-down, streaming, MERGE, catalogs.

Spark + JDBC

Read from RDBMS. Partition scans.

Spark Join Strategy Hints

Force broadcast, merge, shuffle_hash.

Spark JSON Read/Write

Schema inference + explicit.

Spark + Jupyter Integration

PySpark in notebooks.

Koalas → Spark Pandas API Migration

How Koalas was renamed and integrated into Spark's pandas API.

Spark Kryo Serialization

How Kryo serialization speeds up Spark shuffles and cache versus default Java serialization.

Spark on Kubernetes architecture

Deep-dive on Spark on Kubernetes: operator, driver/executor pods, K8s scheduler, dynamic allocation, external shuffle, node pools.

Spark on Kubernetes

How Spark deploys on Kubernetes as native workload.

Spark Lineage + Fault Tolerance

Recompute lost partitions from lineage.

Apache Livy

Submit + monitor Spark via HTTP.

Spark Memory Tuning

executor.memory + fractions.

Spark Version Migration Strategies

How to migrate between Spark versions safely.

Spark ML Overview

What Spark ML provides: DataFrame-based ML pipelines at scale.

Spark ML Classification

LR, RF, GBT, DecisionTree.

Spark ML Clustering

KMeans, GMM, LDA, BisectingKMeans.

Spark ML Cross-Validation

CV + parameter grid search.

Spark ML Feature Engineering

VectorAssembler + scalers + one-hot.

Spark ML Model Persistence

Save + load models.

Spark ML Pipelines

How to build Spark ML Pipelines: chained transformers + estimators.

Spark ML Recommendation

ALS collaborative filtering.

Spark ML Regression

LinearRegression, RF, GLM.

Spark ML Transformers + Estimators

Feature engineering foundation.

Spark MLlib

What MLlib is: legacy RDD-based ML. When to use vs spark.ml.

Spark MLlib Introduction

Distributed ML library.

Spark Monitoring

Spark UI + metrics + Prometheus.

Spark on Kubernetes

K8s cluster manager. Modern default.

Spark Operator on K8s

CRD-based Spark app management.

Spark + ORC

Columnar alternative to Parquet.

Apache Spark Overview

What Spark is, how RDDs and DataFrames give in-memory distributed computing, and where Spark fits versus MapReduce, Flink, and modern warehouses.

Spark PageRank

Classic algorithm on GraphFrames.

Spark Pandas API

How Spark's pandas API provides pandas-like interface on Spark.

Spark Pandas UDFs

Vectorized. Fast Python UDFs.

Spark Parquet Read + Write

Columnar default.

Partition pruning -- skipping data you don't need to read

Deep-dive on Spark partition pruning: avoiding reading data (the fastest scan is the one you never do), static partition pruning (filter on the partit…

Spark Partitioning

How partitioning affects Spark performance: partition count, key selection, skew handling.

Spark Persistence

How cache() and persist() reuse computed RDDs/DataFrames across actions.

Spark Predicate Pushdown

Push filters to storage.

Spark Push-Based Shuffle

3.2+. Server-side merge.

PySpark Performance

How to optimize PySpark: pandas UDFs, arrow, catalyst.

Spark RDDs

How RDDs represent distributed collections with lineage-based recovery, when to use them, and why DataFrames replaced them for most workloads.

Spark RDD Deep Dive

Resilient Distributed Dataset. Low-level API.

Spark S3 Optimization

S3A committer + tuning.

Spark Salting for Skew Joins

Random prefix key. Split hot keys.

Spark Secrets Management

How to manage secrets in Spark: never in code, injection at runtime.

Spark Security

Kerberos + Ranger + TLS + encrypted shuffle.

Spark Shuffle

How Spark shuffle works internally, why it's usually the slowest phase, and how to minimize shuffle bytes.

Spark shuffle architecture

Deep-dive on Spark shuffle: map output + partitioner + external shuffle service + reduce fetch, plus push-based shuffle and AQE.

Spark Shuffle Deep Dive

Data redistribution. Expensive.

Spark Shuffle Hash Join

Alternative. Not default.

Spark External Shuffle Service

How external shuffle service enables executor removal without losing shuffle files.

Spark Shuffle Tuning

Reduce shuffle + tune partitions.

Spark Sort-Merge Join

Default for large tables. Both shuffled + sorted.

Spark on Spot Instances

Cost savings + resilience.

Spark SQL

How Spark SQL provides ANSI-compatible SQL over any DataFrame source (Parquet, Delta, JDBC, Kafka), the query engine architecture, and Thrift server.

Spark SQL Config Tuning

shuffle.partitions + others.

Spark SQL Functions

Built-in functions library.

Spark SQL optimizer architecture

Deep-dive on Catalyst optimizer: analyzer, rule-based rewrites, cost model, physical planner, Tungsten codegen, AQE, and extensions.

Spark Stages and Tasks

How stages are built from transformations, how tasks are the unit of parallelism, and how partition count drives everything.

Spark Statistics + CBO

Table stats for cost-based decisions.

Spark Structured Streaming Deduplication

How to deduplicate streaming records via dropDuplicates.

Spark Structured Streaming Stateful Operations

How stateful operations work in Structured Streaming: state store, checkpointing.

Spark Structured Streaming Watermarking

How watermarks handle late-arriving data in streaming.

Spark Structured Streaming Architecture in Depth

A 2500-word walkthrough of Spark Structured Streaming: sources, streaming DataFrame, watermarks, state store, aggregations, sinks, checkpointing, exac…

Spark Streaming Checkpointing

Fault tolerance state.

Spark Streaming Deduplication

dropDuplicates with watermark.

Spark Streaming Event vs Processing Time

Semantic differences.

Spark Streaming Exactly-Once

Idempotent sink + checkpointing.

Spark Streaming foreachBatch

Custom per-batch logic.

Spark Streaming Joins

Stream-static, stream-stream.

Structured streaming + Kafka architecture

Deep-dive on Spark structured streaming + Kafka: offset tracking, checkpoint, exactly-once, rate limits, schema, restart.

Spark Streaming Kafka Source

Read from Kafka topics.

Spark Streaming Output Modes

Append, Update, Complete.

Spark Streaming Sinks

Kafka, files, foreach, Delta.

Spark Streaming Sources

Kafka, Kinesis, files, socket.

Spark Streaming Stateful Operations

flatMapGroupsWithState + agg.

Spark Streaming Triggers

Micro-batch cadence.

Structured streaming watermark architecture

Deep-dive on Spark structured streaming watermarks: event time, watermark math, state, trigger, output modes, late data, checkpoint.

Spark Streaming Watermarking

Late event handling.

Spark Structured Streaming

DataFrame-based streaming.

Spark Structured Streaming state

Deep-dive on Spark Structured Streaming state management: keyed state stores, checkpointing for exactly-once recovery, watermarks bounding state and h…

Spark Triangle Count

Graph triangle enumeration.

Spark Troubleshooting

The common Spark issues (OOM, skew, slow shuffle, straggler tasks) and how to diagnose + fix.

Spark Tungsten Execution Engine

Memory + CPU optimizations.

Spark Tungsten -- pushing performance to the metal

Deep-dive on Spark's Project Tungsten: the JVM overhead problem (object memory, GC, virtual calls), off-heap managed memory, …

Spark UDF Deep Dive

Slower than built-ins. Costs codegen.

Spark UI

How to read the Spark UI to understand job execution: jobs, stages, tasks, SQL plans.

spark_ui_deep

Spark Wide vs Narrow Dependencies

Shuffle boundary.

Spark Window Functions

OVER PARTITION BY. Running totals + top-N.