Why it matters

Reading the Spark UI is fundamental Spark skill. Understanding accelerates every debugging session.

Advertisement

The architecture

Tabs: Jobs, Stages, Storage, Environment, Executors, SQL.

Drill down: click job → stages → task metrics.

Spark UI navigationJobs tabhigh-levelStages / tasksdrill downSQL / Storagequeries + cacheTask metrics show individual task duration, GC, shuffle read/write — invaluable for skew
UI overview.
Advertisement

How it works end to end

Stages: shows DAG. Task duration distribution reveals skew.

SQL tab: shows physical plans and metrics for DataFrame/SQL jobs.

Executors: shows per-executor stats.

Storage: shows cached RDDs/DataFrames.