Why it matters

Post-job debugging is common. Understanding History Server shapes ops ability.

Advertisement

The architecture

Event log: spark.eventLog.enabled=true. Directory: spark.eventLog.dir.

History server: reads logs, exposes UI.

History server stackEvent logwritten by driverHistory serverreads logsSpark UIfor finished jobsLog files can be large; enable rolling or compress
Historical UI setup.
Advertisement

How it works end to end

Log location: HDFS, S3, or local dir.

Access: HTTP server on chosen port.

UI shows: jobs, stages, tasks, SQL plans, storage.

Compression: gzip logs to save space.