Why architecture matters here

Spark on K8s fails on wrong resource sizing, missing shuffle service, and spot handling. Architecture matters because K8s + Spark abstractions compose.

Advertisement

The architecture: every piece explained

The top strip is lifecycle. Spark submit / operator. Driver pod. Executor pods. K8s scheduler.

The middle row is efficiency. Dynamic allocation. External shuffle. Volume claim. Node pools.

The lower rows are ops. Metrics. IAM / workload identity. Ops — resource + queue + spot.

Spark on Kubernetes — operator + driver/executor pods + dynamic allocation + shufflecloud-native SparkSpark submit / operatorstart jobDriver podcoordinatorExecutor podscomputeK8s schedulerplace podsDynamic allocationscale executorsExternal shufflesurvive executor lossVolume claimspill / cacheNode poolsspot / GPU / on-demandMetricsprometheus + Spark UIIAM / workload identityS3 / GCS accessOps — image + resource + queue + spot handlingscaleshuffleclaimselectwatchaccessaccessoperateoperate
Spark on Kubernetes with operator + dynamic allocation.
Advertisement

End-to-end flow

End-to-end: operator applies SparkApplication CR. Driver pod launches. Executor pods scale via dynamic allocation. External shuffle preserves data. Spot node pool used for cheap executors; on-demand for driver. Metrics stream to Prometheus.