Why architecture matters here

Spark + Hudi fails when procedures skipped + catalog mismatch. Architecture matters because paths + procedures compose.

Advertisement

The architecture: every piece explained

The top strip is integration. Spark SQL / DF. Hudi table format. Write path. Read path.

The middle row is features. Procedures. Timeline access. Streaming source / sink. Catalog integration.

The lower rows are ops. Metrics. vs Delta / Iceberg. Ops — retention + concurrent writers.

Spark + Hudi — write path + read path + procedures + optimizerSpark against Hudi tablesSpark SQL / DFuser codeHudi table formatCOW / MORWrite pathupsert / insert / bulkRead pathsnapshot / incremental / cdcProcedurescompaction / cleaning / clusteringTimeline accessAS OF versionStreaming source / sinkstructured streamingCatalog integrationHive / GlueMetricscommit lag + sizevs Delta / Icebergwhen Hudi fitsOps — snapshot retention + concurrent writersmaintaintravelstreamregisterwatchcomparecompareoperateoperate
Spark + Hudi integration paths.
Advertisement

End-to-end flow

End-to-end: CDC events land as Spark stream to Hudi MOR table upsert. Nightly compact procedure merges deltas. Incremental read powers downstream analytics. Catalog registers table for Impala + Presto.