Why architecture matters here

Hudi fails on wrong table type (COW for high update = slow), missing compaction (MOR read cost), and skew. Architecture matters because table type + services + record index shape performance.

Advertisement

The architecture: every piece explained

The top strip is the substrate. Table types COW vs MOR. Timeline instants + actions. Record index. Base + log files.

The middle row is services. Compaction. Clustering. Cleaning. Incremental read.

The lower rows are ops. Metastore integration. Metrics. Ops — table maintenance.

Hudi — COW + MOR + timeline + record index + clusteringthe lakehouse table format for upserts + streamingTable typesCOW vs MORTimelineinstants + actionsRecord indexfast key lookupBase + log filesparquet + avroCompactionMOR mergeClusteringz-order-likeCleaningold versionsIncremental readchanges since instantMetastore integrationHive + GlueMetricscommits + skewOps — table maintenance + concurrency + retentionmergeclustercleanreadregistermeasuremeasureoperateoperate
Hudi table types + timeline + services.
Advertisement

End-to-end flow

End-to-end: streaming job upserts to MOR table. Log files accumulate. Compaction merges to base. Clustering z-orders hot columns. Incremental read from prior instant supports CDC downstream. Cleaning removes old versions past retention.