Hudi exists because object storage has no update operation. A data lake built on immutable files handles appends beautifully and handles 'this customer changed their address' terribly -- the usual answer being a full partition rewrite, which is why so many lakes run overnight rebuilds of tables that changed by a fraction of a percent. Hudi's proposition is that records have primary keys, that the table maintains an index from key to file, and that a write can therefore locate and update exactly the files containing the affected records. Everything distinctive about it descends from that: the index on the write path, the choice between rewriting files immediately or appending deltas and merging later, the timeline that makes those operations atomic, and the incremental read that hands a downstream job only what changed.
The timeline is the table
Every Hudi table has a .hoodie directory holding its timeline: an ordered log of actions, each stamped with a monotonically increasing instant time. An action passes through requested, inflight and completed states, and the completed marker is what makes the action visible. A reader determines the table's contents by reading the timeline first and then reading only the files that committed instants declare valid.
That single mechanism supplies several properties at once. Atomicity: a failed writer leaves an inflight instant and files that no completed instant references, so readers never see them. Snapshot isolation: a reader pins the timeline at a moment and is unaffected by concurrent writes. Time travel: reading as of an earlier instant means filtering the timeline to that point. Incremental consumption: the set of files written between two instants is exactly what changed.
The action vocabulary is worth recognising in a timeline listing. commit is a write to a copy-on-write table; deltacommit is a write to a merge-on-read table; compaction merges deltas into base files; replacecommit covers clustering and insert-overwrite, which replace file groups rather than adding to them; clean removes obsolete file versions; rollback undoes a failed instant; savepoint pins a state against cleaning so it can be restored later.
Copy-on-write and merge-on-read
The table type is the first decision and it is not easily reversed. Both types organise data into file groups -- a logical bucket of records within a partition, identified by a file ID -- and each file group holds successive file slices, one per commit that touched it.
Copy-on-write stores only columnar base files. Updating one record in a file means reading that file, merging the change, and writing a new version. Reads are as fast as reading Parquet, because that is all they are doing, and there is no merge at query time and no compaction to operate. Write amplification is the price: changing a hundred records spread across a hundred files rewrites a hundred files in full. COW suits tables with modest update rates and heavy read traffic -- dimension tables, curated marts, anything read far more often than written.
Merge-on-read adds row-oriented log files alongside the base files. An update appends a log block rather than rewriting the base file, so write latency and write amplification drop sharply. Reads pay instead: a snapshot query must merge each base file with its outstanding log blocks on the fly. A background compaction periodically folds logs into new base files to bound that cost. MOR suits streaming ingestion, high-frequency upserts, and any case where write latency matters more than the last few percent of read speed.
The practical rule: start with COW unless you are ingesting a stream or updating a large fraction of the table frequently, because MOR is genuinely more machinery to operate -- a compaction schedule, a cleaner interacting with it, and three query types to keep straight instead of one.
The write path — what an upsert actually does
A Spark write to Hudi is a pipeline, and understanding its stages explains most performance behaviour.
First, records are deduplicated within the batch on the record key, keeping the one with the highest value of the precombine field. This is why the precombine field is mandatory in practice: given two versions of the same key in one batch, it is the only thing that decides which one wins, and choosing a field that is not monotonic -- a status string, a nullable timestamp -- means the resolution is effectively arbitrary.
Second, records are tagged by the index. Each incoming key is looked up to determine whether it already exists and, if so, in which file group. Tagged records become updates routed to their existing file group; untagged ones become inserts routed by a bin-packing decision. This lookup is usually the dominant cost of the write and the first place to look when ingestion is slow.
Third, the writer sizes files. Hudi actively fights the small-file problem by directing new inserts into existing under-sized file groups until they reach a target size, rather than creating a new file per batch as a plain Parquet writer would. This is one of its quietly valuable features on streaming ingestion, where the alternative is a partition full of two-megabyte files.
Finally the writer produces new file slices and, if everything succeeded, writes the completed instant. The write operation chosen changes this pipeline: upsert is the full path above; insert skips the index lookup and therefore admits duplicates; bulk_insert skips index and file-sizing heuristics for maximum throughput on an initial load; insert_overwrite replaces whole partitions; delete writes tombstones for keys.
Indexes — the component that makes upserts viable
The index answers 'which file group holds this key', and its choice is the highest-leverage tuning decision in Hudi.
The Bloom index is the historical default. Each base file carries a Bloom filter over its keys plus the min and max key it contains; the writer prunes candidate files by key range, checks Bloom filters, and confirms survivors by reading the file. It works well when keys are roughly ordered -- so that a batch touches few files -- and degrades badly when they are random, because then every incoming key is a candidate for every file and the false-positive confirmations dominate.
The simple index joins incoming keys against the keys of the table itself. Straightforward, predictable, and expensive on large tables -- but on a batch that touches most of the table it can beat Bloom, because Bloom's pruning buys nothing there.
The bucket index assigns each key to a fixed bucket by hash, so the lookup is arithmetic rather than I/O. This is the fastest option and its constraint is the classic one for static hashing: the bucket count is fixed at table creation and resizing means rewriting. Choose it when ingestion throughput dominates and you can estimate final table size.
Newer releases add a record-level index stored in the metadata table -- an explicit key-to-location mapping, giving point-lookup performance without the bucket index's sizing commitment. Where available it is the strongest default for random-key upsert workloads. There is also an external index backed by HBase for very large tables, which trades an operational dependency for lookup speed and is chosen far less often now that the record-level index exists.
Orthogonally, an index is either global or not. A non-global index scopes the key to its partition, so the same key in two partitions is two records and a record cannot move between partitions. A global index enforces uniqueness table-wide and supports updates that change a record's partition, at meaningfully higher lookup cost. If your keys can change partition -- a status field in the partition path is the classic trap -- you need a global index or you will silently accumulate duplicates.
Reading — snapshot, read-optimized, incremental, CDC
Hudi exposes four query types and picking the wrong one is a common source of 'stale data' reports.
Snapshot is the default and returns the current state. On COW it reads the latest base file per file group. On MOR it merges base files with their outstanding log blocks at read time, so it is correct and costs more than reading Parquet.
Read-optimized exists only for MOR and reads base files only, ignoring uncompacted logs. It is as fast as Parquet and it is stale by up to one compaction interval. This is the right choice for dashboards and large analytical scans that can tolerate lag, and the wrong choice for anything that must reflect the latest write -- a distinction that must be documented, because the two queries return different answers from the same table with no error.
Incremental returns only records written between two instants. This is the feature that changes pipeline design: a downstream job reads what changed since its last run instead of scanning the table and diffing. Chaining incremental reads through several tables gives an incremental medallion architecture on batch infrastructure.
df = (spark.read.format("hudi")
.option("hoodie.datasource.query.type", "incremental")
.option("hoodie.datasource.read.begin.instanttime", last_instant)
.load(table_path))CDC goes further, returning before and after images so a consumer can distinguish inserts, updates and deletes rather than just seeing the latest version. It must be enabled on the table at write time, since the writer has to persist the extra information.
The operational catch on incremental reads: they depend on history the cleaner is allowed to delete. If a consumer is down longer than the cleaner's retention window, the instants it needs are gone and it must fall back to a full snapshot. Size cleaner retention against your worst realistic consumer outage, not against your storage bill.
Streaming ingestion with Spark
Two paths exist and they suit different teams.
Structured Streaming to a Hudi sink keeps you in Spark: read from Kafka, transform in DataFrame code, write with the Hudi format and a checkpoint location. Every micro-batch becomes an instant on the timeline. This is the right choice when ingestion involves real transformation logic.
Hudi Streamer -- the utility formerly called DeltaStreamer -- is a packaged Spark application that reads from a source, applies an optional transformer, and writes to a Hudi table, configured entirely by properties. It handles schema registry integration, checkpointing inside the commit metadata rather than in a separate Spark checkpoint, and both one-shot and continuous modes. For plain ingest-from-Kafka or ingest-from-files pipelines it removes an entire application from your codebase, and storing the checkpoint in commit metadata means the table itself knows where consumption stopped -- restoring the table restores the position.
Two configuration points dominate streaming behaviour. Micro-batch interval sets commit frequency, and very frequent commits produce a long timeline and many small file slices, so a target of minutes rather than seconds is usual for lake tables. And on MOR tables, compaction must actually run: inline compaction blocks the writer periodically, async compaction runs in a separate thread or a separate job and keeps write latency flat but needs its own resources and lock coordination.
Table services — compaction, clustering, cleaning
Hudi tables are maintained, not merely written. Three services do the work and each can run inline with the writer or asynchronously.
Compaction applies to MOR only and merges log files into new base files. Scheduling is by number of deltacommits or by accumulated log size. Falling behind is a slow-motion outage: snapshot reads get progressively more expensive as they merge ever more log blocks, and the recovery compaction is large. Alert on the number of pending compactions.
Clustering rewrites data without changing its logical content, to sort it by frequently filtered columns and to merge small files into large ones. Sorting by a query predicate tightens the min/max statistics per file and improves pruning, often dramatically. Because clustering emits a replacecommit, it conflicts with concurrent writers touching the same file groups -- which is the usual reason to run it in a maintenance window unless you have proper concurrency control.
Cleaning deletes file versions no longer needed, governed by a policy that retains either a number of commits or a number of file versions. This is the control that trades storage against time travel and incremental-read history, and it is commonly left at a default that is too small for the consumers that depend on it.
All three are callable as Spark SQL procedures, which is the most convenient way to run them on a schedule or to intervene manually:
CALL run_clustering(table => 'db.events', order => 'user_id');
CALL run_compaction(op => 'run', table => 'db.events');
CALL show_commits(table => 'db.events', limit => 10);
CALL rollback_to_instant(table => 'db.events', instant_time => '2026...');Alongside them sit savepoint and restore, which are the disaster-recovery pair: pin a known-good state so cleaning cannot remove it, and later roll the table back to it after a bad write. Savepoint before any risky bulk operation is a cheap habit.
Multi-writer concurrency
Hudi's default assumption is a single writer, and that assumption buys real simplicity. The moment a second writer -- or an async table service, which is also a writer -- touches the table, you need concurrency control.
The classic model is optimistic concurrency control: writers proceed independently and, at commit, check whether another writer modified the same file groups. Conflicts abort one side. OCC requires an external lock provider to serialise the commit check -- ZooKeeper, the Hive metastore, or a DynamoDB table are the supported options -- and configuring one is not optional. Running two writers without a lock provider produces corruption, not an error message.
OCC works well when writers touch disjoint partitions and badly when they collide, because the loser has done all its work before discovering it must be discarded. That is the standard argument for partitioning the write responsibility -- one writer per source or per partition range -- rather than relying on conflict detection to sort it out. Recent Hudi versions add a non-blocking concurrency model that lets certain overlapping writers proceed by resolving order at merge time, which is aimed squarely at the streaming-plus-backfill case that OCC handles worst.
The most common real-world instance of this is not two ingestion jobs at all: it is an async compaction or clustering job running against a table a streaming writer is actively appending to. That is multi-writer, it needs the lock provider, and skipping it is the standard way a table acquires mysterious missing records.
Catalog integration and the two MOR tables
For engines other than Spark to read the table, it must appear in a catalog. Hudi syncs to the Hive metastore or AWS Glue, either as a step inside the write job or via a standalone sync tool.
The detail that confuses everyone the first time: syncing a merge-on-read table registers two catalog entries. A table with an _ro suffix serves read-optimized queries over base files, and one with an _rt suffix serves real-time snapshot queries that merge the logs. They are the same data with different freshness and cost. Analysts pointed at the wrong one either see stale results or pay merge costs they did not intend, so name them in documentation and, ideally, expose only the one each audience should use.
Partition handling deserves the same care as in any lake table. Hive-style partition paths keep external engines happy; a partition column with high cardinality produces the small-file and metadata problems familiar from plain Parquet, and Hudi's file sizing mitigates but does not eliminate them. The metadata table -- an internal Hudi table holding the file listing and, in newer versions, column statistics and the record index -- is what removes the expensive directory listings on cloud object storage, and it should be enabled on any table of consequence.
Choosing Hudi over Delta Lake or Iceberg
All three are transactional table formats with ACID commits, time travel and schema evolution over object storage. Choosing between them on those shared features is impossible; the differences are elsewhere.
Hudi is built around record keys and an index, so record-level upserts and deletes at streaming frequency are its native case rather than a bolted-on capability. Incremental queries and the Hudi Streamer make continuous incremental pipelines straightforward. The cost is that it has more moving parts -- table type, index type, compaction schedule, cleaner policy -- and it is the least forgiving of being left at defaults.
Iceberg emphasises the open, engine-neutral table specification: hidden partitioning, partition evolution without rewriting, and broad support across engines and vendors. Where the table will be read by several engines you do not control, that portability tends to decide it.
Delta Lake is the most frictionless inside the Databricks ecosystem and has the simplest mental model -- a transaction log of add and remove actions -- with an open-source implementation that is strong on Spark and thinner elsewhere.
The pragmatic decision rule: if the defining requirement is high-frequency upserts with a primary key and incremental downstream consumption, Hudi is the format designed for that problem. If it is broad engine interoperability, Iceberg. If your platform is Databricks, Delta. And whichever you choose, budget for the maintenance jobs -- every one of these formats degrades quietly when compaction, clustering and cleaning are treated as optional.