Why architecture matters here
Bulk load architecture matters because API-based writes at billion-row scale saturate the write path. Bulk load bypasses WAL and MemStore; the result is much less network and disk activity for the same data landing in HBase.
Cost is the MR/Spark job; usually cheap compared to write path saturation.
Reliability is high — atomic move into target table.
The architecture: every piece explained
Walk the diagram top to bottom.
Source data. Raw files, analytics output — data to load.
MR / Spark job. Generates HFiles. Uses HFileOutputFormat2 (MR) or HBaseContext (Spark).
HFile output. Pre-sorted by row key; partitioned to match target regions.
HFile format. Must match target table's column families, compression, block size.
Region alignment. Job configures partitioner using current region split points; each reducer/partition produces one HFile per region.
LoadIncrementalHFiles. Tool that atomically moves generated HFiles into HBase table's storage.
No WAL. Bulk load skips write-ahead log — data is durable via HDFS replication of HFiles.
Speed. 10-100x vs API writes for the same volume.
Compaction implication. Bulk-loaded files count toward compaction trigger; may cause compaction burst after.
vs API writes. Bulk load for large batches; API for small streaming inserts.
End-to-end bulk load flow
Trace a load. 1 billion rows to load into orders table.
Job configures target table; discovers current region split points.
Job runs on Spark: reads source data; sorts by row key; partitions by region; per partition, writes an HFile with correct column families, compression, block size.
Output: HFiles per region, 500 files total (say table has 500 regions).
LoadIncrementalHFiles reads output directory; contacts RegionServers hosting each region; moves each HFile into place. Atomic per region.
Reads immediately see loaded data.
Compaction may kick in: hundreds of new HFiles per region trigger compaction. May run for hours after load.
Total time: 30 min job + minutes for load. Compare API writes: hours to days.