Why it matters
File format choice affects everything downstream. Compression ratio saves storage cost. Columnar layout speeds up analytical scans. Schema evolution support determines how easily you can change tables. Splittability determines parallelism.
The architecture
Text: human-readable, no schema, huge, slow. Never use for anything performance-sensitive.
SequenceFile: binary key-value pairs, splittable, better than text but row-based only.
Avro: row-based, schema evolution built in, cross-language. Good for landing zone or Kafka integration.
ORC: columnar, high compression, splittable, indexed. Optimized for Hive and Spark.
Parquet: columnar, high compression, splittable, cross-engine (Hive/Spark/Impala/Presto). Industry standard.
How it works end to end
Columnar formats (ORC, Parquet) store column data contiguously. Analytical queries reading a few columns skip other columns entirely on disk. This is 10-100x faster than row-based scans of the same data.
ORC has better compression and stat pushdown for Hive-specific workloads. Parquet is more portable across engines.
Schema evolution: Avro handles field addition and rename with schema references. Parquet supports field addition. ORC supports field addition and type widening.