Impala ORC Support, in depth: what Impala can and cannot do with ORC, how the scanner reads it, schema resolution, full ACID tables and when to convert to Parquet

The support matrix

Versions matter here more than usual. Impala's ORC support grew over several releases, and vendor distributions backport features on their own schedule. Before relying on a row in this table, check the release notes of the exact build you run, and test against a copy of your real files rather than a freshly written sample.

How Impala reads an ORC table

There are two paths, and problems usually live in the first. The metadata path: Hive, or whatever writes the files, records the table schema, partitions and, for transactional tables, valid write IDs in the Hive Metastore. Impala's catalogd loads that metadata plus file and block listings, and the statestore broadcasts it to coordinators. Impala only scans files catalogd knows about, so new data written by Hive is invisible until metadata is refreshed, either manually or through automatic event processing if your deployment enables it. Impala's architecture covers these daemons.

The scan path: the coordinator splits files into scan ranges and assigns them to executors, preferring local replicas. Each executor's ORC scanner reads the file tail first, the postscript and footer, to learn the schema, stripe offsets and column statistics. It then reads only the column streams the query needs from the stripes in its range, decodes them in batches, and materializes Impala row batches for filters, joins and aggregation. Because ORC is columnar, a query touching 3 of 40 columns reads a small fraction of the bytes.

How much more can be skipped depends on version and predicate. ORC stores minimum and maximum values per stripe and per row group, and recent Impala versions can use them for simple comparisons so whole stripes are not decoded. Partition pruning on the directory structure happens before any of this, and runtime filters from a join's build side can eliminate whole files or partitions once the scan starts.

Worked example: Hive writes, Impala reads

A team lands web click logs in a staging table every hour and wants dashboards in Impala. The table is ORC because the ingest jobs already run in Hive. First, Hive creates and loads the partitioned ORC table. ZLIB is ORC's default codec; Snappy trades a larger footprint for cheaper decompression.

-- In Hive (beeline): Impala cannot INSERT into ORC, so Hive owns the writes.
-- EXTERNAL, because Hive 3 strict managed tables would make a managed table transactional.
CREATE EXTERNAL TABLE web.clicks (
  user_id    BIGINT,
  url        STRING,
  status     INT,
  latency_ms INT,
  ts         TIMESTAMP)
PARTITIONED BY (dt STRING)
STORED AS ORC
LOCATION '/data/web/clicks'
TBLPROPERTIES ('orc.compress'='ZLIB');

INSERT INTO web.clicks PARTITION (dt='2026-10-01')
SELECT user_id, url, status, latency_ms, ts FROM staging.clicks_raw WHERE dt='2026-10-01';

Then Impala. The first time it sees a table created outside it, the table must be loaded into the catalog; after each later load, a partition-level refresh is enough and far cheaper than invalidating the whole table. Statistics matter as much for ORC as for any format, because the planner uses row counts and distinct values to pick join order and join strategy.

-- In impala-shell
INVALIDATE METADATA web.clicks;                 -- first time Impala sees the table
REFRESH web.clicks PARTITION (dt='2026-10-01');  -- after each later Hive load
COMPUTE INCREMENTAL STATS web.clicks PARTITION (dt='2026-10-01');

SELECT status, count(*), avg(latency_ms)
FROM web.clicks
WHERE dt = '2026-10-01' AND latency_ms > 2000
GROUP BY status;

PROFILE;   -- inspect the HDFS_SCAN_NODE section: bytes read, rows read, scan time

Read the profile after the first runs. For the scan node, compare bytes read against the partition's total size: a large ratio means the query is reading most columns or stripes, and a filter that should prune is not. Check that the planner used the partition predicate and that the statistics are not missing; a missing-statistics warning in the plan is the most common cause of a bad join order on a newly loaded table.

Measured on your own data, this pipeline usually behaves well for dashboards over recent partitions. If the same table is scanned constantly by many concurrent users, the conversion option below is worth timing.

Schema evolution and ORC_SCHEMA_RESOLUTION

ORC files carry their own schema, and the table schema in the Metastore can change after files are written. By default Impala matches table columns to file columns by position. That is fast and correct as long as columns are only ever appended at the end. Drop, reorder or replace columns, and old files are read with the wrong mapping: values from one column appear under another name, with no error if the types are compatible.

-- In Impala: an EXTERNAL table is dropped and re-created over the same files with a
-- new column list. 'status' is gone, 'region' is new; old files still hold status at index 2.
DROP TABLE web.clicks_ext;                       -- external, so the files stay
CREATE EXTERNAL TABLE web.clicks_ext (
  user_id BIGINT, url STRING, latency_ms INT, ts TIMESTAMP, region STRING)
PARTITIONED BY (dt STRING) STORED AS ORC LOCATION '/data/web/clicks';
ALTER TABLE web.clicks_ext RECOVER PARTITIONS;

-- Default POSITION resolution: old status values can surface as latency_ms
SET ORC_SCHEMA_RESOLUTION=NAME;                  -- resolve ORC columns by name instead
SELECT latency_ms, region FROM web.clicks_ext WHERE dt = '2026-09-01' LIMIT 5;

Setting ORC_SCHEMA_RESOLUTION to NAME makes Impala resolve by column name, which is what most people expect after a Hive schema change. It cannot rescue a rename, because a renamed column has a different name in old files. Set it as a default query option for the pool or user that reads evolving ORC tables, and test old partitions after every schema change, comparing a few rows against Hive's output.

Name resolution also depends on what the files actually contain. Files written by some older Hive versions store placeholder column names such as _col0 instead of the real ones, and names cannot match those; position is then the only option. Do not guess: inspect a file from each era of the table before switching modes.

# Print an ORC file's schema, stripes, statistics and compression with the ORC tools jar
hdfs dfs -get /data/web/clicks/dt=2026-09-01/000000_0 /tmp/sample.orc
java -jar orc-tools-<version>-uber.jar meta /tmp/sample.orc

# Look for: the type string (real column names, or placeholders such as _col0),
# the stripe count and sizes, the compression kind, and per-column min/max values

The same output answers most scanner questions. Very small stripes, or thousands of tiny files, mean high per-file overhead; missing column statistics mean no stripe can be skipped; and an unexpected codec explains CPU-heavy scans.

Reading full ACID tables

Hive's transactional tables store data differently. A full ACID table directory holds a base directory from the last major compaction, delta directories with inserted rows, and delete-delta directories that record which rows were deleted. Every row in these files is wrapped with bookkeeping columns, the operation, original transaction, bucket, row ID and current transaction, around the user's columns. An update is a delete plus an insert.

To read such a table, Impala asks the Metastore which write IDs are valid for the query, reads the base and the valid deltas, and removes rows that appear in valid delete deltas. Impala can read these tables but not write them; inserts, updates and deletes still go through Hive. ACID v1 tables from older Hive versions are not supported and must be converted first.

The operational rule is compaction. Every delta and delete-delta directory adds files to open and rows to filter, so a table updated frequently and compacted rarely gets slower to read with every hour. Watch the number of delta directories per partition, and make sure Hive's compactor is running and keeping up. If your new tables need updates and deletes from Impala itself, an Iceberg table is the better path; see Impala and Iceberg.

Performance: ORC versus Parquet in Impala

Impala's documentation is direct that ORC queries are generally slower than equivalent Parquet queries, and complex types especially perform better on Parquet. The reason is investment, not file format quality: Impala writes Parquet natively, its scanner and statistics handling for Parquet received years of tuning, and its page-index work landed there first.

That gives a clear decision rule. Keep data in ORC when it is written by Hive pipelines you do not want to change, scanned occasionally, or updated through Hive ACID. Convert to Parquet, or to Iceberg tables that Impala writes as Parquet, when a table is hot, read by many concurrent users, and written in batches that Impala or Spark can own.

-- Convert a hot, frequently scanned ORC table into Parquet that Impala writes itself
CREATE TABLE web.clicks_pq
PARTITIONED BY (dt)
STORED AS PARQUET
AS SELECT user_id, url, status, latency_ms, ts, dt FROM web.clicks
   WHERE dt >= '2026-09-01';

COMPUTE STATS web.clicks_pq;

Benchmark before and after with the same queries, warm and cold. Remote reads also benefit from the Impala data cache, whatever the format.

Failure modes and operations

• Stale results after a Hive load: catalogd has not seen the new files. Refresh the partition, or confirm that automatic metadata event processing is enabled and keeping up.
• Wrong values in old partitions after a schema change: position-based resolution. Switch to NAME resolution and re-verify.
• Errors on a transactional table: check whether it is ACID v1, or whether the Metastore connection Impala needs for write IDs is missing.
• Slow scans on an updated table: too many delta directories. Run or tune compaction in Hive.
• Bad join orders on new data: missing statistics. Run incremental stats after each load and alert on tables without them.
• Small-file explosions from frequent loads: each file means a footer read and a scan range. Compact or rewrite into fewer, larger files.
• Codec or type surprises: test a sample of every distinct writer's files, including old ones, before declaring a table supported.

What to do next

1. Confirm your Impala version and whether the ORC scanner is enabled, and note the exact release notes for ORC.
2. List ORC tables by writer, update pattern and query volume; mark hot tables as conversion candidates.
3. Add REFRESH and COMPUTE INCREMENTAL STATS to the end of every Hive load job that feeds Impala.
4. Set ORC_SCHEMA_RESOLUTION=NAME for users reading tables whose schema has ever changed, and test old partitions.
5. For full ACID tables, monitor delta directory counts and compaction lag.
6. Benchmark your top five queries on ORC against a Parquet copy, and convert only where the gain is real.