Hadoop Ecosystem in 2026, in depth: a component status map, Java baselines, catalogs and how to choose a stack

Three layers, swapped independently

The useful mental model is a stack of interfaces. At the bottom, storage: HDFS on local disks, Apache Ozone or a cloud object store. Above it, the Hadoop filesystem API in hadoop-common, which lets an engine open hdfs://, s3a:// or abfs:// paths with the same code. Then resource management (YARN or Kubernetes), table formats and a catalog, and the engines on top. Security, ingest and orchestration cut across the stack.

Because the layers meet at stable interfaces, a migration rarely replaces "Hadoop" in one step. A team may move data to object storage while keeping the Hive Metastore, then move Spark from YARN to Kubernetes, then convert tables to Iceberg. Each step has its own risk and rollback, and the status of each component tells you which steps are urgent.

Status map: active, dormant and retired

For projects not in the table, including HBase, Tez, Ranger, Knox, Atlas, Kudu, ZooKeeper and Impala, check the project's own release page and mailing list activity before committing to a multi-year plan. A project with no release in two years is a risk even if it has not formally retired. The Apache Attic is the definitive list of retired projects; a retired project's code stays downloadable, but security reports are no longer handled.

Java baselines set the upgrade order

The biggest change of the past year is not a feature but a runtime. Per the Hadoop 3.5.0 release notes, 3.5 is the first line with full Java 17 support, Java 17 is required on servers and clients support 17 and 21. Spark 4 requires Java 17 as well, and Hive 4.1 added JDK 17 support. The 3.4 line remains the choice for clusters that cannot move off older JVMs yet.

This fixes the order of operations. Upgrade the JVM and validate every component on Java 17 first, including custom UDFs, serde jars and the agents your monitoring injects. Then upgrade Hadoop to 3.5, then the engines. Doing it in another order produces errors that look like Hadoop bugs but are really module-system access errors from Java 17's strong encapsulation, typically InaccessibleObjectException from a library reflecting into JDK internals. The usual workaround, --add-opens flags, should be a documented and temporary list rather than an ever-growing one.

Storage: HDFS, Ozone or an object store

HDFS is still the fastest option for data-local batch on owned hardware, and its operational model is well understood. Its limit is the NameNode, which holds every file and block in memory, so small files cost heap regardless of their size. Ozone separates the namespace from block management and offers both an object store API and a Hadoop-compatible filesystem, which is why it is the main on-premises answer to file-count limits; see Apache Ozone in depth for its architecture.

Cloud object stores remove the cluster from storage altogether. The cost is the semantics: listing is slower, renames are not atomic, and job commits need committers designed for object storage. Hadoop's S3A connector, which moved to the AWS SDK v2 in the 3.4 line, carries most of that work, which is why even Hadoop-free stacks keep a hadoop-aws jar on the classpath. Whichever store you choose, the engines see a filesystem URI, which is what makes storage the easiest layer to change.

Tables and catalogs: where lock-in moved

Ten years ago the hard part to move was HDFS. Today it is the catalog: the service that maps table names to files, schemas and partitions. The Hive Metastore is the de facto shared catalog because Spark, Trino, Impala, Flink and Hive all speak to it. Hive 4.1 ships the metastore as a standalone component and adds a REST catalog server, which matters because open table formats, Iceberg in particular, increasingly standardise on a REST catalog protocol.

Table formats such as Iceberg move transactional metadata into files next to the data, which gives atomic commits, schema evolution and time travel on any store. A practical path keeps the existing metastore while converting tables one at a time:

# Spark 4.x on Kubernetes or YARN: Iceberg tables registered in an existing Hive Metastore,
# data files on S3 through Hadoop's S3A connector. One catalog, two storage generations.
spark-submit \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.warehouse=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.warehouse.type=hive \
  --conf spark.sql.catalog.warehouse.uri=thrift://metastore.internal:9083 \
  --conf spark.sql.catalog.warehouse.warehouse=s3a://lake-warehouse/ \
  migrate_orders.py

# migrate_orders.py
#   spark.sql("CREATE TABLE warehouse.sales.orders USING iceberg PARTITIONED BY (days(order_ts)) "
#             "AS SELECT * FROM spark_catalog.legacy.orders")      # legacy Hive table on HDFS

The legacy table stays readable by existing jobs until its consumers move, and the new table is usable from Trino and Spark immediately. The Iceberg and Trino stack covers the design of the target side.

Compute and scheduling: YARN or Kubernetes

YARN remains a good scheduler for a fixed fleet running mostly JVM batch jobs. It understands data locality, queues and preemption, and its Capacity and Fair Schedulers are mature; HDFS and YARN together explains the interaction. Kubernetes wins where the organisation already runs it, where workloads include non-JVM services and model serving, and where storage is already remote so locality no longer matters.

The trade-off is operational rather than technical. Moving Spark to Kubernetes means replacing YARN queues with namespaces, quotas and a batch scheduler, rebuilding the external shuffle story, and re-solving Kerberos delegation for jobs that still read secured HDFS. Teams that move compute before storage often keep a YARN cluster purely for data access, which costs more than either end state.

Replacing the retired pieces

Retired components are the urgent items because they accumulate unpatched vulnerabilities. Oozie workflows map naturally onto an orchestrator such as Airflow: each action becomes a task, coordinators become schedules, and the XML disappears. Flume agents are usually replaced by Kafka with Kafka Connect, or by NiFi where flows need routing and transformation. Sqoop imports become Spark JDBC reads, which keep the parallel split-by-key model:

# Replacing a Sqoop import with Spark's JDBC source: parallel reads split on a numeric key.
orders = (spark.read.format("jdbc")
          .option("url", "jdbc:postgresql://db.internal:5432/shop")
          .option("dbtable", "public.orders")
          .option("user", user).option("password", password)
          .option("partitionColumn", "order_id")       # like Sqoop's --split-by
          .option("lowerBound", 1).option("upperBound", 50_000_000)
          .option("numPartitions", 16)                 # like --num-mappers; mind the database's limits
          .option("fetchsize", 10_000)
          .load())
orders.writeTo("warehouse.sales.orders_raw").append()

Sqoop's incremental modes become a watermark query on a timestamp or key column, persisted between runs, and its Hive import becomes a table write. Validate row counts and checksums on the first runs, and throttle partitions so the source database is not overwhelmed, which a Sqoop job's mapper count used to cap implicitly.

Worked example: auditing an older cluster

Take a cluster built in 2018: HDFS and YARN on Hadoop 3.1, MapReduce and Hive jobs, Oozie for scheduling, Sqoop for nightly imports from Postgres, Flume shipping web logs and an HBase table behind an internal service. The audit starts by listing what actually runs, then classifying each component.

# eco_audit.py: classify each running component by 2026 status and suggest an action.
STATUS = {
    # component: (status, action) -- statuses checked against Apache release and Attic pages, Oct 2026
    "hdfs":      ("active (3.4.3 Feb 2026, 3.5.0 Apr 2026)", "keep; plan Java 17 for 3.5"),
    "yarn":      ("active, ships with Hadoop",               "keep while the cluster stays"),
    "mapreduce": ("maintained, legacy engine",               "migrate hot jobs to Spark"),
    "hive":      ("active (4.1.0 Jul 2025)",                 "upgrade metastore first"),
    "spark":     ("active (4.1.0 Dec 2025)",                 "plan Java 17 and Scala 2.13"),
    "ozone":     ("active (2.0.0 Apr 2025)",                 "evaluate if HDFS file counts hurt"),
    "ambari":    ("restored from Attic, 3.0.0 Mar 2025",     "check stack support before relying on it"),
    "oozie":     ("retired Feb 2025, in Attic",              "move workflows to Airflow"),
    "sqoop":     ("retired 2021, in Attic",                  "replace with Spark JDBC"),
    "flume":     ("dormant since Oct 2024",                  "replace with Kafka Connect or NiFi"),
}

def audit(running):
    unknown = []
    for name in sorted(running):
        status, action = STATUS.get(name, (None, None))
        if status is None:
            unknown.append(name)
            continue
        print(f"{name:10} {status:42} -> {action}")
    for name in unknown:
        print(f"{name:10} {'UNKNOWN: check the project page':42} -> verify before planning")

audit({"hdfs", "yarn", "mapreduce", "hive", "oozie", "sqoop", "flume", "hbase"})

The output sorts the work. Oozie, Sqoop and Flume are retired or dormant, so they come first regardless of anything else. HBase is reported as unknown: someone must check its current release line and whether the service it backs can tolerate an upgrade. Hadoop itself goes to the latest 3.4.x patch release now, which keeps the current JVM, with a separate project to qualify Java 17 and reach 3.5. Hive's metastore is upgraded before its query engine because every other engine depends on it. Only after that does the team decide whether storage stays on HDFS, moves to Ozone, or moves to a cloud store; migrating Hadoop to the cloud covers that path.

Failure modes in mixed-version stacks

Distribution, managed service or assemble it yourself

There are three ways to consume the ecosystem. A commercial distribution or platform bundles tested versions and security patches, which is what you pay for. A managed cloud service such as EMR, Dataproc or HDInsight removes cluster operations but ties you to its version cadence. Assembling releases yourself, with Apache Bigtop's packaging as a starting point, gives full control and makes your team responsible for compatibility testing across every pair of components.

Choose by how many components you run and how often they change. With HDFS, YARN and Spark only, assembling is feasible. With ten services, Kerberos and Ranger, someone has to own the compatibility matrix, and that work is the real cost of the platform.

What to do next

1. List every component running in each cluster, with its version and the JVM it runs on.
2. Check each against its Apache project page and the Attic; mark retired and dormant ones as urgent.
3. Plan replacements for Oozie, Sqoop and Flume before any feature work on them.
4. Qualify Java 17 for every component and custom jar, then schedule the move to Hadoop 3.5.
5. Upgrade the Hive Metastore schema and service before engines, and test every engine against it.
6. Pick one table and pilot Iceberg through your existing metastore.
7. Add a dependency-convergence check for Guava, Protobuf and the AWS SDK to your builds.
8. Write down whether storage stays on HDFS, moves to Ozone or moves to object storage, and why.