Conversation
…ative backend Adds `hdfs://` to the native Iceberg scan and write paths, backed by iceberg-rust's pure-Rust `hdfs-native` OpenDAL backend (apache/iceberg-rust#3111). This is a second, independent HDFS client: the plain-Parquet native scan reaches HDFS through libhdfs/JNI (`fs.comet.libhdfs.schemes`), while an Iceberg table is opened over pure-Rust RPC. The two share only the `$HADOOP_CONF_DIR` XML and authenticate separately. The scheme arm alone is not sufficient on an HA cluster. opendal's `HdfsNativeBuilder` never dials the authority written in the path: it builds its client against a synthetic authority and synthesizes `dfs.ha.namenodes.<synthetic>` / `dfs.namenode.rpc-address.<synthetic>.nnN` from the comma-separated `hdfs.name-node` value (`init_hdfs_config`). iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a single-NameNode cluster. An HA table location reads `hdfs://<nameservice>/...`, and a nameservice is not a routable host, so without a NameNode list every HA table would fail to connect at execution time -- after the planner had already committed to the native scan. `hadoopToIcebergHdfsProperties` therefore resolves the endpoints from the session Hadoop configuration (`dfs.ha.namenodes.<ns>` plus each `dfs.namenode.rpc-address.<ns>.<nn>`), joined in declaration order, so a standard HDFS client configuration needs no extra settings. An explicit catalog `hdfs.name-node` still wins. A partially resolved list yields nothing rather than a short failover list, which would silently turn a failover into an outage. Because one `hdfs.name-node` overrides the authority of every path the FileIO opens, a scan whose data/delete files span more than one HDFS authority now falls back: the second nameservice would otherwise be read from the first one's NameNode at the same relative path, returning wrong data rather than an error. Changes: - `storage_factory_for`: `hdfs` arm; `hdfs.`/`hadoop.` added to `STORAGE_PROPERTY_PREFIXES` so the NameNode list and client overrides reach FileIO - `CometScanRule.icebergReadableSchemes` and `CometIcebergNativeWrite.SupportedStorageSchemes`: admit `hdfs` - `IcebergTaskValidationResult.dataFileHdfsAuthorities` + multi-authority fallback - `native/Cargo.toml`: enable `opendal-hdfs-native` Testing: - `CometIcebergHdfsSuite`: end-to-end reads against an in-process MiniDFSCluster (plain read, pushdown filter, partitioned table across multiple data files), asserting the data location is genuinely `hdfs://` - 5 unit tests pinning the HA NameNode translation, 2 pinning the native scheme and property forwarding - Full Rust workspace suite (1442 tests) and the three affected JVM suites pass Note: writes are gated and property-forwarded but have no functional test; only reads are covered end to end. The HA translation is unit-tested only, as MiniDFSCluster is single-NameNode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The opendal synthetic-authority mechanic -- that `hdfs.name-node` overrides the authority of every path the FileIO opens -- was spelled out at seven sites. Keep it once, in `hadoopToIcebergHdfsProperties`'s scaladoc, and leave short pointers at the rest. Also drops a paragraph that narrated the eight lines of code beneath it, a `@param` restating its own signature, and a sentence duplicated four lines apart inside `hadoopToIcebergHdfsProperties`. Retained the reasoning that cannot be recovered from the code: why opendal ignores the path authority, why the NameNode list is all-or-nothing, why `getRawAuthority` rather than `getHost`, and that hdfs-native and libhdfs/JNI are two separate clients. Comments only; no behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`hadoop-client-minicluster` is pinned at 3.3.4 while Spark supplies `hadoop-client-api`/`runtime` 3.4.2 on the Spark 4.x profiles, so `HttpServer2` resolves a shaded Jetty class the older jar does not carry and the NameNode web server fails to start. Record that in `beforeAll` and cancel the tests instead of aborting the suite, so CI stays green on profiles where the fixture cannot run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
enhancement
… supplies `hadoop.version` was a single global 3.3.4, but only Spark 3.4/3.5 ship hadoop 3.3.x. On the Spark 4.x profiles `HttpServer2` comes from Spark's newer hadoop-client-runtime and resolves shaded Jetty classes the 3.3.4 minicluster does not carry, so MiniDFSCluster's NameNode web server fails to start and any suite using `WithHdfsCluster` cannot run. Set `hadoop.version` per Spark profile, as `parquet.version` already is: 3.4 -> 3.3.4, 3.5 -> 3.3.4, 4.0 -> 3.4.1, 4.1 -> 3.4.2, 4.2 -> 3.5.0. The 3.x profiles are unchanged; a global bump would only have moved the skew onto them. `CometIcebergHdfsSuite` now runs on the default profile instead of cancelling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Sounds good, thanks @mixermt |
Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Found that |
…scans The scan input bridge only ran when the native block's root was the scan itself. `hasScanInput` recognised `CometNativeScanExec` alone, so an Iceberg scan fused below a filter, project or aggregate, or feeding the native shuffle writer, never reported bytesRead/recordsRead and the Spark UI showed "Input: 0 B / 0 records" for those stages. The listener was also registered after `CometExecIterator`, so it ran before the final native metric flush whenever the iterator was not drained. Register the bridge in `CometExecRDD.compute` ahead of the iterator, drop the per-exec `compute` overrides and the `hasScanInput` gate (the bridge is a no-op without `bytes_scanned` leaves), and increment rather than set the task input metrics, claiming each accumulator once per task through the registry the spill reporter already uses so overlapping or repeated registrations count once while other input sources still add up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CometNativeShuffleWriter registered reportScanInputMetrics after CometNativeShuffleInputRDD had resolved its inputs, so for a nested native block that a failed or undrained writer plan left unflushed, the writer's listener claimed the nested scan leaves before their iterator published final values and the nested block's own listener then added nothing. Register scan input reporting next to spill reporting in CometNativeShuffleInputRDD.compute, before the input producers and the writer's iterator, and drop the writer's registration. Rename the RDD's metric node to taskMetricNode since it now feeds both task metrics, and state that a coalesced task's repeated registrations are deduplicated rather than summed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…L UI The SQL UI aggregates accumulator updates from listener events and never reads driver-side values. Iceberg posts its planning metrics (manifests and files scanned or skipped, planning duration, file sizes) for the original BatchScanExec's accumulators, whose ids are not in the Comet plan graph, and CometIcebergNativeScanExec never posted its own, so every planning metric showed 0. Under DPP the driver-side values were 0 as well, because planning runs on a copy of the original plan while the lazy metrics read the original's accumulators. Post the planning metrics as a driver metric update once per SQL execution from the first execution-time read of commonData, read them from the plan copy that actually planned, register them under Iceberg's descriptions so the labels match BatchScan, and keep them out of the executor metric tree so tasks do not report zeros into the per-task statistics. Document that output rows and recordsRead count rows after iceberg-rust's row filter, and that row deletes applied has no native counterpart. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Reconciles the Iceberg planning metrics work with apache#6027, which posts them through the new CometLeafExec.sendDriverMetrics hook: keep the hook and Iceberg's metric names, and layer on the per-execution post guard, the DPP-correct plannedOriginalPlan, and the exclusion of planning metrics from the executor metric tree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 22, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
NOT FOR MERGE — pending apache/iceberg-rust#3111.
This PR pins
iceberg/iceberg-storage-opendalto a personal fork(
mixermt/iceberg-rust@9d7d2d89) because the HDFS backend it depends on has not mergedupstream yet. That pin must not be merged into Comet. It is opened now for early review of
the Comet-side design; once apache/iceberg-rust#3111 lands and Comet's pinned rev includes
it, the dependency change drops out and the diff becomes feature-only.
Upstream status: apache/iceberg-rust#3111 is open with changes requested.
Which issue does this PR close?
Part of #5894.
Rationale for this change
Comet's native Iceberg scan cannot read a table whose data lives on HDFS:
hdfs://is absent from the scheme allowlist that mirrorsstorage_factory_for, so every Iceberg table on HDFS falls back to the JVM reader. That is a gap relative to the plain-Parquet native scan, which does read HDFS through libhdfs/JNI (fs.comet.libhdfs.schemes). On-premise Iceberg deployments are commonly HDFS-backed and get no native Iceberg acceleration today.iceberg-rust is gaining a pure-Rust HDFS backend (
hdfs-native, no JNI or libhdfs) in apache/iceberg-rust#3111. This PR is the Comet side of that.Worth stating plainly, because it surprises: this introduces a second, independent HDFS client into the same process. The plain-Parquet path reaches HDFS through libhdfs/JNI; an Iceberg table is opened over pure-Rust RPC. Both link into
libcomet, they read the same$HADOOP_CONF_DIRXML, but they hold separate connections and separate Kerberos state — the Rust client does not reuse the JVM's Kerberos subject.What changes are included in this PR?
Native (
iceberg_common.rs):storage_factory_for: routehdfstoOpenDalStorageFactory::HdfsNative.STORAGE_PROPERTY_PREFIXES: forwardhdfs.andhadoop.to the nativeFileIO. Without these the NameNode list never reaches iceberg-rust.JVM gates:
CometScanRule.icebergReadableSchemesandCometIcebergNativeWrite.SupportedStorageSchemesadmithdfs, keeping both in lockstep withstorage_factory_foras their docstrings require. Onlyhdfsitself — a libhdfs alias scheme has no iceberg-rust arm.NameNode resolution (
CometIcebergNativeScan.hadoopToIcebergHdfsProperties) — the part that is not obvious:HdfsNativeBuildernever dials the authority written in the path. It builds one client against a synthetic authority and synthesizes the HA config from the comma-separatedhdfs.name-nodevalue (init_hdfs_configinopendal-service-hdfs-native). iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a realhost:port.hdfs://<nameservice>/..., and a nameservice is not a routable host. So without a resolved NameNode list, every HA table would fail to connect at execution time — after the planner had already committed to the native scan.dfs.ha.namenodes.<ns>plus eachdfs.namenode.rpc-address.<ns>.<nn>), joined in declaration order, so a standard HDFS client configuration needs no new settings. An explicit cataloghdfs.name-nodestill wins. A partially resolved list yields nothing rather than a short failover list, which would silently turn a failover into an outage.Test-fixture fix (
pom.xml), needed for the end-to-end suite to run at all:hadoop.versionwas a single global 3.3.4, but only Spark 3.4/3.5 ship hadoop 3.3.x. On the Spark 4.x profilesHttpServer2comes from Spark's newer hadoop-client-runtime and resolves shaded Jetty classes the 3.3.4 minicluster does not carry, so MiniDFSCluster's NameNode web server fails to start and every suite usingWithHdfsClusteris unrunnable. This is pre-existing and not specific to HDFS Iceberg support.hadoop.versionper Spark profile, asparquet.versionalready is: 3.4 -> 3.3.4, 3.5 -> 3.3.4, 4.0 -> 3.4.1, 4.1 -> 3.4.2, 4.2 -> 3.5.0. The 3.x profiles are unchanged; a global bump would only have moved the skew onto them. Verified with reactor builds onspark-4.0andspark-3.5.Single-NameNode-per-scan gate:
hdfs.name-nodeoverrides the authority of every path the FileIO opens, so a scan whose data/delete files span more than one HDFS authority now falls back. Otherwise the second nameservice would be read from the first one's NameNode at the same relative path — wrong data rather than an error. This mirrors the existing multi-bucket S3 check.How are these changes tested?
CometIcebergHdfsSuite: end-to-end reads against an in-processMiniDFSCluster, running on the default (Spark 4.1) profile — plain read, pushed-down filter, and a partitioned table spanning multiple data files. Each asserts a singleCometIcebergNativeScanExecin the plan and result parity with Spark, and the first asserts the resolved data location really ishdfs://, so the suite cannot silently degrade into duplicate local-filesystem coverage.CometIcebergNativeScanSuite: five cases pinning the HA translation — declaration order, non-HA authority yielding nothing, all-or-nothing on a partial list, no doublehdfs://prefix, and non-hdfs/authority-less inputs ignored.CometScanSchemeFallbackSuite:hdfsadmitted by the Iceberg gate; hostlesshdfs:///declined.iceberg_common: thehdfsarm resolves for read and write, and thehdfs./hadoop.prefixes survive the property narrowing.Known gaps, stated rather than hidden
hdfsbe dropped fromSupportedStorageSchemesuntil a write test exists.MiniDFSClusteris single-NameNode, sohdfs.name-nodehas never resolved a live nameservice; the translation logic is pinned by tests, real failover is not.deps:PR.AI Disclosure
Developed with AI assistance (Claude Code): drafting the implementation, tests, and this description. I reviewed the changes and ran all verification locally.