CometScanRule reads fs.comet.libhdfs.schemes from SQLConf, but native reads it from the Hadoop conf #6837

Describe the bug

CometScanRule and native Parquet scans read fs.comet.libhdfs.schemes from different places, so the rule can decide a scheme is or is not served by libhdfs differently from native.

• The rule reads COMET_LIBHDFS_SCHEMES, the SQLConf entry spark.hadoop.fs.comet.libhdfs.schemes, defaulting to hdfs. It uses that list to classify the scan's root paths, for the "Unsupported filesystem schemes" gate, and to skip the object_store path probe for libhdfs paths.
• Native gets the list from the Hadoop conf: NativeConfig forwards hadoopConf.get("fs.comet.libhdfs.schemes") to the scan's object store options, and is_hdfs_scheme (parquet_support.rs) routes a scheme through libhdfs only when it is in that option, or is hdfs when the option is absent.
• fs.comet.s3Compliant.schemes is already read from the Hadoop conf in the same method, so core-site.xml is honored for it but not for the libhdfs list.

The two disagree in both directions:

1. Set only in core-site.xml (or another Hadoop-only source), e.g. fs.comet.libhdfs.schemes=hdfs,webhdfs. Native would read webhdfs:// through libhdfs, but the rule does not see the list, so the scan falls back to Spark with "Unsupported filesystem schemes: webhdfs".
2. Set at runtime with SET spark.hadoop.fs.comet.libhdfs.schemes=hdfs,webhdfs. Spark's SessionState.newHadoopConf copies SQLConf keys as they are, without dropping the spark.hadoop. prefix, so the Hadoop conf has no fs.comet.libhdfs.schemes. The rule admits the scan, native does not route webhdfs through libhdfs, and the scan fails at execution when object_store rejects the URL ("Unable to recognise URL").

Steps to reproduce

1. Put fs.comet.libhdfs.schemes=hdfs,webhdfs in core-site.xml, enable the native Parquet scan, and read Parquet files from a webhdfs:// path. The scan falls back with "Unsupported filesystem schemes: webhdfs".
2. Without it in core-site.xml, run SET spark.hadoop.fs.comet.libhdfs.schemes=hdfs,webhdfs in the session and read the same path. Comet claims the scan, and the query fails when native opens the file.

Expected behavior

The rule uses the same libhdfs list native will use: read it from the scan's Hadoop conf (newHadoopConfWithOptions(relation.options)), as fs.comet.s3Compliant.schemes already is. Then case 1 runs natively and case 2 falls back instead of failing.

Additional context

Found in #6753 (comment), which moves the multi-store fallback check in the same method to the Hadoop conf list. The root classification and the scheme gate stay on SQLConf there, since changing them changes which scans Comet claims.

A list set in the Spark conf at startup (spark.hadoop.fs.comet.libhdfs.schemes) reaches both places, so the common setup is not affected.

Willingness to contribute

I can contribute a fix for this bug independently