Skip to content

[SPARK-59903] Upgrade Iceberg example to use Iceberg 1.12.0 - #915

Closed
dongjoon-hyun wants to merge 1 commit into
apache:mainfrom
dongjoon-hyun:SPARK-59903
Closed

dongjoon-hyun wants to merge 1 commit into
apache:mainfrom
dongjoon-hyun:SPARK-59903

Conversation

@dongjoon-hyun

@dongjoon-hyun dongjoon-hyun commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR aims to upgrade Iceberg to 1.12.0 in the spark-connect-server-iceberg.yaml example.

Why are the changes needed?

To use the latest Iceberg release.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs and manually tests.

$ kubectl apply -f examples/spark-connect-server-iceberg.yaml
sparkapplication.spark.apache.org/spark-connect-server-iceberg created

$ kubectl get sparkapp
NAME                           AGE   CURRENT STATE
spark-connect-server-iceberg   6s    DriverReady

$ kubectl get pod
NAME                                         READY   STATUS    RESTARTS   AGE
localstack                                   1/1     Running   0          50s
spark-connect-server-iceberg-0-driver        1/1     Running   0          15s
spark-kubernetes-operator-5c4476f5d9-8fflq   1/1     Running   0          5m15s

$ kubectl logs spark-connect-server-iceberg-0-driver | grep iceberg-spark-runtime | head -n2
org.apache.iceberg#iceberg-spark-runtime-4.1_2.13 added as a dependency
	found org.apache.iceberg#iceberg-spark-runtime-4.1_2.13;1.12.0 in central

$ kubectl port-forward spark-connect-server-iceberg-0-driver 15002
Forwarding from 127.0.0.1:15002 -> 15002
Forwarding from [::1]:15002 -> 15002

$ bin/spark-connect-shell --remote sc://localhost:15002
...
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.1.3
      /_/

Type in expressions to have them evaluated.
Spark connect server version 4.1.3.
Spark session available as 'spark'.
scala>
sql("""CREATE TABLE taxis(vendor_id bigint, trip_id bigint, trip_distance float, fare_amount double, store_and_fwd_flag string) PARTITIONED BY (vendor_id);""").show()

scala>
sql("""INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');""").show()

scala> sql("SELECT * FROM taxis").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1        |1000374|8.4          |42.13      |Y                 |
|1        |1000371|1.8          |15.32      |N                 |
|2        |1000373|0.9          |9.01       |N                 |
|2        |1000372|2.5          |22.15      |N                 |
+---------+-------+-------------+-----------+------------------+

scala> sql("SELECT * FROM taxis.history").show(false)
+-----------------------+-------------------+---------+-------------------+
|made_current_at        |snapshot_id        |parent_id|is_current_ancestor|
+-----------------------+-------------------+---------+-------------------+
|2026-09-30 21:35:59.331|1296194957924373327|NULL     |true               |
+-----------------------+-------------------+---------+-------------------+

scala> sql("SELECT * FROM taxis VERSION AS OF 1296194957924373327").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1        |1000374|8.4          |42.13      |Y                 |
|1        |1000371|1.8          |15.32      |N                 |
|2        |1000373|0.9          |9.01       |N                 |
|2        |1000372|2.5          |22.15      |N                 |
+---------+-------+-------------+-----------+------------------+

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

@dongjoon-hyun

Copy link
Copy Markdown
Member Author

Thank you, @peter-toth !

@dongjoon-hyun dongjoon-hyun added this to the 1.1.0 milestone Sep 30, 2026
@dongjoon-hyun

Copy link
Copy Markdown
Member Author

Merged to main

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-59903 branch September 30, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants