Bump `arrow` to `56.1` and `datafusion` to `50` by grtlr · Pull Request #11248 · rerun-io/rerun

grtlr · 2025-09-17T10:56:21Z

TODO

Double check .expect(). (copied over from DataFusion 49 upgrade #11159)
Pin datafusion to 50.0.0?
Finish https://github.com/rerun-io/dataplatform/pull/1859

Reviewing

Please expand files like package_lock.json (collapsed on GitHub) to see if everything makes sense.
For good measure, please git diff against DataFusion 49 upgrade #11159 too.

github-actions · 2025-09-17T10:56:43Z

Web viewer built successfully.

Result	Commit	Link	Manifest
✅	`fb1079b`	https://rerun.io/viewer/pr/11248	`+nightly` `+main`

View image diff on kitdiff.

^{Note: This comment is updated whenever you push a commit.}

github-actions · 2025-09-17T11:31:56Z

Latest documentation preview deployed successfully.

Result	Commit	Link
✅	`a32c3d1`	https://landing-q6iwgakpa-rerun.vercel.app/docs

^{Note: This comment is updated whenever you push a commit.}

timsaucer · 2025-09-17T11:43:32Z

The issue I foresee with this is that dataplatform relies on lance which was just updated to datafusion 49. If we merge this PR then we will need those to both update to get a consistent build.

emilk · 2025-09-23T15:32:00Z

Arrow 56.2.0 is released: https://github.com/apache/arrow-rs/releases/tag/56.2.0

Now, here, you see, it takes all the running you can do, to keep in the same place

Lewis Carroll

timsaucer · 2025-09-23T17:32:47Z

Arrow 56.2.0 is released: https://github.com/apache/arrow-rs/releases/tag/56.2.0

Now, here, you see, it takes all the running you can do, to keep in the same place
* Lewis Carroll

The bigger hold up was datafusion-python 50 which I released this morning, so now I can push lance to update

timsaucer · 2025-09-23T23:18:12Z

PR for lance update that we need: lance-format/lance#4778

timsaucer · 2025-10-07T21:41:42Z

I just did a rebase and resolved a bunch of issues. I'll take a look at CI tomorrow.

timsaucer · 2025-10-10T13:18:17Z

        │ METADATA:                                                                                                                                                                       │
        │ * entity_path: /this/that                                                                                                                                                       │
-        │ * heap_size_bytes: 944                                                                                                                                                          │
+        │ * heap_size_bytes: 896                                                                                                                                                          │


This is not surprising with the arrow update

timsaucer · 2025-10-15T10:57:28Z

Hurray! CI is in the green. I am working on the other repo so we can upgrade these side by side.

emilk · 2025-10-15T11:37:52Z

@rerun-bot full-check

github-actions · 2025-10-15T11:38:18Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/18527577409

emilk · 2025-10-15T17:28:34Z

@rerun-bot full-check

github-actions · 2025-10-15T17:29:07Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/18537333513

timsaucer · 2025-10-28T20:30:09Z

@grtlr Can you help me on this? I'm getting all kinds of image snapshot changes and I don't know how to dive in to see if there is a real problem or not. I wouldn't expect these arrow/datafusion changes to impact the viewer. Maybe we can pair on it tomorrow?

emilk · 2025-10-28T21:20:12Z

 js-sys = "0.3.77"
 jsonwebtoken = { version = "9.3", default-features = false }
-lance = { version = "0.36.0", default-features = false } # When you update this, also update the list of features enabled for `datafusion` (~50 lines up)
+lance = { version = "0.38.2", default-features = false } # When you update this, also update the list of features enabled for `datafusion` (~50 lines up)


Reminder to follow this instruction before merging!

Verified and updated comment

grtlr

LGTM—thank you!

Unfortunately I can only give you this poor man's approval because this is my own PR:

✅

This reverts commit 2a91682.

The other side is not ready yet, and right now Redap `main` is broken. * Reverted: #11248 * The other side: https://github.com/rerun-io/dataplatform/pull/1859

Revival of #11248 which was reverted in #11712. * DNM: Blocked on sibling: https://github.com/rerun-io/dataplatform/pull/1859 --- ### Related * Currently blocking #11221. * Supersedes #11159. ### TODO * [x] Double check `.expect()`. (copied over from #11159) * [x] Pin `datafusion` to `50.0.0`? * [x] Finish https://github.com/rerun-io/dataplatform/pull/1859 * [x] `@rerun-bot full-check` ### Reviewing * Please expand files like `package_lock.json` (collapsed on GitHub) to see if everything makes sense. * For good measure, please `git diff` against #11159 too. --------- Co-authored-by: Jochen Görtler <me@jgoertler.com> Co-authored-by: Tim Saucer <timsaucer@gmail.com>

grtlr requested a review from timsaucer September 17, 2025 10:56

grtlr added the exclude from changelog PRs with this won't show up in CHANGELOG.md label Sep 17, 2025

emilk reviewed Sep 17, 2025

View reviewed changes

Comment thread Cargo.lock Outdated

grtlr marked this pull request as draft September 17, 2025 11:13

grtlr force-pushed the grtlr/bump-arrow branch from bf84356 to 2b9604f Compare September 17, 2025 11:31

grtlr force-pushed the grtlr/bump-arrow branch 2 times, most recently from 03a1766 to c97e75a Compare September 17, 2025 13:58

grtlr changed the title ~~Bump arrow to 56.1~~ Bump arrow to 56.1 and datafusion to 50 Sep 17, 2025

grtlr added the blocked can't make progress right now label Sep 17, 2025

grtlr force-pushed the grtlr/bump-arrow branch from c97e75a to 67b867c Compare September 17, 2025 14:14

timsaucer force-pushed the grtlr/bump-arrow branch from 67b867c to a80d8e4 Compare October 7, 2025 21:40

timsaucer force-pushed the grtlr/bump-arrow branch from a80d8e4 to d91d4e2 Compare October 10, 2025 12:07

timsaucer added dependencies concerning crates, pip packages etc and removed blocked can't make progress right now labels Oct 10, 2025

timsaucer reviewed Oct 10, 2025

View reviewed changes

timsaucer force-pushed the grtlr/bump-arrow branch from 8acccde to f78daa6 Compare October 14, 2025 16:55

timsaucer force-pushed the grtlr/bump-arrow branch from f78daa6 to a32c3d1 Compare October 15, 2025 14:28

timsaucer force-pushed the grtlr/bump-arrow branch from a32c3d1 to 493173f Compare October 28, 2025 18:08

emilk reviewed Oct 28, 2025

View reviewed changes

timsaucer force-pushed the grtlr/bump-arrow branch from b13ca9b to 5124237 Compare October 29, 2025 11:35

grtlr and others added 13 commits October 29, 2025 10:06

Initial commit

92d9f0d

remove unused dep

849b42c

minor CI issues

6e6f235

Update snapshots

32c1a3f

minor update to cargo lock

1d346ce

reverting part of 9ba66cb after merge in from main

9873f3f

rustfmt

f0d1b57

Minor lock update

543aeda

Update unit tests after rebase

4a983b3

Update comment, verified no changes required on features

3bc04fd

fmt

ee37189

Update snapshots because memory size has changed for objects

0d03b86

set snapshots back to main

fb1079b

timsaucer force-pushed the grtlr/bump-arrow branch from 5124237 to fb1079b Compare October 29, 2025 14:41

grtlr marked this pull request as ready for review October 29, 2025 16:42

grtlr commented Oct 29, 2025

View reviewed changes

timsaucer merged commit 2a91682 into main Oct 29, 2025
69 of 70 checks passed

timsaucer deleted the grtlr/bump-arrow branch October 29, 2025 17:54

teh-cmc added a commit that referenced this pull request Oct 30, 2025

Revert "Bump arrow to 56.1 and datafusion to 50 (#11248)"

cc3ccf6

This reverts commit 2a91682.

teh-cmc mentioned this pull request Oct 30, 2025

Revert Arrow/Datafusion update #11712

Merged

teh-cmc added a commit that referenced this pull request Oct 30, 2025

Revert Arrow/Datafusion update (#11712)

7cd9b0d

The other side is not ready yet, and right now Redap `main` is broken. * Reverted: #11248 * The other side: https://github.com/rerun-io/dataplatform/pull/1859

teh-cmc restored the grtlr/bump-arrow branch October 30, 2025 08:53

teh-cmc mentioned this pull request Oct 30, 2025

Bump arrow to 56.1 and datafusion to 50 #11714

Merged

4 tasks

Uh oh!

Conversation

grtlr commented Sep 17, 2025 • edited by timsaucer Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related

TODO

Reviewing

Uh oh!

github-actions Bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timsaucer commented Sep 17, 2025

Uh oh!

emilk commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timsaucer commented Sep 23, 2025

Uh oh!

timsaucer commented Sep 23, 2025

Uh oh!

timsaucer commented Oct 7, 2025

Uh oh!

Uh oh!

timsaucer Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

timsaucer commented Oct 15, 2025

Uh oh!

emilk commented Oct 15, 2025

Uh oh!

github-actions Bot commented Oct 15, 2025

Uh oh!

emilk commented Oct 15, 2025

Uh oh!

github-actions Bot commented Oct 15, 2025

Uh oh!

timsaucer commented Oct 28, 2025

Uh oh!

emilk Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

timsaucer Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

grtlr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

grtlr commented Sep 17, 2025 •

edited by timsaucer

Loading

github-actions Bot commented Sep 17, 2025 •

edited

Loading

github-actions Bot commented Sep 17, 2025 •

edited

Loading

emilk commented Sep 23, 2025 •

edited

Loading

grtlr left a comment •

edited

Loading