feat: pass WriteParams to MergeInsertBuilder

[zlepper]

Fixes #6426.

Merge-insert currently creates data fragments with WriteParams::default(), so callers cannot configure blob handling or any other write behavior.

Add MergeInsertBuilder::with_write_params, which accepts a callback over the current parameters and forwards the resulting configuration through both full-schema merge fragment-write paths.

The regression test verifies that merge-insert forwards allow_external_blob_outside_bases for an external Blob V2 URI.

Do note: I'm not entirely sure about the target branch here, but copilot figured this was the correct target to get these changes in a minor release :)

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: af3b598c-bffa-4add-9536-a8ab44f7990d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[wjones127]

Sorry for the late review. I agree we should add this feature, but I'm not sure if this API is the right direction as it doesn't align with the InsertBuilder API.

[wjones127]

question(blocking): why pass a closure instead of just a pointer to some parameters? That's what we do for InsertBuilder.

https://docs.rs/lance/latest/lance/dataset/write/struct.InsertBuilder.html#method.with_params

[wjones127]

question(blocking): what's the purpose of this wrapper? What would we lose if we dropped it?

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The linked problem is valid, but builder-level write configuration must reach every supported merge-insert fragment writer. A viable revision should carry it through the indexed partial-schema fallback as well, or route that case to a writer that honors it, with a regression covering that path.

[lance-gatekeeper]

with_write_params is still ignored for indexed partial-schema inserts. Because use_index defaults to true, a scalar index on the join key makes can_use_create_plan fall back to update_fragments; its handle_new_fragments still passes Default::default() to write_fragments_internal. Callers therefore receive the same outside-base Blob error even after setting this flag. Carry the configured fragment-write options through this fallback, including its update writers, or ensure this supported case uses the configured full-row writer.

Reproducer

I extended the submitted test with a nullable extra target field, created a scalar index on id, omitted extra from the source schema, and inserted unmatched id = 2 with the same external Blob URI and allow_external_blob_outside_bases = true.

cargo test -p lance test_merge_insert_forwards_write_params_for_indexed_partial_schema_blobs -- --nocapture

Expected: the merge succeeds and the inserted blob reads inserted.

Observed on this head: the merge returns InvalidInput: External blob URI ... is outside registered external bases ... Set allow_external_blob_outside_bases=true.

[ragnorc]

Carried forward in #9888, rebased onto main. It addresses the review here: MergeInsertBuilder::with_params(&WriteParams) matches InsertBuilder, the parameters stay out of MergeInsertParams so no wrapper is needed, and every path that writes rows receives them, including the indexed partial-schema fallback. Thanks @zlepper for starting this.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Per the continuation update, #9888 carries this work forward on main, including forwarding write parameters through the indexed partial-schema fallback. It is a separate, unmerged PR; this revision remains unchanged. If #7969 is still intended to land on release/v8.0, its write configuration must reach that fallback too, with regression coverage. Otherwise, the work can continue in #9888.