Skip to content

fix: disable ThinLTO in test-fast - #1407

Merged
inureyes merged 2 commits into
mainfrom
fix/issue-1406-disable-test-thinlto
Aug 25, 2026
Merged

inureyes merged 2 commits into
mainfrom
fix/issue-1406-disable-test-thinlto

Conversation

@inureyes

@inureyes inureyes commented Aug 25, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Disable LTO in [profile.test-fast] so macOS workspace integration-test binaries no longer enter the ThinLTO link path that produced missing internal Rust symbols.
  • Preserve the optimized test behavior that matters for MLX numerics: opt-level = 3, codegen-units = 16, incremental compilation, unstripped local test binaries, and inherited unwind behavior remain unchanged.
  • Leave [profile.release] unchanged with fat LTO and one codegen unit, and update Cargo, Makefile, contributor, installation, and nightly guidance to document the test/release trade-off accurately.
  • Add bilingual technical reports covering the reproduction, decision, validation evidence, and residual release-codegen gap.

Root cause

The full workspace no-run graph repeatedly failed while linking large integration targets such as molmo_parity and qwen3_omni_moe_parity, with unresolved LLVM-suffixed Rust drop-glue symbols. The failure reproduced with CARGO_BUILD_JOBS=1, while a targeted integration-test build could pass under a smaller unit graph, isolating the failure to the explicit ThinLTO workspace link shape rather than the test source or concurrent Cargo link jobs.

Validation

  • TOML profile assertions: [profile.test-fast] has lto = false, opt-level = 3, and codegen-units = 16; [profile.release] remains lto = true, codegen-units = 1, strip = true, opt-level = 3, and panic = "unwind".
  • cargo test --profile test-fast --features metal,accelerate --test molmo_parity --no-run
  • cargo test --profile test-fast --features metal,accelerate --test qwen3_omni_moe_parity --no-run
  • cargo test --workspace --profile test-fast --features metal,accelerate --no-run — passed; all workspace test targets linked in 9m19s on the cold corrected profile.
  • cargo test --workspace --profile test-fast --features metal,accelerate --no-fail-fast -- --test-threads=1 — passed with zero failures across all workspace test and doctest binaries.
  • cargo clippy --workspace --all-targets --features metal,accelerate -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check origin/main...HEAD
  • GitHub CI: formatting, clippy, dependency policy, crate versions, kernel dtype keys, cross-repository references, OpenXLA feature compilation, and CLA checks passed before the report-only commit; the refreshed checks are expected to exercise the same implementation tree.

Review

  • Independent correctness review: no findings.
  • Security/performance review: no findings at any severity; the intentional test-only LTO trade-off and unchanged release workflow were verified.
  • Finalization review: scope, closing link, profile isolation, documentation, and focused link checks are complete with no blockers.

Reports

  • TECHNICAL_REPORTS/1407-disable-test-thinlto-20260825.en.md
  • TECHNICAL_REPORTS/1407-disable-test-thinlto-20260825.ko.md

Closes #1406

The full macOS workspace gate can fail before execution when large integration test binaries link with explicit ThinLTO under the current rustc, LLVM, and Xcode toolchain. The optimized numerics requirement for the gate depends on opt-level 3, not on cross-crate LTO, and release binaries still use the release profile.

This changes only the test-fast LTO setting to lto=false while preserving opt-level 3, codegen-units 16, incremental compilation, unstripped local test binaries, and the inherited release panic behavior. The surrounding Makefile, contributor, installation, and nightly workflow guidance now describe the no-cross-crate-LTO test profile and keep the manual release-codegen escape hatch.

Validation: parsed Cargo.toml to confirm test-fast lto=false and release lto=true/codegen-units=1, ran cargo fmt --check, ran git diff --check on the changed files, and linked the molmo_parity and qwen3_omni_moe_parity integration test binaries with cargo test --profile test-fast --features metal,accelerate --test <target> --no-run.

Closes #1406
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions priority:high High priority platform:macos macOS (Apple Silicon) specific status:review Under review labels Aug 25, 2026
@inureyes

Copy link
Copy Markdown
Member Author

Security/performance review

  • Findings: no CRITICAL, HIGH, MEDIUM, or LOW security/performance findings in the changed build-profile, Makefile, workflow, or documentation files.
  • Build-profile audit: [profile.test-fast] now overrides inherited release LTO with lto = false while retaining inherits = "release", opt-level = 3, codegen-units = 16, incremental = true, and strip = false; [profile.release] remains fat-LTO with codegen-units = 1 and is unchanged by this PR.
  • CI/performance trade-off: the change intentionally removes test-time cross-crate LTO from the workspace gate to avoid macOS ThinLTO linker failure and should reduce test-link pressure; the documented residual risk is limited to defects that reproduce only under release LTO or single-codegen-unit release codegen, with release.yml and manual cargo test --release --features metal,accelerate still covering that escape hatch.
  • Validation: TOML profile assertions passed, git diff --check passed for changed files, .github/workflows/release.yml has no PR diff, and cargo metadata --no-deps --format-version 1 parsed successfully. Broad workspace Cargo gates were intentionally not rerun here because the orchestrator already owns them.

@inureyes inureyes added status:done Completed and removed status:review Under review labels Aug 25, 2026
@inureyes
inureyes merged commit fb26416 into main Aug 25, 2026
11 checks passed
@inureyes
inureyes deleted the fix/issue-1406-disable-test-thinlto branch August 25, 2026 10:36
@inureyes inureyes self-assigned this Aug 27, 2026
be-free-snowboy pushed a commit to freestylejs/snowbrain-mlxcel that referenced this pull request Oct 27, 2026
## Summary

- Disable LTO in `[profile.test-fast]` so macOS workspace integration-test binaries no longer enter the ThinLTO link path that produced missing internal Rust symbols.
- Preserve the optimized test behavior that matters for MLX numerics: `opt-level = 3`, `codegen-units = 16`, incremental compilation, unstripped local test binaries, and inherited unwind behavior remain unchanged.
- Leave `[profile.release]` unchanged with fat LTO and one codegen unit, and update Cargo, Makefile, contributor, installation, and nightly guidance to document the test/release trade-off accurately.
- Add bilingual technical reports covering the reproduction, decision, validation evidence, and residual release-codegen gap.

## Root cause

The full workspace no-run graph repeatedly failed while linking large integration targets such as `molmo_parity` and `qwen3_omni_moe_parity`, with unresolved LLVM-suffixed Rust drop-glue symbols. The failure reproduced with `CARGO_BUILD_JOBS=1`, while a targeted integration-test build could pass under a smaller unit graph, isolating the failure to the explicit ThinLTO workspace link shape rather than the test source or concurrent Cargo link jobs.

## Validation

- TOML profile assertions: `[profile.test-fast]` has `lto = false`, `opt-level = 3`, and `codegen-units = 16`; `[profile.release]` remains `lto = true`, `codegen-units = 1`, `strip = true`, `opt-level = 3`, and `panic = "unwind"`.
- `cargo test --profile test-fast --features metal,accelerate --test molmo_parity --no-run`
- `cargo test --profile test-fast --features metal,accelerate --test qwen3_omni_moe_parity --no-run`
- `cargo test --workspace --profile test-fast --features metal,accelerate --no-run` — passed; all workspace test targets linked in 9m19s on the cold corrected profile.
- `cargo test --workspace --profile test-fast --features metal,accelerate --no-fail-fast -- --test-threads=1` — passed with zero failures across all workspace test and doctest binaries.
- `cargo clippy --workspace --all-targets --features metal,accelerate -- -D warnings`
- `cargo fmt --all -- --check`
- `git diff --check origin/main...HEAD`
- GitHub CI: formatting, clippy, dependency policy, crate versions, kernel dtype keys, cross-repository references, OpenXLA feature compilation, and CLA checks passed before the report-only commit; the refreshed checks are expected to exercise the same implementation tree.

## Review

- Independent correctness review: no findings.
- Security/performance review: no findings at any severity; the intentional test-only LTO trade-off and unchanged release workflow were verified.
- Finalization review: scope, closing link, profile isolation, documentation, and focused link checks are complete with no blockers.

## Reports

- `TECHNICAL_REPORTS/1407-disable-test-thinlto-20260825.en.md`
- `TECHNICAL_REPORTS/1407-disable-test-thinlto-20260825.ko.md`

Closes lablup#1406
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

platform:macos macOS (Apple Silicon) specific priority:high High priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(test): disable ThinLTO in the workspace test gate

1 participant