Skip to content

chore(build): make the MLX pin a single source of truth so a partial bump cannot silently build the old MLX #1047

Closed

Description

@inureyes

Problem / Background

The pinned MLX commit is written in two places that must agree, with no check that they do. Updating only one of them produces a build that looks bumped but compiles the old MLX, with no error and no warning.

The two locations, both currently 2c46b953db88965c4270cc7306eda6887a3247f2:

  • src/lib/mlx-cpp/CMakeLists.txt:95, the GIT_TAG passed to FetchContent_Declare(mlx ...). This decides what gets fetched.
  • src/lib/mlxcel-core/build.rs:217, const MLX_EXPECTED_COMMIT. This drives purge_stale_mlx_cache and mark_mlx_cache_valid, and is also exported to the crate as MLXCEL_MLX_COMMIT via cargo:rustc-env (build.rs:26).

The doc comment on the constant already states the coupling ("must match GIT_TAG in mlx-cpp/CMakeLists.txt") but nothing enforces it.

The failure mode, concretely

CMakeLists.txt prefers an existing cached checkout over the declared tag:

if(EXISTS "${_mlx_cached_source_dir}/CMakeLists.txt")
  set(mlx_SOURCE_DIR "${_mlx_cached_source_dir}")
  set(mlx_BINARY_DIR "${_mlx_cached_binary_dir}")
  message(STATUS "Using cached MLX source from ${mlx_SOURCE_DIR}")
else()
  FetchContent_Declare(
    mlx
    GIT_REPOSITORY "https://github.com/ml-explore/mlx.git"
    GIT_TAG 2c46b953db88965c4270cc7306eda6887a3247f2)

The comment above that block says so explicitly: "Stale-cache validation is handled by build.rs (purge_stale_mlx_cache) which runs before CMake, ensuring _deps/ is either valid or absent." So cache invalidation is delegated entirely to build.rs, which compares the _deps/.mlx-build-commit marker against MLX_EXPECTED_COMMIT and returns early when they match.

If a contributor bumps GIT_TAG alone, the marker still equals MLX_EXPECTED_COMMIT, _deps/ is judged valid and never purged, FetchContent is never re-run, and the build compiles the previous MLX. Nothing surfaces the mismatch: the tag in the repo says one commit, the linked library is another, and MLXCEL_MLX_COMMIT reports the stale one.

This was hit during the bump in #1042 / PR #1046 and caught only because the fetched _deps/mlx-src HEAD was checked by hand. It is the same class of silent-no-op that made the broken RMSNorm overlay in #824 / #829 survive as long as it did: the build appeared to do what was asked.

Proposed Solution

In preference order:

  1. Make build.rs parse GIT_TAG out of src/lib/mlx-cpp/CMakeLists.txt at build time and use that value, deleting MLX_EXPECTED_COMMIT as an independent constant. One place to edit, no possible divergence.
  2. Keep both but add a build-time assert in build.rs that reads the GIT_TAG line and fails the build with a clear message when it disagrees with MLX_EXPECTED_COMMIT. Cheap and fully removes the silent mode, though it still requires two edits.
  3. At minimum, after a successful build, verify the fetched _deps/mlx-src HEAD equals the expected commit and fail if not. This also catches a cache that was restored or seeded out of band, which options 1 and 2 do not.

Options 1 and 3 are complementary: 1 removes the divergence, 3 verifies what actually landed on disk. Consider both.

Already in place, no action needed

cargo:rerun-if-changed=../mlx-cpp/CMakeLists.txt is already present at src/lib/mlxcel-core/build.rs:181, so editing the tag does re-run the build script. Under option 1 the parsed value would then flow through correctly on that re-run. This is listed only so the implementer does not add a duplicate.

Acceptance Criteria

  • Editing the MLX pin in exactly one place produces a correct build.
  • Verified by bumping to a different commit and confirming the fetched _deps/mlx-src HEAD, the _deps/.mlx-build-commit marker, and the binary's embedded MLXCEL_MLX_COMMIT all move together.
  • A deliberately mismatched pin fails the build with an actionable message instead of silently compiling the old MLX.
  • No duplicate rerun-if-changed line is introduced for ../mlx-cpp/CMakeLists.txt.

Technical Considerations

purge_stale_mlx_cache runs from build_mlx() before Config::new("../mlx-cpp"), and mark_mlx_cache_valid runs after the CMake build returns, so any parsed-tag value must be resolved before the purge decision, not after. Both the purge comparison and the cargo:rustc-env export must consume the same resolved value, otherwise the fix reintroduces the divergence one level down.

Refs: found during #1042 / PR #1046.

Activity

  1. added
    type:choreMaintenance tasks (build, CI, etc.)
    area:coremlxcel-core: MLX FFI, primitives, KV cache, layers
    and removed on Aug 5, 2026
  2. added a commit that references this issue on Aug 20, 2026
    6276f55
  3. self-assigned this
    on Aug 31, 2026
  4. added 2 commits that reference this issue on Oct 27, 2026
    d980423
    45247b4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:coremlxcel-core: MLX FFI, primitives, KV cache, layerspriority:mediumMedium prioritystatus:doneCompletedtype:choreMaintenance tasks (build, CI, etc.)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions