[inureyes]

Summary

Zero-config pipeline bring-up works end to end: mlxcel-server --pp-auto N pushes a stage assignment to N --pp-peer processes over the authenticated cluster handshake, in static and mdns discovery, and the peers need neither a copied TOML nor a guessed --node-id.

Design as implemented

• New src/distributed/stage_assignment.rs (re-exported from mlxcel::distributed): after client_handshake/server_handshake (protocol v2 unchanged), u32-length-prefixed JSON frames run Offer -> Accepted/Rejected -> Commit -> Committed, 64 KiB cap checked before allocation, 10 s per frame. The coordinator sends Commit only after every peer accepted, so a failed offer phase commits nobody; AssignmentPushFailed names each failing peer with its reason (never connected plus the last connect error, handshake failure, no answer, or the peer's typed rejection). The offer carries ModelIdentity (SHA-256 of config.json, sorted safetensors names and sizes, layer count) and the coordinator's layer range; the peer re-derives both from its checkpoint.
• Peer verification, in order: protocol, mlxcel version, cluster name, model identity, stage count not above the layer count, layer range, node_id == "stage-{stage_index}", the TOML (parses, holds our stage at our listener port, exactly one non-stage node whose address agrees with the handshake-bound coordinator identity), and coordinator_control_addr against the bound identity. Protocol, version, model, and layer-range mismatches end the peer with AssignmentRefused { reason, detail }; the rest return it to listening. One committed assignment per run (atomic claim, listener dropped, beacon task stopped).
• Peer listener: pre-auth sockets bounded by PendingHandshakeLimiter::new(8, 4) from hardening(distributed): Bound pre-auth slots, type handshake rejections, extend cluster auth #2655, handshake failures logged through the transport's rate limiter, 600 s overall wait, 60 s wait for Commit after Accepted (not capped by the 10 s frame timeout). In mdns mode the peer runs broadcast_beacon_loop (new target parameter from --cluster-discovery-target, default 255.255.255.255) every second until commit.
• Coordinator: push_stage_assignments drives all peers concurrently; connect plus handshake is retried every 500 ms (covers a static peer starting late and a peer whose pending pool is briefly full) and stops early only on HandshakeRejected. The whole push is bounded by ASSIGNMENT_ACK_TIMEOUT (60 s); the offer phase ends 10 s early (commit_reserve) so the commit phase always has a full frame window inside that bound.
• run_zero_config_bring_up: the coordinator binds the mdns receiver to the unspecified address, refuses a loopback control address with non-loopback peers (error names --cluster-control-addr and --host), and calls push_stage_assignments between plan_cluster and the TOML write. The --pp-peer branch resolves the secret fail-closed before any socket, waits in await_stage_assignment, writes the localized cluster (coordinator address replaced by the handshake-bound identity when offered unspecified, own address replaced by the listener address, no secret_file) to --cluster-config-out (default .mlxcel/cluster-<node-id>.toml, node id re-checked to [A-Za-z0-9_-]), and sets distributed_config/node_id, so resolve_distributed_startup and the stage service are unchanged. --pp-peer --distributed-config --node-id keeps the manual path.
• CLI: --cluster-discovery-target <IP> in mlxcel-server and mlxcel serve; help for --pp-auto, --pp-peer, --cluster-discovery, --cluster-name, --cluster-peers, --cluster-discovery-port, --cluster-control-addr, --cluster-config-out rewritten. --cluster-control-addr is the peer's listen address (required with a concrete port in static mode, 0.0.0.0:0 default in mdns). docs/distributed.md gains a "Zero-config bring-up" walkthrough (secret distribution, two peers plus coordinator for static and mdns, firewall ports, timeouts) and the beacon bullet covers the assignment channel. No new environment variable.

Deviations from the issue text

• The cluster rewrite (bound coordinator identity, listener address) happens inside await_stage_assignment rather than in startup.rs, so peer_substitutes_bound_identity_for_unspecified_coordinator can assert on CommittedAssignment.cluster; startup only renders and writes it.
• --dry-run with --pp-auto plans and writes the TOML without pushing assignments, and --pp-peer --dry-run (without --distributed-config) is refused before any socket is bound; otherwise peers would commit to a coordinator that exits.
• The pre-auth limit is the shared PendingHandshakeLimiter (8 total, 4 per IP) instead of an ad-hoc cap of 8 concurrent handshakes.
• Additional peer checks beyond the issue (node id must be stage-{index}, coordinator node in the TOML must match the bound identity, stage count bounded by layer count), added during the security pass: without them a secret holder acting as coordinator could make a peer write outside .mlxcel/ or point the stage at a foreign coordinator.
• The offer phase ends at 50 s of the 60 s push bound to reserve the commit window, so a late Accepted cannot leave some peers committed while the push fails on time.
• zero_config_cli_* take fixed ports from 19500-19549 when free (this host's allotted block) and fall back to ephemeral ports.

Verification (GB10 spark-102, --profile test-fast --features cuda, --test-threads=1, GPU runs under gpu-lock run --tag issue-2658)

Final head:

• cargo test --lib distributed::stage_assignment::: 14 passed. cargo test --lib server::startup: 97 passed.
• cargo test --test zero_config_cluster_init: 4 passed (static and mdns weightless bring-ups), 3 ignored.
• cargo test --test cli_help_consistency: 27 passed.
• cargo clippy -p mlxcel --lib --tests -- -D warnings, cargo fmt --check, make verify-env-var-docs: clean.

Earlier passes on this branch: cargo test --lib distributed:: 1411 passed (security-pass head); ignored zero_config_cli_static_matches_single_node and zero_config_cli_mdns_matches_single_node with MLXCEL_REQUIRE_MODELS=1 MLXCEL_SDPA_DETERMINISTIC=1: 2 passed (developer, review, and security heads). --test distributed_integration not run: it references none of the changed functions.

Real-binary E2E (final head; qwen3-0.6b-4bit, MLXCEL_SDPA_DETERMINISTIC=1, single node and all three cluster processes under one gpu-lock run --tag issue-2658 -- /usr/bin/timeout 800 bash <script> with PID cleanup, loopback ports 19500-19549)

mdns: peers mlxcel-server -m <model> --pp-peer --cluster-discovery=mdns --cluster-discovery-target 127.0.0.1 --cluster-discovery-port 19549 --cluster-control-addr 127.0.0.1:1950{1,2} --cluster-secret-file <f> --cluster-config-out peer{0,1}.toml --host 127.0.0.1 --port 1951{1,2}; coordinator mlxcel-server -m <model> --pp-auto 2 --cluster-discovery=mdns --cluster-discovery-port 19549 --cluster-control-addr 127.0.0.1:19500 --cluster-secret-file <f> --cluster-config-out coordinator.toml --host 127.0.0.1 --port 19520; single node mlxcel-server -m <model> --host 127.0.0.1 --port 19530. Request: /v1/chat/completions, temperature: 0, max_tokens: 24, prompt "Name three primary colors.", enable_thinking: false.

	mdns	static
single-node HTTP status	200	200
cluster HTTP status	200	200
text equal to single node	yes	yes

Text (both modes, both servers): Three primary colors are **red, blue, and yellow**. These are the basic colors in the color wheel, representing. Logs: coordinator Pushing stage assignments to 2 peer(s) as 127.0.0.1:19500 (timeout 60s), Peer 127.0.0.1:19501 committed stage assignment stage-0, Peer 127.0.0.1:19502 committed stage assignment stage-1, Zero-config pipeline cluster ready: wrote .../coordinator.toml (2 stage(s)); peers Committed stage assignment stage-0 / stage-1.

static: same processes with --cluster-discovery=static, no target or discovery port, and --cluster-peers 127.0.0.1:19501,127.0.0.1:19502 on the coordinator.

Real-broadcast check (developer head; default target 255.255.255.255, discovery only, nothing bound beyond loopback except the coordinator's UDP receiver on 0.0.0.0:19548): one --pp-peer --cluster-discovery=mdns --cluster-control-addr 127.0.0.1:19541 beaconing with the default target, one --pp-auto 2 --cluster-discovery=mdns --cluster-discovery-port 19548 --cluster-control-addr 127.0.0.1:19540. The coordinator logged once per second cluster discovery: rejected beacon from 10.100.64.122: beacon advertises 127.0.0.1 but was sent from 10.100.64.122 and then timed out: the limited-broadcast datagram reaches the unspecified-bound receiver, and the source-IP binding rejects a loopback listener advertised from the LAN, as designed. A full broadcast bring-up would need the stage listeners on a LAN address, which this run does not bind.

Pre-merge check (main merged in, head 0e3fd2d)

• make verify-fmt, make verify-env-var-docs, cargo clippy --features cuda,test-utils -p mlxcel --lib --tests -- -D warnings: clean.
• --lib distributed:: server::startup: 1509 passed, 0 failed.
• Real-model remote pipeline (MLXCEL_REQUIRE_MODELS=1 MLXCEL_SDPA_DETERMINISTIC=1): pipeline_remote_real_models llama/gemma3/gemma4/qwen3/qwen35 parity plus llama drain 6/6, pipeline_server_remote_real_models 5/5, pipeline_ci_multi_stage_real_models two-host smoke 1/1.
• --test zero_config_cluster_init --include-ignored: 7/7, including the real-model static and mdns single-node match tests.

Known limitations (left open)

• Assignment and stage-transport handshakes carry no channel label, so they are interchangeable for a secret holder; per-frame MAC is out of scope.
• A beacon advertising 0.0.0.0:port signs no IP; a LAN attacker who can spoof UDP can replay it from another source, which can only fail the bring-up (the handshake still binds identity).
• If some peers fail during the commit phase itself, the peers that already committed stay committed while the coordinator exits; this is inherent to a coordinator without an abort message.

Closes #2658

[inureyes]

Implementation Review Summary

Intent

--pp-auto N pushes and commits stage assignments to N --pp-peer processes over the authenticated handshake, in static and mdns discovery, with no hand-copied TOML or guessed --node-id (#2658).

Findings Addressed

• Peer commit wait was cut to 10 s: the wait for Commit went through read_frame, whose ASSIGNMENT_IO_TIMEOUT fired before the 60 s ASSIGNMENT_COMMIT_TIMEOUT, so a bring-up failed whenever the coordinator needed more than 10 s to reach a later peer. Fixed in c8d257c with regression test peer_waits_for_commit_beyond_frame_io_timeout (HIGH)
• AssignmentRejectReason doc and docs/distributed.md called the "first four" rejections terminal, which included cluster name and missed layer range. Both now name the four terminal reasons (MEDIUM)

Remaining Items

• --pp-peer --dry-run waits for and commits an assignment, then exits with "--dry-run was requested but --pp-auto was not provided", leaving the coordinator with a dead stage. It should refuse before binding (MEDIUM)
• The mdns walkthrough in docs/distributed.md omits --cluster-control-addr on the peers, so they listen on an OS-assigned port, but the firewall paragraph says to open each peer's control port. A fixed port such as 0.0.0.0:19001 would make the example firewallable (MEDIUM)
• The 0.0.0.0:0 default listen address in mdns mode is exercised only by unit logic; every real-binary run used explicit loopback addresses (LOW)
• When a connect never completes, the push failure reads "last connect error: none; connected but no answer" (LOW)
• tests/zero_config_cluster_init.rs hard-codes the 19500-19549 port block from this host's allotment; it falls back to ephemeral ports (LOW)

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass

Reviewed on GB10 under gpu-lock at c8d257c: stage_assignment 11 passed, cluster_init 19, server::startup::tests 88, zero_config_cluster_init 4 plus the 2 ignored real-binary qwen3 tests (MLXCEL_REQUIRE_MODELS=1 MLXCEL_SDPA_DETERMINISTIC=1), clippy -D warnings and fmt clean.

[inureyes]

Security and performance review of the assignment channel, fixed in 0cb9fd3.

Fixed:

• HIGH: path traversal through the offered node id. The peer wrote .mlxcel/cluster-<node_id>.toml with create_dir_all, and the TOML validator allows / and .. in ids, so any secret holder posing as a coordinator could create directories and overwrite .toml files the peer user can write. The peer now accepts only stage-<stage_index>, and startup re-checks the id before building the path.
• MEDIUM: the coordinator node in the offered TOML was not bound to the handshake identity (only the coordinator_control_addr field was). The TOML must now hold exactly one coordinator node that agrees with the bound identity.
• MEDIUM: pre-auth slot exhaustion failed the bring-up. Idle unauthenticated sockets holding the peer's pending-handshake slots made the peer close the coordinator's socket at accept, and the coordinator did not retry. It now retries connect and handshake every 500 ms until the push deadline; HandshakeRejected still fails at once.
• LOW: num_stages larger than the layer count is refused before the partition, which allocates per stage. Pre-validation log lines print the offered node id with {:?}.

Not fixed (low impact or by design):

• Offer and commit share one 60 s deadline. A peer that accepts late can leave too little time for Commit, and then some peers commit while the push fails.
• The handshake transcript has no channel label, so assignment and stage-transport handshakes are interchangeable. A network attacker cannot exploit this without the secret, and frames carry no MAC anyway (out of scope).
• A beacon advertising 0.0.0.0:port signs no IP. A LAN attacker who spoofs UDP can replay it from another source IP ahead of the original. That attacker cannot complete the handshake, so the only effect is a failed bring-up.
• The peer's accept failed warning is not rate limited. The 50 ms sleep caps it at 20 lines per second.

Checked and fine: the 64 KiB cap is enforced before the body allocation; JSON is parsed only after the handshake; the pending-handshake limiter (8 total, 4 per IP) is held until the handshake finishes; every wait is bounded; no task leaks; the secret resolves before any socket is bound; and secret_file is cleared on both sides. Beacons still verify the source IP with the receiver on the unspecified address. The loopback-control check runs before the push.

Tests: distributed:: lib (1411 pass), server::startup (96 pass), zero_config_cluster_init (4 pass), and ignored real-model zero_config_cli_{static,mdns}_matches_single_node on qwen3-0.6b-4bit (2 pass). Clippy and fmt are clean.