Skip to content

1bit route: print the scorer's load and decision times - #91

Merged
bong-water-water-bong merged 1 commit into
mainfrom
laya/route-timing
Sep 25, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
laya/route-timing

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

1bit route now prints how long the Laya scorer took to load and to decide, on stderr, so the per-request cost of 1bit serve --laya-model can be seen.

Measured on Strix Halo (pinned root checkpoint, three candidate devices): load 2.8 s once, then 8.75 s per decision. In 1bit serve --laya-model, every chat request waits that long before the backend starts generating. The option is opt-in, and making the decision cheaper is the next step for the router.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Timing Measurement Accuracy

The timing measurements use std::chrono::steady_clock, which is appropriate for measuring intervals. However, the reported times (2.8 s load, 8.75 s per decision) seem unusually high for a model loading and routing operation, especially in a hardware-accelerated environment like Strix Halo. This could indicate either a measurement artifact or a performance issue that should be verified on the actual hardware. The PR description claims these numbers are measured on Strix Halo, but without a specific test or benchmarking setup, it's unclear if these values are accurate or representative.

const auto t0 = std::chrono::steady_clock::now();
onebit::laya::Scorer scorer;
if (!scorer.load(laya_model)) {
    std::fprintf(stderr, "1bit route: %s\n", scorer.error().c_str());
    return 1;
}
const auto t1 = std::chrono::steady_clock::now();
const std::string device = onebit::laya::route_device(scorer, state, devices);
if (device.empty()) {
    std::fprintf(stderr, "1bit route: %s\n", scorer.error().c_str());
    return 1;
}
const auto t2 = std::chrono::steady_clock::now();
std::printf("%s\n", device.c_str());
auto ms = [](auto a, auto b) { return std::chrono::duration<double, std::milli>(b - a).count(); };
std::fprintf(stderr, "1bit route: scorer loaded in %.0f ms, decision in %.0f ms\n", ms(t0, t1), ms(t1, t2));

@bong-water-water-bong
bong-water-water-bong merged commit 1b1a847 into main Sep 25, 2026
3 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the laya/route-timing branch September 25, 2026 18:34
bong-water-water-bong added a commit that referenced this pull request Sep 25, 2026
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant