NPU lane: fresh hw_context per generate(), fixes the idle timeout

[bong-water-water-bong]

After about 8 s idle, the NPU lane's single long-lived hw_context failed its next runlist with ERT_CMD_STATE_TIMEOUT. It also left a stuck context that blocked every NPU user for 1–2 minutes, which broke 1bit serve's NPU route between requests.

Fix: Lane::begin() creates the context (init load_pdi run + lm-head config) at the start of each generate(), and Lane::end() drops it while warm, also on the exception path. Weights, KV and activation buffers are independent of the context and stay allocated once.

Measured on strixhalo (lane_idle2, Qwen3-0.6B, runtime PM at its default control=auto):

• 15 s idle: 10/10 rounds, 97–101 tok/s (re-verified on this branch).
• 120 s idle: 10/10 rounds, 94–100 tok/s (pi agent-74509b's run).
• Afterwards xrt-smi shows no hardware contexts.

Before the fix, round 1 onwards failed at 8 s and at 15 s idle.

Follow-ups:

• The npu-draft bench drives the lane with step() directly, so it needs begin()/end() when it lands.
• The 35B path (npu/lax.cpp) also keeps long-lived contexts and has not been tested for idle yet.

Root cause and patch: pi agent-74509b (goal mugi4zva); docs note added in docs/npu.md.

🤖 Generated with Claude Code

[context7]

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit a594d00

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit a594d00)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Context Management in generate() The new generate() function now calls lane.begin() at the start and lane.end() at the end (including in exception paths). This ensures a fresh hardware context is used for each generation, preventing the timeout issue caused by idle contexts. However, this change assumes that generate_once does not rely on any state that might be invalidated by the context recreation. If generate_once or any of the functions it calls (like step) depend on prior context state, this could lead to incorrect behavior or errors. GenerateResult generate(Lane& lane, const Model& model, const std::vector<int>& prompt, const GenerateOptions& opt) { lane.begin(); try { GenerateResult r = generate_once(lane, model, prompt, opt); lane.end(); return r; } catch (...) { lane.end(); throw; } } Hardware Context Lifecycle The begin() and end() methods in Lane::Impl now manage the lifetime of the hw_context. The begin() method creates a new context and configures it with the init and lm-head configurations, while end() resets runlists, kernels, and the context itself. This approach correctly addresses the idle timeout issue, but care must be taken to ensure that all resources managed by the context are properly cleaned up in end() to avoid resource leaks or dangling references. void begin() { const Bytes init = assemble_full_elf(ctx1, pdi, "flinit", PdiMode::kInitOnly); ctx = std::make_unique<xrt::hw_context>(dev, xrt::elf(reinterpret_cast<const char*>(init.data()), init.size())); { xrt::ext::kernel k(*ctx, "flinit"); xrt::ext::bo unused(dev, 4096); xrt::run r(k); r.set_arg(0, unused); r.start(); if (r.wait(std::chrono::milliseconds(5000)) != ERT_CMD_STATE_COMPLETED) throw std::runtime_error("NPU init run did not complete"); } ctx->add_config(xrt::elf(reinterpret_cast<const char*>(head.data()), head.size())); lmhead = std::make_unique<xrt::ext::kernel>(*ctx, "flhead"); layer_kernels.clear(); slots[0].rl.reset(); slots[0].runs.clear(); slots[1].rl.reset(); slots[1].runs.clear(); } // Destroy the hw_context while it is warm (every run completed), so its // destructor never waits on a stuck command. Drop the runlists, runs, // layer kernels and lm-head kernel before the context they reference. void end() { slots[0].rl.reset(); slots[0].runs.clear(); slots[1].rl.reset(); slots[1].runs.clear(); layer_kernels.clear(); lmhead.reset(); ctx.reset(); }

[github-actions]

Persistent review updated to latest commit a594d00