diff --git a/README.md b/README.md index b1b32544..5311b109 100644 --- a/README.md +++ b/README.md @@ -26,10 +26,12 @@ with AMD Ryzen AI hardware behind it: - every Hugging Face model architecture, kept current by a daily census > **Status:** steps 1–3 of 5 have landed: embedded Lemonade ([docs/lemonade.md](docs/lemonade.md)), -> HRX + Vulkan in one build ([docs/hrx.md](docs/hrx.md)), and the NPU fast lane on full ELFs -> ([docs/npu.md](docs/npu.md)). Also: MLX on Apple Silicon ([docs/apple.md](docs/apple.md)), and -> ZINC with NVIDIA through its CUDA backend ([docs/zinc.md](docs/zinc.md)). The working engine is -> being ported from [1bit-MONSTER](https://github.com/1bit-MONSTER/1bit-MONSTER); see +> HRX + Vulkan in one build on AMD's live ggml-hrx ([docs/hrx.md](docs/hrx.md)), and the NPU engine on +> full ELFs with the upstream XDNA stack pinned ([docs/npu.md](docs/npu.md); its layer kernel is +> not yet built from source). Also: MLX on Apple Silicon through Lemonade's `mlx` recipe +> ([docs/apple.md](docs/apple.md)), and ZINC, which reaches NVIDIA GPUs through its CUDA backend +> ([docs/zinc.md](docs/zinc.md)). Next is step 4, the Laya router. The working engine is being ported from +> [1bit-MONSTER](https://github.com/1bit-MONSTER/1bit-MONSTER) in five steps; see > [docs/PORTING.md](docs/PORTING.md). Measured results are on the > [wiki](https://github.com/1bit-MONSTER/engine/wiki). This repository holds the verified code > without the development history. diff --git a/docs/PORTING.md b/docs/PORTING.md index 03b67e75..c1003203 100644 --- a/docs/PORTING.md +++ b/docs/PORTING.md @@ -23,7 +23,7 @@ Each step below is one PR (or a short series) that builds and runs on Strix Halo |---|---|---|---| | 1 | **Embedded Lemonade.** Lemonade's server core runs inside the `1bit` binary, with the `onebit` backend added | `third_party/lemonade` (v11.9.0 plus local deltas, see its `UPSTREAM.md`); `tools/unified_server.cpp` `run_embedded_lemonade` | **landed** ([docs/lemonade.md](lemonade.md)): `1bit lemonade` serves health, the catalog and system info; a smoke test runs in CI | | 2 | **HRX with Vulkan.** llama.cpp with `GGML_HRX=ON` and `GGML_VULKAN=ON` in one build, on AMD's tested pair | `1bit-MONSTER/llama.cpp` `1bit/hrx-vulkan` (AMD's `hrx-graph-develop-v2`) + `ROCm/hrx-system`, pinned from `ROCm/ggml-staging-automation` | **landed** ([docs/hrx.md](hrx.md)): Lemonade serves the same checkpoint on `Vulkan0` and `HRX0`; kept current by `bump-hrx.yml` | -| 3 | **NPU engine.** Full ELFs only, and the open 16-tile layer kernel built from source | `engine/npu` (`npu_engine_universal.cpp`, `I8Ctx::init_elf`); ELF dispatch table on `backup/iso-build-elf-native-2026-09-22`; kernel on `bench/fastlane-16tile-corrections-2026-09-22` | **3a landed** ([docs/npu.md](npu.md)): full ELFs generated in C++, reproducing all 5,153 captured contexts. **3b/3c**: the lane runtime (logits bit-identical to the reference lane, 24/24 steps, 11.0 ms/token), tokenizer, `1bit unified`, NPU models registered with `onebit`. Done when Lemonade loads Qwen3-0.6B on the NPU through `onebit` | +| 3 | **NPU engine.** Full ELFs only, and the open 16-tile layer kernel built from source | `engine/npu` (`npu_engine_universal.cpp`, `I8Ctx::init_elf`); ELF dispatch table on `backup/iso-build-elf-native-2026-09-22`; kernel on `bench/fastlane-16tile-corrections-2026-09-22` | **3a–3c landed** ([docs/npu.md](npu.md)). 3a (#8): full ELFs generated in C++, reproducing all 5,153 captured contexts. 3b/3c (#9): the lane runtime (logits bit-identical to the reference lane, 24/24 steps, 11.0 ms/token), tokenizer, `1bit unified`, and Lemonade serving Qwen3-0.6B on the NPU through `onebit`. The XDNA driver and XRT are pinned upstream and built privately (#12). **Open:** the layer kernel and lm-head artifacts are not yet built from source (npu.md, "Open") | | 4 | **Laya router.** A non-autoregressive scorer that picks where each request runs | `src/laya_scorer.cpp`, `include/laya_scorer.h` on `backup/laya-and-results-2026-09-22`; model at `~/models/laya` | matches its Python reference; routes requests | | 5 | **Every HF model, kept current.** The architecture registry (569 tokens mapping 2,030 HF arch strings) and the daily HF census that finds new architectures and proposes mappings | `src/model_registry*.cpp`, `Testing/census_*.py` and `.json`, `.github/workflows/census-{watch,sweep,autopr}.yml` | the census runs daily in CI; docs report *mapped* and *run and checked* counts separately | | 6 | **ZINC (NVIDIA and more).** Upstream `zolotukhin/zinc`, a Zig GGUF engine with Vulkan, ROCm, CUDA and Metal backends; its CUDA backend reaches NVIDIA GPUs (Ada `sm_89`, Blackwell `sm_120`) | not in 1bit-MONSTER; pinned from upstream `main` | **pinned** ([docs/zinc.md](zinc.md)): `scripts/build-zinc.sh` builds it privately; the Vulkan build gives 12095 (" Paris") at 295 tok/s on Strix Halo; the CUDA build answers " Paris." at 167–173 tok/s on an RTX 5090 (Qwen3.5-9B); kept current by `bump-zinc.yml`. `1bit lemonade` serves it as the `zinc` recipe (`-DONEBIT_ZINC=ON`, e2e passes) | diff --git a/docs/hrx.md b/docs/hrx.md index 99b332e1..4410a872 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -79,26 +79,33 @@ cmake --build build --target onebit llama-server as `IREE_HAL_AMDGPU_LIBHSA_PATH`, unless it is already set. To run llama-server or llama-bench by hand, export that variable yourself. -## Verified (2026-09-23, llama.cpp `f1a0aca`, hrx-system `fab1624`) +## Verified (2026-09-23, llama.cpp `f1a0aca`, hrx-system `51b1739`) -`tests/hrx_lemonade_e2e.sh` passes. `unsloth/Qwen3-0.6B-GGUF:Q4_0` answers -"Paris" through both recipes, served by this build's llama-server on `Vulkan0` -and on `HRX0`. +`tests/hrx_lemonade_e2e.sh` and `ctest` pass, re-run after the first daily bump +(#13). `unsloth/Qwen3-0.6B-GGUF:Q4_0` answers "Paris" through both recipes, +served by this build's llama-server on `Vulkan0` and on `HRX0`. llama-bench, Qwen3-0.6B, pp512 / tg128 tok/s, against the previous build (April llama.cpp with ggml-hrx2 on `HRX20`): | Device, file | previous build | this build | |---|---|---| -| Vulkan0, Q4_K_M | 10627 / 306 | **13905 / 338** | -| Vulkan0, UD-Q4_K_XL | 10668 / ~300 | 13505 / 330 | -| HRX, Q4_K_M | 1046 / 68 | **21702 / 303** | -| HRX, UD-Q4_K_XL | 837 / 83 | 18059 / 288 | +| Vulkan0, Q4_K_M | 10627 / 306 | **14097 / 349** | +| Vulkan0, UD-Q4_K_XL | 10668 / ~300 | 13943 / 324 | +| HRX, Q4_K_M | 1046 / 68 | **21990 / 314** | +| HRX, UD-Q4_K_XL | 837 / 83 | 17906 / 293 | +| Vulkan0, BF16 | n/a / 115 | 4504 / 144 | +| HRX, BF16 | n/a / 40 | 7646 / 101 | HRX prefill is now faster than Vulkan's. ## Not yet +- **IQ2/IQ3 types on HRX.** ggml-hrx has no IQ3_XXS matmul, but it still accepts + the op when the graph is scheduled, so there is no CPU fallback and the decode + fails (`unsupported HRX node ... iq3_xxs`). The Unsloth Dynamic UD-Q2_K_XL, + UD-IQ2_M and UD-IQ1_S files fail on `HRX0` for this reason; they run on + `Vulkan0`. UD-Q4_K_XL runs on both. - **Q4NX on HRX.** Our Q4NX kernels lived in ggml-hrx2 and are not in ggml-hrx. They are kept on `1bit/hrx2-archive` until they are ported. - **First-request cost.** Lemonade's telemetry for the first short HRX chat shows diff --git a/docs/lemonade.md b/docs/lemonade.md index d9f4cc77..e608ff8e 100644 --- a/docs/lemonade.md +++ b/docs/lemonade.md @@ -34,6 +34,8 @@ deltas listed in its `UPSTREAM.md`: - the `onebit` backend and `GET /v1/registry` - the `hrx-b66` pin and the HRX model-registry annotations - a small CMake patch that makes it embeddable +- the `hrx_device` option, which points the HRX recipe at this build ([hrx.md](hrx.md)) +- the `mlx` backend for Apple Silicon ([apple.md](apple.md)) It is copied from 1bit-MONSTER `main` (`third_party/lemonade`, last changed in `256e68dd7`). Files in `third_party/` keep their own license (Apache-2.0) and @@ -51,11 +53,15 @@ are exempt from this repository's copyright notice. On Strix Halo the catalog has 197 models across 16 recipes, and Lemonade reports the NPU as present. +## Since step 1 + +- **Step 2:** `1bit lemonade` serves the `llamacpp-hrx` and `llamacpp` (Vulkan) + recipes with this repository's HRX + Vulkan build instead of downloading AMD's + ([hrx.md](hrx.md)). +- **Step 3:** the `onebit` backend lists the NPU models found in the model + directories and serves them on the NPU ([npu.md](npu.md), "Serving (3c)"). + ## Not yet -- **The `onebit` backend lists no models.** Native artifacts are registered - when the NPU engine lands (step 3). -- **Lemonade's HRX recipe downloads AMD's HRX build.** Step 2 points it at this - repository's HRX + Vulkan build instead. - **The web UI is not built.** That needs Node.js; without it, Lemonade serves its static status page at `/`. diff --git a/docs/npu.md b/docs/npu.md index 1bc35190..9c7bf961 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -27,8 +27,8 @@ Step 3 lands in three parts: | Part | What | State | |---|---|---| | 3a | Full-ELF generation: per-context control code and full-ELF assembly | landed (#8) | -| 3b | The lane runtime: model reader, buffer packing, runlist decode (`npu/lane`, `npu/generate`, `1bit npu-run`) | **this PR** | -| 3c | Serving: tokenizer, `1bit unified`, NPU models registered with Lemonade's `onebit` recipe | **this PR** | +| 3b | The lane runtime: model reader, buffer packing, runlist decode (`npu/lane`, `npu/generate`, `1bit npu-run`) | landed (#9) | +| 3c | Serving: tokenizer, `1bit unified`, NPU models registered with Lemonade's `onebit` recipe | landed (#9) | ## A model directory @@ -216,7 +216,7 @@ lm-head artifacts do not meet that rule yet: - The instruction ELFs came from the old per-context generator. - The PDI came from the rounding-fixed `layer.xclbin` (1bit-MONSTER #2651). -This PR removes the need for per-context files and for the xclbin at run time. It does +Step 3a removed the need for per-context files and for the xclbin at run time. It does not produce the kernel itself. The from-source 16-tile kernel (1bit-MONSTER #2666) dispatches at 0.343 ms per layer, but it does not decode correctly yet: 0/24 tokens (#2668). Replacing the artifacts with a from-source kernel is tracked in [PORTING.md](PORTING.md).