Skip to content

Commit 86508a0

Browse files
justinchubyCopilot
andcommitted
docs: fix review comments on ONNX export skill
Address all 5 review findings plus remove --optimize from examples: 1. Fix --optimize description: clarify it applies mobius rewrite rules (group_query_attention, packed_attention, skip_norm), not general constant folding. Remove from basic command examples since it's optional, not default. 2. Fix EP descriptions: 'default' is portable ONNX (not CPU-specific), 'onnx-standard' inlines custom-domain functions (not DML-specific). Add note about 'mobius list eps' for all available EPs. 3. Replace Olive direct-API snippets with config-driven olive.run() pattern matching the repo's examples/olive/ convention. Reference the ministral example for a complete working setup. 4. Add tokenizer + processor config files to the multi-model quantization copy step — without these ORT GenAI won't load. 5. Replace fabricated generate_golden_data() with the real scripts/generate_golden.py entrypoint and compare_golden() from mobius._testing.parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
1 parent aae74a4 commit 86508a0

1 file changed

Lines changed: 78 additions & 40 deletions

File tree

  • .agents/skills/onnx-export-quantization

.agents/skills/onnx-export-quantization/SKILL.md

Lines changed: 78 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ Use this skill when:
2727
mobius build \
2828
--model <hf-model-id> \
2929
--dtype <f16|bf16> \
30-
--optimize \
3130
--ep <default|cuda|onnx-standard> \
3231
--runtime ort-genai \
3332
--external-data safetensors \
@@ -41,9 +40,9 @@ mobius build \
4140
|------|-------------|
4241
| `--model <id>` | HuggingFace model ID (e.g. `google/gemma-4-27b-it`) |
4342
| `--dtype <f16\|bf16>` | Model precision — `f16` (float16) or `bf16` (bfloat16) |
44-
| `--optimize` | Apply ONNX graph optimizations (constant folding, fusion) |
43+
| `--optimize [RULES]` | Apply mobius rewrite rules after building (e.g. `group_query_attention`, `packed_attention`, `skip_norm`). Use without value for all rules, or specify comma-separated names. Not needed for basic exports. |
4544
| `--ep <variant>` | Execution provider variant (see below) |
46-
| `--runtime ort-genai` | Generate `genai_config.json` for ORT GenAI runtime |
45+
| `--runtime ort-genai` | Generate `genai_config.json` and copy tokenizer files for ORT GenAI runtime |
4746
| `--external-data safetensors` | Store weights externally in safetensors format |
4847
| `--max-shard-size 5GB` | Split external data into shards ≤ 5GB |
4948

@@ -54,17 +53,20 @@ rewrites and fused ops:
5453

5554
| EP | Flag | When to use |
5655
|----|------|-------------|
57-
| `default` | `--ep default` | CPU inference. No custom ops — pure standard ONNX. Compatible with all runtimes. |
56+
| `default` | `--ep default` | Portable ONNX — no vendor-specific fusions. Compatible with all execution providers and runtimes. This is the default if `--ep` is omitted. |
5857
| `cuda` | `--ep cuda` | NVIDIA GPU inference. Emits `com.microsoft` fused ops (GroupQueryAttention, MoE, etc.) for maximum CUDA performance. |
59-
| `onnx-standard` | `--ep onnx-standard` | DirectML / cross-platform GPU. Standard ONNX ops only — no contrib ops. Works on AMD, Intel, and NVIDIA via DML. |
58+
| `onnx-standard` | `--ep onnx-standard` | Strict ONNX-only — inlines all custom-domain functions into standard ONNX ops. Use when targeting runtimes that don't support `com.microsoft` ops. |
59+
60+
Other EPs are available (`cpu`, `dml`, `webgpu`, `trt-rtx`). Run
61+
`mobius list eps` to see all options.
6062

6163
**Typical export matrix:** Build each dtype × EP combination:
6264

6365
```bash
6466
for dtype in f16 bf16; do
6567
for ep in default cuda onnx-standard; do
6668
mobius build --model google/gemma-4-12b-it \
67-
--dtype $dtype --optimize --ep $ep \
69+
--dtype $dtype --ep $ep \
6870
--runtime ort-genai \
6971
--external-data safetensors --max-shard-size 5GB \
7072
output/${dtype}/${ep}/
@@ -118,36 +120,49 @@ pip install cupy-cuda12x
118120
K-quant quantization uses mixed block sizes with importance-based bit
119121
allocation. Q4_K_M is a good balance of quality and size.
120122

121-
```python
122-
from olive.passes.onnx import OnnxKQuantQuantization
123-
124-
pass_config = OnnxKQuantQuantization(bits=4, block_size=32)
125-
result = pass_config.run(model_path, output_dir)
123+
The repo uses Olive's config-driven `olive.run()` pattern (see
124+
`examples/olive/` for working examples). A typical Olive config for
125+
k-quant quantization:
126+
127+
```json
128+
{
129+
"input_model": { "type": "OnnxModel", "model_path": "decoder/model.onnx" },
130+
"passes": {
131+
"kquant": {
132+
"type": "OnnxKQuantQuantization",
133+
"bits": 4,
134+
"block_size": 32
135+
}
136+
},
137+
"output_dir": "output/Q4_K_M/default/decoder"
138+
}
126139
```
127140

128-
**CLI equivalent with Olive:**
129-
130141
```bash
131-
olive quantize \
132-
--method kquant \
133-
--bits 4 \
134-
--block-size 32 \
135-
--input-model output/f16/default/decoder/model.onnx \
136-
--output-dir output/Q4_K_M/default/decoder/
142+
olive run --config kquant_config.json
137143
```
138144

139145
### NF4 quantization (4-bit NormalFloat)
140146

141147
NF4 uses a normal-distribution-optimized 4-bit format. Fast native C++
142148
implementation — no GPU needed.
143149

144-
```python
145-
from olive.passes.onnx import OnnxBnb4Quantization
146-
147-
pass_config = OnnxBnb4Quantization(precision="nf4")
148-
result = pass_config.run(model_path, output_dir)
150+
```json
151+
{
152+
"input_model": { "type": "OnnxModel", "model_path": "decoder/model.onnx" },
153+
"passes": {
154+
"nf4": {
155+
"type": "OnnxBnb4Quantization",
156+
"precision": "nf4"
157+
}
158+
},
159+
"output_dir": "output/NF4/default/decoder"
160+
}
149161
```
150162

163+
> See `examples/olive/ministral-3-3b-vlm/` for a complete working
164+
> example that combines mobius export with Olive quantization.
165+
151166
### GPU acceleration with cupy
152167

153168
Installing `cupy-cuda12x` gives a **19–51x speedup** for k-quant
@@ -168,30 +183,37 @@ pip install cupy-cuda12x
168183
### Quantizing multi-model exports
169184

170185
Quantize each sub-model independently. Typically only the decoder is
171-
quantized (it has the most parameters):
186+
quantized (it has the most parameters). Copy all other files needed
187+
for a complete ORT GenAI package:
172188

173189
```bash
174190
# Quantize decoder only (largest model)
175-
olive quantize --method kquant --bits 4 --block-size 32 \
176-
--input-model output/f16/default/decoder/model.onnx \
177-
--output-dir output/Q4_K_M/default/decoder/
191+
olive run --config kquant_decoder.json
178192

179193
# Copy other sub-models as-is (already small)
180194
cp -r output/f16/default/embedding/ output/Q4_K_M/default/embedding/
181195
cp -r output/f16/default/vision_encoder/ output/Q4_K_M/default/vision_encoder/
182-
cp -r output/f16/default/genai_config.json output/Q4_K_M/default/
196+
197+
# IMPORTANT: Copy config, tokenizer, and processor files too
198+
cp output/f16/default/genai_config.json output/Q4_K_M/default/
199+
cp output/f16/default/tokenizer* output/Q4_K_M/default/
200+
cp output/f16/default/image_processor.json output/Q4_K_M/default/ 2>/dev/null
201+
cp output/f16/default/audio_processor.json output/Q4_K_M/default/ 2>/dev/null
183202
```
184203

204+
Without the tokenizer and processor config files, ORT GenAI will fail
205+
to load the model.
206+
185207
## HuggingFace upload structure
186208

187209
### Standard directory layout
188210

189211
```
190212
<org>/<model>-onnx/
191213
├── f16/
192-
│ ├── default/ # CPU EP
193-
│ ├── cuda/ # CUDA EP
194-
│ └── onnx-standard/ # DML/cross-platform EP
214+
│ ├── default/ # Portable ONNX (no vendor fusions)
215+
│ ├── cuda/ # CUDA EP (fused ops)
216+
│ └── onnx-standard/ # Strict ONNX-only (inlined functions)
195217
├── bf16/
196218
│ ├── default/
197219
│ ├── cuda/
@@ -304,17 +326,33 @@ section on precision behaviour.
304326

305327
### L4: Golden data generation
306328

307-
Generate reference outputs from the full-precision model, then compare
308-
quantized model outputs:
329+
Generate reference outputs from the full-precision HuggingFace model
330+
using the golden data generation script:
331+
332+
```bash
333+
# Generate golden files for all test cases
334+
python scripts/generate_golden.py
335+
336+
# Generate for a specific task type
337+
python scripts/generate_golden.py --task-type causal-lm
338+
339+
# Generate for a single test case
340+
python scripts/generate_golden.py --case testdata/cases/causal-lm/gpt2.yaml
341+
342+
# Use GPU for large models
343+
python scripts/generate_golden.py --device cuda
344+
```
345+
346+
Golden reference files are stored in `testdata/golden/` as JSON. Use
347+
`compare_golden()` from `mobius._testing.parity` to compare model
348+
outputs against the reference:
309349

310350
```python
311-
# Generate golden data from f16 model
312-
from mobius._testing import generate_golden_data
351+
from mobius._testing.parity import compare_golden
313352

314-
generate_golden_data(
315-
model_path="output/f16/default/",
316-
output_path="golden/f16_default.npz",
317-
prompt="Hello, world!",
353+
compare_golden(
354+
model_output=output_logits,
355+
golden_path="testdata/golden/causal-lm/my_model.json",
318356
)
319357
```
320358

0 commit comments

Comments
 (0)