Question: can I take INT4-AWQ for llama and FP8 for the stable_diffusion SDXL-Turbo UNet? #1928

Topic

Architecture or contribution guidance

Question

@zhenshanx-nv I'd like to contribute two quantization paths that follow your self-quantization series (#1907, #1898, #1918). Could you confirm both are free to take, so I stay out of your queue?

1. llama: native INT4-AWQ (W4A16), first

llama accepts fp8 and nvfp4 today. Native INT4-AWQ exists in qwen3_8, qwen3_5, internvl and phi4_multimodal, all loading prepared ModelOpt W4A16_AWQ checkpoints. llama has no public W4A16 checkpoint, so I'd self-quantize the original checkpoint at build time:

• calibrate_awq_scales.py (developer tool, ModelOpt INT4_AWQ_CFG) produces the per-layer pre_quant_scale, checked in and keyed to the checkpoint, like fp8_activation_scales.json.
• The build computes INT4 weights and block scales (block size 128) from the original weights and emits INT4 constant → Dequantize → MatMul with the smoothing multiply on the activation, matching _AWQFormat in qwen3_8.
• Decoder q/k/v/o and MLP projections are quantized; embeddings, norms and lm_head stay FP16. A missing scale fails the build.
• E2E case on nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1, already pinned at d552708a in the llama manifests.

2. stable_diffusion: FP8 for the SDXL-Turbo UNet, after #1909 lands

• Same structure as feat(ast): FP8 self-quantization of the original checkpoint #1907: quantization.py, calibrate_fp8_scales.py, checked-in scales, fp16 Dequantize.
• UNet attention and feed-forward linears are quantized; convolutions, norms, time embedding, text encoders and VAE stay FP16.

Questions

1. Are both free to take? If either is already on your list, which family would you like an external contributor to pick up instead?
2. For INT4-AWQ, is a checked-in pre_quant_scale file (a few MB for a 4B model) acceptable, or do you prefer calibration at build time?
3. I validate on an RTX 3090 (SM86) and an RTX 6000 Ada (SM89) with TensorRT 11.1: quantized GEMMs confirmed in the plan inspector, engine size and latency against the fp16 engine, and output parity against the unquantized reference. Does that cover review without B200/B300 numbers?

What have you tried?

Read #1907, #1898, #1918 and the qwen3_8 INT4-AWQ path (_AWQFormat, load_qwen3_8_awq). No open PR or issue covers llama INT4-AWQ or stable_diffusion quantization.

TensorRT-Model-Connect version

main @ 2c2256c

Model and revision

nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 @ d552708a9d575fa8d4a690b988fd870d65279f98; stabilityai/sdxl-turbo at the revision pinned in families/stable_diffusion/tests/manifests/sdxl-turbo.json

Environment details

Ubuntu (Linux 6.8), RTX 3090 (SM86) + RTX 6000 Ada (SM89), driver 580.178.04, TensorRT 11.1.0.106, source build

Submission checks

• I searched the documentation and open and closed issues first.
• This question contains no secrets, private URLs, or restricted artifacts.