Repository navigation
feat(aiperf): profile existing text-to-image models - #1787
Conversation
Connect the persistent native server to the public image-generation Task. Reuse the performance server's request mapping and return complete PNG bytes before temporary-file cleanup, with native artifact identity and Task timing records. Add existing-checkpoint discovery, a published-client image profiling command, transport regression tests and a concise usage guide. Owning-family correctness and group-wide GPU qualification remain separate prerequisites; model specs and quality criteria are unchanged. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
Apply the existing qualification profile's request controls before formatting AIPerf inputs. Omit unspecified guidance sentinels and require explicit dimensions, denoising steps, seed and guidance so requests describe the actual fixed workload. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Summary
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 19.72% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 71 functions across 25 files. (5 skipped:… | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Title check The title clearly and concisely identifies the main change: adding AIPerf profiling for existing text-to-image models. Description check The description completes all required sections with detailed motivation, exit criteria, implementation scope, change categories, validation evidence, environment details, remaining gaps, self-review,… Linked Issues check The PR addresses the coding requirements in #1673. It adds native image-generation dispatch andPOST /v1/images/generations, returns completeb64_jsonPNG data, records request identity and warmup…Out of Scope Changes check The changes remain connected to #1673. Server, native worker, qualification, catalog, test, tokenizer-build, and documentation changes support image qualification, public image responses, or reproduci…Family Ownership Boundary No family-ownership violation is introduced. The changed FLUX code (families/flux/model.py:1031-1043, 1098-1100) reads its own checkpoint tokenizer assets and uses the external Transformers tokenizer.… Shared Semantic Neutrality Shared changes are model-agnostic. endpoint_catalog.pyderives image inventory and video exclusion from existing catalog task, build, and testcase metadata.image_profile.pyobtains controls from …Benchmark Validation Integrity PASS. The changed paths keep a clear timing contract. apps/server/native_worker.cpp:548-560and:678-692measure only the public image Task call, then save PNG output after the timer. `apps/server…Shared Change Blast Radius The shared-surface change has a documented model-agnostic need and clear consumers. apps/server/native_worker.cppdispatches both legacyIImageGenerationand SDKTextToImagetasks through one pr…Full details: Description check
-
Explanation
The description completes all required sections with detailed motivation, exit criteria, implementation scope, change categories, validation evidence, environment details, remaining gaps, self-review, and risk rationale.
Full details: Linked Issues check
-
Explanation
The PR addresses the coding requirements in
#1673. It adds native image-generation dispatch andPOST /v1/images/generations, returns completeb64_jsonPNG data, records request identity and warmup data, joins Task and HTTP results, and rejects unsupported controls. It reports seven existing checkpoints, unchanged family checks on served bundles, published AIPerf 0.13.0 runs, separate latency metrics, reproducible commands, software, hardware, and validation evidence. The tokenizer changes support the required builds and preserve existing assets and contracts.
Full details: Out of Scope Changes check
-
Explanation
The changes remain connected to
#1673. Server, native worker, qualification, catalog, test, tokenizer-build, and documentation changes support image qualification, public image responses, or reproducibility. The Qwen and FLUX tokenizer fixes address build defects required to qualify the pinned existing checkpoints. No new models, manifests, variants, thresholds, or unrelated feature area is identified.
Full details: Family Ownership Boundary
-
Explanation
No family-ownership violation is introduced. The changed FLUX code (families/flux/model.py:1031-1043, 1098-1100) reads its own checkpoint tokenizer assets and uses the external Transformers tokenizer. The changed Qwen-Image code (families/qwen_image/model.py:496-511, 560, 601) does the same for its own checkpoint. The new tests import only their owning family modules. The central endpoint catalog change (apps/aiperf_qual/trtmc_aiperf_qual/endpoint_catalog.py:21-48) is a shared task-group inventory and metadata filter; it does not register or select one family through another. No changed family file imports, reads, links to, or reuses another family’s implementation, fixtures, references, or validation artifacts.
Full details: Shared Semantic Neutrality
-
Explanation
Shared changes are model-agnostic.
endpoint_catalog.pyderives image inventory and video exclusion from existing catalog task, build, and testcase metadata.image_profile.pyobtains controls fromresolve_case()and family-owned qualification configuration, then verifies generic bundle provenance. The serving changes add generic image API validation, PNG transport, capability registration, request records, and one-image output handling. They do not add a family name, model topology, tensor rule, engine composition, model-specific preprocessing, or model-specific metric or threshold.
Full details: Benchmark Validation Integrity
-
Explanation
PASS. The changed paths keep a clear timing contract.
apps/server/native_worker.cpp:548-560and:678-692measure only the public image Task call, then save PNG output after the timer.apps/server/python/trtmc_server/app.py:551-573records that Task time separately from worker round-trip time, whileapps/aiperf_qual/trtmc_aiperf_qual/image_profile.py:177-183reports separatetask_call_ms, HTTP latency, and throughput metrics. PNG validation and base64 serialization are included in the HTTP path and are not misreported as Task time. The profiler uses concurrency 1, counts one output per request, excludes warmups from measured statistics, and joins each client request to its native record by request ID. The existing text profiling contract uses the same separate native-call and client-latency accounting intext_profile.py:179-202. The changedimage_generationconsumers are covered by the image and video call sites and protocol tests.
Full details: Shared Change Blast Radius
-
Explanation
The shared-surface change has a documented model-agnostic need and clear consumers.
apps/server/native_worker.cppdispatches both legacyIImageGenerationand SDKTextToImagetasks through one protocol;trtmc_server/app.pyexposes/v1/images/generations;trtmc_perf_servingmaps image and video requests; andimage_profile.pyconsumes the catalog for all image profiles. The description states the prior deleted-URL problem, completeb64_jsonoutput, one-image and control compatibility rules, timing separation, provenance and PNG identity checks, and the seven-checkpoint validation. Added server, native-worker, serving, catalog, and profiler tests cover these contracts. These endpoint, transport, worker, and qualification behaviors are shared by multiple model families, so they cannot remain family-owned; the tokenizer fixes remain isolated in their owning families.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 19.72% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 71 functions across 25 files. (5 skipped: 5 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Use the public bundle inspector to validate the existing profile's checkpoint, immutable revision, Task, precision, quantization and explicit build controls before qualification. Preserve actual bundle build metadata in the result, including effective precision boundaries. Reject mismatched bundles before inference and cover identity and build-setting mismatches with regression tests. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
Keep frame count, duration and frame-rate options exclusive to the existing video transport. The single-image route now rejects those fields instead of silently discarding them. Add request-mapping regressions and cover quantization mismatches in bundle qualification. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
The existing pinned Qwen-Image snapshot provides vocabulary, merges and tokenizer configuration without tokenizer.json, so bundle building failed after compiling all three engines. Serialize those checkpoint assets with its Transformers fast tokenizer before engine construction. Preserve pre-serialized bytes when present and never write into the immutable snapshot. Regression tests retain token IDs, added tokens and cache contents; existing model specifications and correctness criteria remain unchanged. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
github-actions
Bot
removed
the
run-internal-ci
Preserve the server's no-text timing-log contract for image requests. Forward the full negative prompt to the native Task, but record only its SHA-256 and byte count alongside the numeric generation controls. Verify that private prompt text never enters the JSONL record. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
Serialize the official split vocabulary and merges with the checkpoint tokenizer before building engines. Preserve existing serialized assets, native bundle sections and cached checkpoint files. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
github-actions
Bot
removed
the
run-internal-ci
github-actions
Bot
removed
the
run-internal-ci
Use the startup timeout exercised by qualification so the example also works when replacing the small image bundle with a large existing model. Signed-off-by: Yizhuo Zhang <yizhuoz@nvidia.com>
github-actions
Bot
removed
the
run-internal-ci
Background
The performance server returned a server-local image URL whose artifact was deleted by default, and the persistent native server had no image-generation operation. Stage 6 of #1667 connects every existing text-to-image checkpoint to published AIPerf. Closes #1673.
Exit Criteria
88ceb9f(seven checkpoints, 20 existing variants).Implementation
b64_jsonbefore temporary-file cleanup; preserve files until cancelled native calls finish.image-profile. Verify checkpoint/revision/precision/build provenance and each decoded PNG's native artifact hash, geometry and count. Reject unsupported batching, streaming, URL, quality and video-only image controls.pip install aiperf==0.13.0, existing build-guide links and an extended startup timeout verified with large bundles. Timing records retain prompt hashes and numeric controls.Change categories
Validation
Commands and Results
All seven checkpoints passed their original owning-family model criteria on the exact served bundle, then actual PyPI AIPerf 0.13.0. Each client run used three prompts, concurrency one, five measured requests plus one warmup, zero errors and six complete PNG/native SHA-256 matches: 35 measured + 7 warmup, 42 verified responses in total. The corpus uses the original family prompt, a red bicycle at a white wall, and a watercolor mountain lake at sunrise.
Settings below list precision, resolution, steps and guidance/CFG; seed is 42 except Stable Diffusion (0). Existing mixed-precision layer defaults are retained. These shared-host, small-sample measurements use different existing workloads and are not capacity rankings.
All unique HTTP outputs were visually inspected. Stable Diffusion's existing 256×256/ten-step profile produces weak, often abstract images despite passing its unchanged PSNR/SSIM contract; PNG transport identity is not a semantic-quality claim.
Correctness: six checkpoints passed their full unchanged direct E2E selector (one requested/executed case, no requested skips). FLUX.2-dev's full BF16 build succeeded, but the combined build/native test OOMed loading the text encoder. That original failure remains recorded. After the build process exited, the identical BF16 bundle passed the owner's original
_native,_official_referenceand_assert_contract, with byte-identical matched latents, unchanged settings and thresholds: one fresh-process case passed in 356.45 seconds, zero skips. The paired native/reference images show the requested cat at sunset. Its subsequent real AIPerf run is the BF16 row above.python -m tools.ci pipeline source-quality: 307 passed with Ruff/clang-format onf5d08c74a.python -m tools.ci pipeline package: source archive, native wheel, installed SDK/CLI and immediate native-library loading passed for all 136 families on frozenf5d08c74a.9ebd43aea; changed serving/qualification and tokenizer-asset tests: 33 passed in the detachedf5d08c74areview. Qwen/FLUX family CPU suites: 20/36 passed. Native/HF tokenizer IDs match on 129/111 actual-checkpoint prompts; original vocabularies and merges are preserved.e16c79cbe; its only change fromf5d08c74ais the guide's tested startup-timeout flag and rationale.Reproduce each row with its table profile, retained already-built bundle and three-prompt JSONL; the environment points to the native runtime, server interpreter and pip-installed client:
For direct serving, use
trtmc-server BUNDLE --model-name MODEL --startup-timeout 1800 --request-timeout 1800; the qualifier sets those timeouts automatically. The website guide contains the directaiperf profileexample. Original correctness entry point:python -m pytest families/OWNER/tests/test_e2e.py --e2e-model PROFILE -qwith the documented runtime/cache environment. The FLUX.2 retained-artifact verification imports and calls the same owner's native/reference/comparison functions after its successful full build; it does not replace or modify the failed combined selector.Hardware, Environment, and Revisions
Linux aarch64, shared GB300, CUDA 13.3, TensorRT 11.1, PyTorch 2.12, Diffusers 0.39.0, published AIPerf 0.13.0. Checkpoint pins and model settings are unchanged from the linked manifests. Implementation head:
e16c79cbefe3adeef561bb704d2dbf92c8ed075a.Measurement epochs remain explicit: PixArt
216358918; Stable Diffusion/Z-Image27148b4d0; both Qwen models and both FLUX modelsf5d08c74a. Later changes add request validation/provenance, prompt-record privacy, family tokenizer assets and the guide timeout; measurements are not relabelled as executions on a later head. Per-request raw data, hashes, bundles, software/configuration and failure attempts are retained locally.Not Run / Remaining Gaps
Additional precision, quantization, distributed and partial/L0 variants are inventoried without qualification claims. The original combined FLUX.2 BF16 selector remains an OOM failure; its unchanged model criteria and actual serving workload are independently verified as described above. Required internal CI passed on current head
e16c79cbebefore normal squash merge. Stable Community CI, CPU, metadata, DCO and CodeRabbit passed; no review threads remained unresolved. Optional experimental Dev CI was still running at merge and is not claimed as passed. Repository merge requirements remain unchanged.Contributor Self-Review
Notes For Future Readers
One image per nonstreaming request is the qualified workload. Model semantics and quality remain family-owned; HTTP/native identity certifies transport. Editing and video are the following separate stages. The guide is
website/docs/user-guides/profile-images-with-aiperf.md.Risk level
Persistent worker dispatch and media lifetime change. Complete-output/ownership regressions, real existing-model qualification and exact-head CI provide separate validation tiers.