[yizhuoz004]

Background

Closes #1670; part of #1667. Serve complete prerecorded audio through the existing native server so pip-installed AIPerf can qualify real speech-to-text inference.

Exit Criteria

• Multipart /v1/audio/transcriptions invokes a declared native transcription Task and returns the complete transcript.
• Preserve audio identity, request correlation, admission, errors, upload lifetime, and separate public Task timing.
• The actual Whisper bundle passes unchanged family correctness/WER criteria and a real published-client run.

Implementation

• Dispatch legacy ITranscription and public SDK SpeechTranscription capabilities. Reuse native media I/O; family preprocessing, resampling, decoding, and model math remain family-owned.
• Accept one complete mono PCM16 WAV file, validate its actual rate/sample count/duration, and support JSON/plain-text transcripts. Reject unsupported streaming, translation, language, prompt, sampling, and timestamp controls. max_output_tokens is an explicit TRTMC extension passed through declared native configuration.
• Retain uploaded files until native completion, including cancellation and retirement; reuse existing workers, admission, metrics, health, and terminal records. Add python-multipart to optional serving/test dependencies.
• Add legacy/SDK and HTTP regressions, a pinned LibriSpeech preparation helper using the existing dataset pin, a three-step guide, and a pilot coverage row. Corpus preparation preserves complete PCM samples and audits skipped/failed rows. Reading a local pinned parquet shard avoids leaving streaming filesystem callbacks active at interpreter shutdown.

Public Task ABI, bundle format, numerical criteria, and checkpoint weights are unchanged. The Whisper family now honors checkpoint GELU in the convolution stem and encoder/decoder MLPs, evaluating its negative tail in FP32 before restoring the graph dtype. The explicitly FP32 encoder also disables TF32 operand rounding; the first checkpoint convolution RMSE against FP32 PyTorch drops from 6.37e-5 to 1.35e-7 on the same input. The decoder computes the output projection in FP32 before token selection, preserving close candidate differences that a cast after FP16 computation loses; TF32 is disabled for this FP32 boundary as well. Older Whisper bundles require rebuilding. The correction stays within the owning family; no sibling family implementation changes.

Change categories

• Model or runtime behavior
• Public API
• Dependencies
• CI or developer tooling

Validation

Commands and Results

Tested head: 572e28db766daa0279e4dbec90e7f615420d732c; base: d45de067106482104fe0564edef4cf030ece772c. The worktree was clean for the fresh final corpus and published-client run after the GELU, FP32 encoder, and output projection corrections.

PYTHONPATH=apps/server/python:apps/perf_serving:apps/aiperf_qual:core/builder:. \
  python -m pytest apps/server/python/tests apps/perf_serving/tests -q --import-mode=importlib
ctest --test-dir build --output-on-failure \
  -R '^server_(worker|sdk_worker|sdk_worker_process|embedding_worker|ranking_worker|transcription_worker)$'
PYTHONPATH=core/builder:apps/benchmark:. python -m pytest \
  families/whisper/tests/test_e2e.py --e2e-model whisper-tiny-fp16 \
  --e2e-testcase whisper-tiny-fp16 -q
PYTHONPATH=core/builder:apps/benchmark:. python -m tools.community_ci source-quality --base github/main
npm --prefix website run test:model-support
npm --prefix website run build
git diff --check github/main...HEAD

• Serving/transport tests: 146 passed. Native serving tests: 6 passed, including legacy and SDK transcription and cancellation/input-resource ownership regressions.
• Selected owning-family suite: 8 passed, including the actual checkpoint/native/reference testcase with its unchanged strict transcript criteria, plus real TensorRT FP16/FP32 GELU and close-candidate output projection checks against PyTorch. The GELU checks fail against unchanged main and pass with the family-local correction. Reinstating the legacy projection also fails the new FP16 close-candidate regression; the FP32 projection preserves the expected ranking. The exact served bundle also passed the profile's unchanged ten-clip WER checks: 0.0 WER to the FP32 reference, and 3.59% WER to gold on both sides (max_wer_to_reference=0.1, maximum increase from reference 0.02).
• Minimum declared serving versions (FastAPI 0.115.0, Pydantic 2.11.0, python-multipart 0.0.20): 26/26 ASR tests passed. Removed an unsupported Starlette multipart option while retaining the global body cap and bounded file reads. The full minimum-version serving suite had 85 passes and one existing ASGI 2.4 streaming-disconnect failure, also reproduced on unchanged main (3 passes / 1 failure in that file). Current declared serving stack passes all 146 serving/transport tests.
• Source quality: 307 passed; website inventory tests: 14 passed; Node 22 production build passed using an isolated cache/output directory.
• Published AIPerf 0.13.0: 50/50 measured requests plus 5/5 warmups. Every transcript matched the same input's native output exactly; zero client failures, unmatched IDs, or missing metric exports. Ten separate HTTP/native verification calls were recorded before the client run and excluded from its measurements.

Pilot	HTTP p50 / p95 (ms)	Public Task p50 / p95 (ms)	Requests/s	Audio seconds/s	Client / Task RTF p50
Whisper tiny, FP16, complete recorded clips, concurrency 1	67.99 / 199.53	66.01 / 196.83	11.52	105.42	0.00975 / 0.00937

Throughput uses the measured client request span. Audio seconds/s divides the sum of measured clip durations (457.625 seconds) by that span. Per-request RTF divides client latency or Task-call time in seconds by actual clip duration. Build/load/startup, verification, and warmup are excluded. Prototype runs are excluded from performance reporting.

Hardware, Environment, and Revisions

Current GPU serving/reference environment: PyTorch 2.12.0+cu130, Transformers 5.2.0, FastAPI 0.141.1, Pydantic 2.14.1, Uvicorn 0.52.4, python-multipart 0.0.32; published client AIPerf 0.13.0, preparation SoundFile 0.13.1.

One NVIDIA GB300; Linux aarch64; TensorRT 11.1.0.106; CUDA toolkit 13.3; driver 580.105.08; Python 3.12.3. Checkpoint openai/whisper-tiny revision 169d4a4341b33bc18d8881c4b69c2e104e1cc0af, FP16 with the profile's FP32 encoder selector 0 and family-owned FP32 normalization/GELU arithmetic, FP32 output projection, TF32 disabled for the FP32 encoder and decoder output boundary, decoder context 128, output limit 120, one persistent replica. LibriSpeech openslr/librispeech_asr revision 71cacbfb7e2354c4226d01e70d77d5fca3d04ba1, test-clean IDs 6930-75918-0000 through 0009; complete clips range from 3.315 to 23.315 seconds. Reference uses the same checkpoint/inputs at its profile-declared FP32 precision. Bundle SHA-256: 4b79f533903ca8b0a060298c427effe3c3e9fab640f09a7fb66c115647e4f9d1.

Not Run / Remaining Gaps

Other ASR models, stereo and non-WAV input, streaming, translations, audio QA, and capacity sweeps are outside this recorded-clip pilot and have not been qualified. TP2/TP4 and Whisper BF16 were not rerun. The existing ASGI 2.4 disconnect test fails on the old minimum serving stack on main as well; this stage does not modify streaming behavior. ASR-specific minimum-version tests pass.

Contributor Self-Review

• I have completed a self-review of this change.

I reviewed family isolation, declared capabilities/configuration, PCM identity, complete outputs, request/resource lifetime, errors, timing boundaries, package/checkpoint/data pins, and evidence scope.

Notes For Future Readers

Review the upload lifetime/cancellation boundary, strict multipart schema, native Task dispatch, and input/output identity checks first. This stage qualifies one recorded-clip pilot. Other ASR models, codecs, stereo input, streaming, translated speech, audio QA, and capacity sweeps require separate evidence. The 50-request run is an integration example, not a statistical performance gate; it establishes no incremental TTFT/ITL or parallel GPU batching. Generated reports, clips, bundles, and environment state are excluded from this PR.

Risk level

• Low
• Medium
• High

Risk: medium. The change adds an upload endpoint, resource cleanup, and family-local Whisper activation/FP32 encoder/logit corrections requiring bundle rebuilds; native/HTTP regressions and real family/client qualification cover those paths. Merge requires all current-head checks, including the protected internal premerge gate.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-Model-Connect/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 8872ccd8-372d-4e06-8225-30a440ca0209

📥 Commits

Reviewing files that changed from the base of the PR and between 59a3a80 and c907f17.

📒 Files selected for processing: 1

• families/whisper/decoder_tp_builder.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

📝 Summary

Summary

Adds prerecorded-audio transcription at /v1/audio/transcriptions. The endpoint accepts one mono PCM16 WAV, validates its audio metadata and duration, and dispatches the audio_transcription capability to a native worker. Legacy ITranscription and SDK SpeechTranscription Tasks are supported. Responses can use JSON or plain text.

The server rejects unsupported controls and oversized audio. It retains the uploaded file until native work completes, including cancellation and worker retirement. Request records include audio metadata and Task-call timing.

Adds a LibriSpeech input-preparation helper and an AIPerf guide. The linked issue reports qualification with Whisper Tiny, ten pinned clips, five warmups, and 50 measured requests. It reports matching transcript identities and successful WER checks. Reported pilot measurements include 67.70 ms HTTP p50, 65.59 ms Task p50, and 105.49 audio-seconds per second. Streaming ASR, other ASR models, and performance gates remain outside the qualified scope.

Whisper changes use the configured activation, compute GELU in FP32 for half-precision inputs, disable TF32 in specified encoder and decoder paths, and compute output projection logits in FP32 before token selection. Older Whisper bundles may need rebuilding. The linked issue reports no change to the public Task ABI, bundle format, or numerical criteria.

Reported tests include selected Whisper E2E tests, 146 serving and transport tests, and 26 minimum-version ASR tests. Current-head CI and merge remain pending. These are reported results, not independently verified test runs.

Architecture impact

• Family-owned files: families/whisper/ changes activation selection, GELU behavior, TF32 settings, and logits projection. The LibriSpeech helper and AIPerf guide are application and documentation files.
• Changed shared surfaces: The native worker protocol, capability registry, HTTP endpoint, request recording, CLI configuration, and CMake targets change. ServerConfig gains max_audio_seconds. The linked issue reports no change to the Task ABI or bundle format.
• Dependency directions: The native server links trtmc_media_io. The transcription worker test links the server, media I/O, and JSON libraries. Python serving and test extras add python-multipart. The documented AIPerf workflow uses the published package without requiring an AIPerf source checkout at runtime.
• Affected consumers: Native server users can send transcription requests to workers that advertise audio_transcription. Consumers of the shared registry, worker protocol, request-recording code, and server configuration may also be affected.
• Review outcome — HUMAN REVIEW REQUIRED: The shared server and worker changes affect more than the qualified Whisper workload. Available evidence does not resolve compatibility across all existing server bundles and Task implementations. No current review findings or severity counts were supplied, so counts are unavailable. This outcome records unresolved blast radius; it does not establish a violation or prove safety.

The server adds multipart WAV transcription through legacy and SDK native tasks. It validates audio and request fields, records timing, and manages uploaded input through worker completion. The change also adds LibriSpeech preparation and AIPerf qualification materials, and updates Whisper activation and logits calculations.

Changes: Recorded-clip transcription

Layer / File(s)	Summary
Validate and submit transcription requests apps/server/python/trtmc_server/*, apps/server/python/tests/test_transcriptions.py, pyproject.toml	The server accepts multipart WAV requests, validates audio and supported fields, checks worker output, and returns JSON or plain text. It applies an audio-duration limit, records requests, and cleans up uploaded input across completion, cancellation, and failure paths.
Dispatch audio to native transcription tasks apps/server/native_worker.cpp, CMakeLists.txt, apps/server/tests/test_*worker.cpp	The native worker validates WAV input and dispatches legacy and SDK transcription tasks. Worker responses include transcript text, audio sample metadata, and task-call timing. Tests cover audio-task dispatch and error cases.
Prepare and report the AIPerf qualification apps/aiperf_qual/examples/prepare_asr_inputs.py, apps/aiperf_qual/README.md, website/docs/user-guides/profile-transcription-with-aiperf.md, website/sidebars.js	The helper selects eligible LibriSpeech clips and writes WAV files, a manifest, and a selection audit. The guide and pilot report describe the AIPerf run, reported results, timing data, and recorded-clip scope.

Whisper activation and logits behavior

Layer / File(s)	Summary
Build Whisper activations and logits families/whisper/decoder_tp_builder.py, families/whisper/graph_blocks.py, families/whisper/graph_ops.py, families/whisper/model.py, families/whisper/tests/test_e2e.py	Whisper encoder and decoder layers use the configured hidden activation, defaulting to GELU. GELU computes in FP32 for non-FP32 inputs before restoring the input dtype. Logits projection uses FP32 inputs and weights. E2E tests compare GELU output with PyTorch and check logits candidates across precisions.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant TranscriptionEndpoint
  participant UploadedInput
  participant NativeWorker
  participant SpeechTranscription
  Client->>TranscriptionEndpoint: Submit multipart WAV and transcription fields
  TranscriptionEndpoint->>UploadedInput: Store uploaded WAV
  TranscriptionEndpoint->>NativeWorker: Submit WAV path and configuration
  NativeWorker->>SpeechTranscription: Pass decoded audio and sample rate
  SpeechTranscription-->>NativeWorker: Return transcript
  NativeWorker-->>TranscriptionEndpoint: Return transcript, audio metadata, and timing
  TranscriptionEndpoint-->>Client: Return JSON or plain text
  TranscriptionEndpoint->>UploadedInput: Release input when worker request ends

Loading

Merge Risk: ⚪ Minimal · up to c907f

The decoder changes support the intended Whisper numerical behavior, and no actionable regression is indicated in the selected change. Merge risk is minimal.

Pre-merge checks: 6 · 3

Failed checks: 3 warnings

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 77 functions across 18 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Shared Semantic Neutrality		Shared qualification semantics remain model-specific. The new apps/aiperf_qual/examples/prepare_asr_inputs.py:5,43-60 hard-codes LibriSpeech, the clean/test shard, first-eligible selection, mono PCM…	Move prepare_asr_inputs.py and the Whisper-specific pilot settings and evidence into Whisper-owned tooling or a model-specific profile. If the helper remains shared, pass the dataset, selection contract, input constraints, and validation …
Benchmark Validation Integrity		The prior evidence gap remains. The PR changes the regular decoder to use FP32 logits and configured GELU at families/whisper/model.py:396-419. It changes the tensor-parallel decoder to use the same…	Run the rebuilt Whisper TP2 FP16 and TP4 FP32 native/reference E2E cases. Retain the unchanged transcript and WER gates. Record tensor-parallel size, precision, bundle, logits/GELU validation, and matched task/sample units. If GPU execution…

Passed checks: 6 passed

Check name	Status	Explanation
Linked Issues check		Issue #1670 requires one complete recorded-clip ASR workload through the native server. The PR adds multipart /v1/audio/transcriptions handling for mono PCM16 WAV files, validates audio metadata, di…
Out of Scope Changes check		The changes stay within issue #1670. Server, native worker, upload, dependency, test, qualification, and documentation changes support recorded-clip transcription. The Whisper GELU, TF32, and FP32 pro…
Family Ownership Boundary		PASS. The PR changes only families/whisper under the model-family tree. The changed Whisper imports remain intra-family (families/whisper/model.py:31-44 and graph_blocks.py:30), and the new vali…
Shared Change Blast Radius		The PR identifies a concrete model-agnostic need: the existing server must expose declared audio transcription capabilities through a common multipart endpoint and worker control plane. The descriptio…
Title check		The title clearly and concisely describes the primary change: serving recorded audio through the server with AIPerf.
Description check		The description completes the required template sections. It documents motivation, exit criteria, implementation, change categories, validation commands and results, environment, remaining gaps, self-…

Full details: Shared Semantic Neutrality

Explanation

Shared qualification semantics remain model-specific. The new apps/aiperf_qual/examples/prepare_asr_inputs.py:5,43-60 hard-codes LibriSpeech, the clean/test shard, first-eligible selection, mono PCM16 input, and a 30-second limit. Its audit also records transcription-specific text, audio, and hash evidence. The changed apps/aiperf_qual/README.md:360,372-382 adds Whisper profile settings, WER results, gold-corpus evidence, and Whisper numerical behavior to shared qualification coverage. The existing shared dataset plugin supplies only the LibriSpeech revision and generic loader (datasets.py:16-23, plugins.yaml:19-31); it does not supply the selection or validation contract. These changes violate shared semantic neutrality and the prior failure remains unresolved.

Resolution

Move prepare_asr_inputs.py and the Whisper-specific pilot settings and evidence into Whisper-owned tooling or a model-specific profile. If the helper remains shared, pass the dataset, selection contract, input constraints, and validation criteria as explicit family-supplied inputs. Remove fixed LibriSpeech, Whisper, WER, profile, and numerical evidence from shared qualification code and coverage prose.

Full details: Benchmark Validation Integrity

Explanation

The prior evidence gap remains. The PR changes the regular decoder to use FP32 logits and configured GELU at families/whisper/model.py:396-419. It changes the tensor-parallel decoder to use the same behavior at families/whisper/decoder_tp_builder.py:420-444. The added tests exercise only graph_ops.add_activation and graph_ops.add_logits_projection (families/whisper/tests/test_e2e.py:521-595); they do not build or run either decoder consumer. The TP2 and TP4 manifests are unchanged, and no changed test evidence covers their native/reference transcript or WER contracts. This violates the requirement for evidence for affected consumers.

Resolution

Run the rebuilt Whisper TP2 FP16 and TP4 FP32 native/reference E2E cases. Retain the unchanged transcript and WER gates. Record tensor-parallel size, precision, bundle, logits/GELU validation, and matched task/sample units. If GPU execution is unavailable, add builder-level regressions that exercise both decoder paths and document the remaining native/reference limitation.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}