fix(techniques): rename mtp_layers → num_speculative_tokens, add max_num_batched_tokens - #5
Merged
Conversation
…num_batched_tokens
The benchmark artifact on vngcloud/InferenceX renamed the per-row layer
count field to the canonical vLLM/SGLang name `num_speculative_tokens`
(now applies to any draft method, not just MTP). It also gained a new
top-level `max_num_batched_tokens` field. Mirroring both on the
dashboard side.
ETL (normalizers.ts):
- TECHNIQUE_KEYS picks up num_speculative_tokens + max_num_batched_tokens
- parseTechniques now reads top-level fields too (the new artifact emits
them flat, not nested under a `techniques` object); nested-object form
is also accepted for forward-compat
- PREFIX_TO_NUM_SPEC_TOKENS replaces PREFIX_TO_MTP_LAYERS for the legacy
gemma4n4/n6 fallback path
Migration 008:
- Drop benchmark_results_mtp_layers_idx, add expression indexes for
num_speculative_tokens and max_num_batched_tokens
- TRUNCATE so re-ingest starts under the new shape
UI (recipe-data.ts):
- describeTechniques renders `MTP×6` / `EAGLE×3` based on
techniques.num_speculative_tokens
Tests updated. db 214/214 + app recipe-data 8/8 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aistackdev
pushed a commit
that referenced
this pull request
Jul 15, 2026
…te (SemiAnalysisAI#383) * feat(compare-per-dollar): add cost-per-million-tokens SSR sibling to /compare Adds /compare-per-dollar/<model>-<a>-vs-<b> as a perf-per-dollar view of /compare/<...>, sharing all slug parsing, alias resolution, availability filtering, redirect machinery, and benchmark cache slots. Each per-dollar page differs only in: - Comparison table shows only Cost ($/M tok) + Concurrency rows (CompareInterpolatedTable now takes optional visibleMetricLabels prop) - Chart y-axis defaults to y_costh (Cost per Million Total Tokens — Owning Hyperscaler) instead of y_tpPerGpu (InferenceProvider now takes optional initialYAxisMetric prop) - Title/header/metadata/OG/JSON-LD use "Performance per Dollar" framing with SEO terms: performance per dollar, performance normalized by cost, dollars per million tokens, owning-hyperscaler TCO Both routes cross-link to each other in the slug-page header so navigation flows both ways and search engines crawl the topical pair densely. Shared server-side helpers (getCachedBenchmarks, summarize, computeCompareTableData, buildJsonLd, KNOWN_MODELS/SEQUENCES/PRECISIONS validators) moved out of compare/[slug]/page.tsx into @/lib/compare-ssr — the cache slot is keyed on dbKeys, so /compare and /compare-per-dollar for the same model hit the same blob entry (no duplicate fetches). buildJsonLd takes a 'full' | 'per-dollar' variant that swaps ItemList/Dataset name+description. Sitemap jumps from 184 to 350 URLs (165 /compare + 165 /compare-per-dollar + 1 standalone /compare-per-dollar). Master index uses the same Neon availability filter as /compare so only (model, GPU-pair) combos with benchmark data on both sides show up. 44 unit tests pass; lint, format, typecheck clean. Two new Cypress files cover the redirect chain (mirroring compare-redirect.cy.ts) and the slimmed-table + cross-link behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(compare): enumerate every model name in master-index lede Both /compare and /compare-per-dollar indexes now read "...comparisons across DeepSeek R1, DeepSeek V4 Pro, Kimi K2.5/K2.6, Qwen 3.5, GLM 5/5.1, MiniMax M2.5/M2.7, Llama 3.3 70B, and gpt-oss 120B." instead of a bare count. Search engines see every model name in the indexable description, so the index pages rank for "<model name> GPU benchmark" / "<model name> cost per million tokens" queries instead of just generic terms. Oxford-comma serial join via a small formatModelList helper colocated in each index page. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(compare-per-dollar): add "Per Dollar" link to top nav Slots after "Comparisons" as a sibling rather than a sub-item — same priority class, distinct destination. Also tightens isActive so a path that merely starts with the literal string `/compare` doesn't light up the Comparisons link when the user is actually on `/compare-per-dollar` (both nav items were highlighting before). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(compare-per-dollar): rename Cost row to "Dollar per Million Tokens" The /compare-per-dollar table now reads "Dollar per Million Tokens" instead of the terser "Cost ($/M tok)" — matches the page's "Performance per Dollar" framing and surfaces the SEO term verbatim inside the rendered HTML. Implementation: new optional `metricLabelOverrides?: Record<string, string>` prop on CompareInterpolatedTable, keyed by the metric's internal label. The /compare side omits the prop and keeps the terse "Cost ($/M tok)" label that fits its four-row table; the per-dollar page passes { 'Cost ($/M tok)': 'Dollar per Million Tokens' }. Cypress test updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(compare): SSR narrative paragraph + H1 framing + master index order Three coordinated changes: 1. **Plain-English narrative above each table** — compareTableNarrative() in @/lib/compare-ssr picks the first interactivity target where both GPUs have data, builds a 2-3 sentence summary describing throughput, cost, and the headline ratio at that operating point, and points the reader to the table for the broader Pareto picture. Two templates per route: - /compare: "On {model}, at {target} tok/s/user (the middle of {min}–{max}), {a} delivers {tput} tok/s/GPU at ${cost}, while {b} delivers {tput}/{cost}. {cheaper} is N% cheaper per token; {faster} delivers N% more tok/s/GPU..." - /compare-per-dollar: "On {model}, {a} costs ${cost} at {target} tok/s/user interactivity; {b} costs ${cost}. {cheaper} is N% more cost-efficient at this point..." Handles single-GPU-only and tied-cost edge cases. Server-rendered, indexable text — passed through as a `narrative` prop to each route's page-client, rendered as a <p> inside the header card. 2. **/compare-per-dollar H1 includes "Performance per Dollar"** — the <h1> now reads "DeepSeek V4 Pro — GB300 NVL72 vs MI355X Performance per Dollar" (was: just the model + GPU pair, with the framing only in the small eyebrow). The H1 is what search engines weight heaviest; the SEO term now lives there too. 3. **Master index model order** — COMPARE_MODEL_SLUGS reordered to the product-spec sequence: DeepSeek V4 Pro → DeepSeek R1 → Kimi K2.5/K2.6 → GLM 5/5.1 → MiniMax M2.5/M2.7 → Qwen 3.5 → gpt-oss 120B → Llama 3.3 70B. Qwen slots between MiniMax and gpt-oss to keep the Chinese-developed open-model cluster contiguous before the US transition. Both /compare and /compare-per-dollar index sections, sitemap, lede enumerations, and generateStaticParams reorder automatically. 32/32 unit tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(compare-per-dollar): show $/GPU/hr per SKU + cite SemiAnalysis TCO source Adds a small caption inside the per-dollar slug page header that prints the owning-hyperscaler $/GPU/hr for each compared GPU, pulled from the existing HW_REGISTRY.costh value the per-dollar math already uses (no duplicate pricing source). Followed by an attribution link: "GPU pricing (owning hyperscaler): GB300 NVL72 $2.65/GPU/hr · MI355X $1.48/GPU/hr. Source: SemiAnalysis Market August 2025 Pricing Surveys & AI Cloud TCO Model." The source phrase links to https://newsletter.semianalysis.com/p/ai-cloud-economics (the canonical TCO Model doc). target="_blank" + rel="noopener noreferrer"; analytics event compare_per_dollar_tco_source_clicked fires on click. Gracefully degrades to "—" if HW_REGISTRY has no costh entry for a GPU (falls back to DEFAULT_SPECS' zero, and the caption hides entirely when both are zero). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(compare): TCO source link + Bugbot review fixes (4 findings) 1. Source link in per-dollar header now points to the canonical TCO Model doc: https://semianalysis.com/ai-cloud-tco-model/ (was the newsletter URL). 2. Narrative no longer mislabels the picked row as "the middle of" the interactivity range — we pick the first row with data on both sides, which is often not the mid index. Reworded to "within the <range> benchmarked" so the claim matches the selection logic. 3. Narrative now guards against zero costs (HW_REGISTRY.costh missing or zero throughput upstream): the ratio division would emit Infinity/NaN and fmtPctDelta would render junk. Falls through to a values-only sentence when either side is non-positive. 4. 'full' narrative now handles near-ties on cost AND throughput independently (mirrors the 1% tie-detection already in 'per-dollar'). Previously could emit "X is 0% cheaper per token" with the wrong GPU labeled as winner when costs were equal. 5. (Bugbot #4) Extracted bucketComparePairsByVendor() and formatModelList() to @/lib/compare-ssr so /compare and /compare-per-dollar share the sort+bucket+slug-building logic. Each route keeps its own group description strings (cost-per-token framing differs intentionally between the two indexes) and href prefix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(compare): clarify narrative is anchored to slug defaults (Bugbot #5) Bugbot flagged that the SSR narrative doesn't recompute when a reader changes model/sequence/precision client-side (the table and chart do). Making the narrative client-reactive would require duplicating the interpolation pipeline on the client; the SSR text is also the SEO payload and must remain server-rendered. Mitigate with an inline italicized caveat appended to the narrative on both routes: "(Numbers reflect the default <sequence> · <precision> selection for this URL — table and chart below update if you change sequence, precision, or model in the controls.)" — sets reader expectations without losing the indexable prose. JSDoc on compareTableNarrative documents the design choice. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
You're shipping a new artifact format on
vngcloud/InferenceX:infmax_model_prefix: gemma4n4/gemma4n6infmax_model_prefix: gemma4(unified)num_speculative_tokens: 6(top-level int)max_num_batched_tokens: 4096(top-level int | null)This PR mirrors the rename on the dashboard side and adds plumbing for both new fields. No schema change —
techniquesjsonb absorbs them by name.Changes
packages/db/src/etl/normalizers.ts—TECHNIQUE_KEYSpicks upnum_speculative_tokens+max_num_batched_tokens.parseTechniquesnow reads flat top-level fields (forward-compat: nestedtechniques: {}object also accepted). Legacygemma4n4/gemma4n6prefix aliases still producetechniques.num_speculative_tokensso historical artifacts re-ingest correctly.benchmark_results_mtp_layers_idx, adds expression indexes for the renamed key + the newmax_num_batched_tokens. TRUNCATEs so re-ingest under the new shape is clean.recipe-data.ts— variant label rendersMTP×6/EAGLE×3fromtechniques.num_speculative_tokens. Spec method short codes (≤6 chars) → ALL CAPS.Test plan
pnpm typecheckcleanpnpm lint && pnpm fmtcleantechniques->>'num_speculative_tokens'shows4/6for the two MTP variants, andmax_num_batched_tokenspopulates if the artifact sets it.🤖 Generated with Claude Code