feat(schema): skinny configs + techniques per measurement (MTP-ready) - #1
Merged
Conversation
`configs` now describes the deployment (model, hw, framework, precision,
parallelism). Per-run knobs (spec_method, mtp_layers, future
kv_cache_dtype, chunked_prefill, prefix_cache, …) live in a new
`techniques` jsonb on benchmark_results and eval_results, so we can
absorb new dimensions without DDL or natural-key bloat.
Motivated by the gemma4 MTP work: the benchmark side encoded the MTP
layer count in the model prefix (gemma4n4 / gemma4n6), which never
matched a model key and silently dropped 40 rows on yesterday's ingest.
With techniques.mtp_layers as a measurement-level dimension, the model
collapses to gemma4 and the layer count rides on the row.
Schema (migration 006):
- configs drops spec_method (column + lowercase check + natural key)
- benchmark_results + eval_results gain `techniques jsonb default {}`
- latest_benchmarks MV adds techniques to row identity to keep
same-config-different-technique pairs distinct
- Expression indexes on techniques->>'spec_method' and ->>'mtp_layers'
- Full-rewrite truncate: re-ingest from artifacts after deploy
ETL:
- parseTechniques() reads new-shape `techniques` object, falls back to
legacy `spec_decoding` field, infers mtp_layers from gemma4n4/n6
prefix aliases
- benchmark-mapper / eval-mapper drop specMethod from ConfigParams,
add techniques to BenchmarkParams / EvalParams
- config-cache key is now 16-col (down from 17)
- bulk insert / upsert paths write techniques; availability
denormalizes spec_method via the same projection
Queries:
- All five query files SELECT COALESCE(br.techniques->>'spec_method',
'none') as spec_method + raw techniques jsonb. Downstream readers
keep working unchanged because the projection name is preserved.
Constants: add `gemma4` model key, register `*_acceptance_rate` and
`*_tokens_per_step` metric keys for upcoming spec-dec measurements.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lib/api.ts BenchmarkRow + EvalRow gain a required `techniques` field mirroring what the API actually returns now. spec_method stays as a derived projection so every downstream consumer (charts, CSV export, TCO calculator, eval drawer, …) keeps working unchanged. unofficial-run/route.ts (artifact ZIP overlay path) reads spec_method from params.techniques.spec_method and carries the raw techniques jsonb through, matching the SQL row shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DB-side mappers / config-cache:
- benchmark-mapper: techniques.spec_method assertions, plus a new test
for the gemma4n4/n6 → mtp_layers inference path and one for the
new-shape `techniques` object on the artifact
- eval-mapper: techniques.spec_method assertion
- config-cache: 16-col key, updated golden string
App-side mocks:
- compare-pair-defaults, benchmark-transform, evaluation/chart-data,
unofficial-run-provider: add `techniques: {}` to BenchmarkRow /
EvalRow stubs so they satisfy the updated row shape
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aistackdev
marked this pull request as ready for review
May 26, 2026 04:19
2 tasks
aistackdev
added a commit
that referenced
this pull request
May 26, 2026
PR #1 left benchmark_results_unique and eval_results_unique unchanged when it demoted spec_method to a per-measurement jsonb. With two MTP variants (mtp_layers=4 vs 6) now sharing one config_id, rows with the same (workflow_run, config, isl, osl, conc) but different techniques collided on the existing constraint and the later insert overwrote the earlier one. First post-deploy ingest of gemma4 run 26387431571 landed 6 of ~24 expected rows for exactly this reason. Migration 007 adds techniques to both unique constraints and re- truncates so re-ingest from scratch is clean. The bulk insert paths also need techniques in their ON CONFLICT clause and in their in-batch dedup discriminator — fixed in benchmark-ingest.ts and eval-ingest.ts. Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
aistackdev
added a commit
that referenced
this pull request
May 26, 2026
PR #1 added `gemma4: 'Gemma-4-31B-it'` to DB_MODEL_TO_DISPLAY in constants/, but missed the parallel registration in the app's own Model enum + MODEL_CONFIG (data-mappings.ts). Result: MODEL_OPTIONS doesn't include Gemma-4-31B-it, so: GlobalFilterContext.availableModels = MODEL_OPTIONS.filter(... matches availability ...) // → [] because gemma4 isn't in MODEL_OPTIONS to begin with The PR #4 fallback effect saw `availableModels.length === 0` and bailed out, leaving the user stuck on the hardcoded DeepSeek_R1 default which has no data → "No data for DeepSeek-R1-0528" empty state on /evaluation. Add Gemma_4_31B to the enum + MODEL_CONFIG so the model surfaces in the dropdown and the fallback effect can land on it. Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aistackdev
pushed a commit
that referenced
this pull request
Jul 15, 2026
Pin MiniMax first and OpenAI second in the landing page carousel so MiniMax is the quote shown on initial load. Also reorder the CAROUSEL_ORGS whitelist (MiniMax, then OpenAI) so the FAQ supporter listing stays consistent. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
configsslims down: dropsspec_method. The table now describes a deployment (model × hw × framework × precision × parallelism). Natural key shrinks 17 → 16 columns.techniques jsonbis added tobenchmark_resultsandeval_results. It's a free-form bag for runtime knobs — first two keys arespec_method(string) andmtp_layers(number); future entries (kv_cache_dtype,chunked_prefill,prefix_cache, …) land here without further DDL.infmax_model_prefix(gemma4n4/gemma4n6). Mapper now collapses those tomodel='gemma4'and writestechniques.mtp_layers ∈ {4,6}. The 40 rows that silently skipped on the 2026-05-25 ingest will land on the next dispatch.Design background and reasoning are in
design/database.mdanddesign/intent-and-users.md; the same-day brainstorm in chat covered the fork-in-the-road (config dimension vs measurement metadata) — we picked measurement metadata.Out of scope (PR2)
techniques.mtp_layerson/inference. Today the existing spec-method filter still works because every query derivesspec_methodserver-side fromtechniques->>'spec_method'.median_acceptance_rate,median_tokens_per_step) ininference-chart-config.json. Metric keys are registered in constants; chart config wiring deferred.availability.spec_methoddenormalization (kept as a projection for date-picker ergonomics).Migration mechanics
packages/db/migrations/006_skinny_configs.sqlis a full-rewrite migration:configs(drop column, drop+recreate natural-key UNIQUE).techniques jsonb default '{}'tobenchmark_resultsandeval_results.techniques->>'spec_method'andtechniques->>'mtp_layers'.latest_benchmarksmaterialized view withtechniquesin row identity (otherwise two formerly-different configs sharing a newconfig_idwould collide on the MV's PK).benchmark_results_seq_history_idxto includetechniques.auto-ingest.ymltick will repopulate.Test plan
pnpm typecheckcleanpnpm test:unit— DB / mcp / constants 100%; app 1913/1914 (only pre-existingvisit-tracking.test.tstimezone flake fails)pnpm lint && pnpm fmtcleanpnpm admin:db:migrate --yes --no-ssl, thenpnpm admin:db:ingest:run https://github.com/vngcloud/InferenceX/actions/runs/26387431571 vngcloud/InferenceX --no-ssl. Verify:SELECT model, count(*) FROM configs WHERE model='gemma4' GROUP BY model→ one rowSELECT techniques, count(*) FROM benchmark_results GROUP BY techniques→ buckets for{spec_method:mtp, mtp_layers:4}and{spec_method:mtp, mtp_layers:6}[SKIP] unmapped modellines in the ingest log for gemma4/inferenceshows gemma4 series; existing spec-method filter chip still works.Follow-ups
/inference+ new chart-config metric entriesvngcloud/InferenceXso future runs emittechniques: {spec_method, mtp_layers}as a top-level object instead of relying on the legacyspec_decoding+ prefix-encoded fallback path (which still works today)🤖 Generated with Claude Code