Skip to content

feat(schema): skinny configs + techniques per measurement (MTP-ready) - #1

Merged
aistackdev merged 3 commits into
masterfrom
feat/skinny-configs-techniques
May 26, 2026
Merged

feat(schema): skinny configs + techniques per measurement (MTP-ready)#1
aistackdev merged 3 commits into
masterfrom
feat/skinny-configs-techniques

Conversation

@aistackdev

Copy link
Copy Markdown

Summary

  • configs slims down: drops spec_method. The table now describes a deployment (model × hw × framework × precision × parallelism). Natural key shrinks 17 → 16 columns.
  • Per-measurement techniques jsonb is added to benchmark_results and eval_results. It's a free-form bag for runtime knobs — first two keys are spec_method (string) and mtp_layers (number); future entries (kv_cache_dtype, chunked_prefill, prefix_cache, …) land here without further DDL.
  • Gemma4 MTP regression fix falls out for free: the benchmark side encodes layer count in infmax_model_prefix (gemma4n4 / gemma4n6). Mapper now collapses those to model='gemma4' and writes techniques.mtp_layers ∈ {4,6}. The 40 rows that silently skipped on the 2026-05-25 ingest will land on the next dispatch.

Design background and reasoning are in design/database.md and design/intent-and-users.md; the same-day brainstorm in chat covered the fork-in-the-road (config dimension vs measurement metadata) — we picked measurement metadata.

Out of scope (PR2)

  • UI filter chip for techniques.mtp_layers on /inference. Today the existing spec-method filter still works because every query derives spec_method server-side from techniques->>'spec_method'.
  • New y-axis entries (median_acceptance_rate, median_tokens_per_step) in inference-chart-config.json. Metric keys are registered in constants; chart config wiring deferred.
  • Lifting availability.spec_method denormalization (kept as a projection for date-picker ergonomics).

Migration mechanics

packages/db/migrations/006_skinny_configs.sql is a full-rewrite migration:

  1. Alter configs (drop column, drop+recreate natural-key UNIQUE).
  2. Add techniques jsonb default '{}' to benchmark_results and eval_results.
  3. Add expression indexes on techniques->>'spec_method' and techniques->>'mtp_layers'.
  4. Drop + recreate latest_benchmarks materialized view with techniques in row identity (otherwise two formerly-different configs sharing a new config_id would collide on the MV's PK).
  5. Drop + recreate benchmark_results_seq_history_idx to include techniques.
  6. TRUNCATE all data tables. Safe for our fork because the existing data is yesterday's failed gemma4 ingest + a handful of smoke runs; ingest is idempotent so a re-dispatch from auto-ingest restores everything cleanly. CI deploy will run the migration on master and the next auto-ingest.yml tick will repopulate.

Test plan

  • pnpm typecheck clean
  • pnpm test:unit — DB / mcp / constants 100%; app 1913/1914 (only pre-existing visit-tracking.test.ts timezone flake fails)
  • pnpm lint && pnpm fmt clean
  • On staging DB: run pnpm admin:db:migrate --yes --no-ssl, then pnpm admin:db:ingest:run https://github.com/vngcloud/InferenceX/actions/runs/26387431571 vngcloud/InferenceX --no-ssl. Verify:
    • SELECT model, count(*) FROM configs WHERE model='gemma4' GROUP BY model → one row
    • SELECT techniques, count(*) FROM benchmark_results GROUP BY techniques → buckets for {spec_method:mtp, mtp_layers:4} and {spec_method:mtp, mtp_layers:6}
    • No [SKIP] unmapped model lines in the ingest log for gemma4
  • Spot-check the live dashboard after deploy: /inference shows gemma4 series; existing spec-method filter chip still works.

Follow-ups

  • PR2: techniques filter chips on /inference + new chart-config metric entries
  • Ship the matching artifact-side change on vngcloud/InferenceX so future runs emit techniques: {spec_method, mtp_layers} as a top-level object instead of relying on the legacy spec_decoding + prefix-encoded fallback path (which still works today)

🤖 Generated with Claude Code

Ngô Quang Hòa and others added 3 commits May 26, 2026 11:04
`configs` now describes the deployment (model, hw, framework, precision,
parallelism). Per-run knobs (spec_method, mtp_layers, future
kv_cache_dtype, chunked_prefill, prefix_cache, …) live in a new
`techniques` jsonb on benchmark_results and eval_results, so we can
absorb new dimensions without DDL or natural-key bloat.

Motivated by the gemma4 MTP work: the benchmark side encoded the MTP
layer count in the model prefix (gemma4n4 / gemma4n6), which never
matched a model key and silently dropped 40 rows on yesterday's ingest.
With techniques.mtp_layers as a measurement-level dimension, the model
collapses to gemma4 and the layer count rides on the row.

Schema (migration 006):
  - configs drops spec_method (column + lowercase check + natural key)
  - benchmark_results + eval_results gain `techniques jsonb default {}`
  - latest_benchmarks MV adds techniques to row identity to keep
    same-config-different-technique pairs distinct
  - Expression indexes on techniques->>'spec_method' and ->>'mtp_layers'
  - Full-rewrite truncate: re-ingest from artifacts after deploy

ETL:
  - parseTechniques() reads new-shape `techniques` object, falls back to
    legacy `spec_decoding` field, infers mtp_layers from gemma4n4/n6
    prefix aliases
  - benchmark-mapper / eval-mapper drop specMethod from ConfigParams,
    add techniques to BenchmarkParams / EvalParams
  - config-cache key is now 16-col (down from 17)
  - bulk insert / upsert paths write techniques; availability
    denormalizes spec_method via the same projection

Queries:
  - All five query files SELECT COALESCE(br.techniques->>'spec_method',
    'none') as spec_method + raw techniques jsonb. Downstream readers
    keep working unchanged because the projection name is preserved.

Constants: add `gemma4` model key, register `*_acceptance_rate` and
`*_tokens_per_step` metric keys for upcoming spec-dec measurements.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lib/api.ts BenchmarkRow + EvalRow gain a required `techniques` field
mirroring what the API actually returns now. spec_method stays as a
derived projection so every downstream consumer (charts, CSV export,
TCO calculator, eval drawer, …) keeps working unchanged.

unofficial-run/route.ts (artifact ZIP overlay path) reads spec_method
from params.techniques.spec_method and carries the raw techniques jsonb
through, matching the SQL row shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DB-side mappers / config-cache:
  - benchmark-mapper: techniques.spec_method assertions, plus a new test
    for the gemma4n4/n6 → mtp_layers inference path and one for the
    new-shape `techniques` object on the artifact
  - eval-mapper: techniques.spec_method assertion
  - config-cache: 16-col key, updated golden string

App-side mocks:
  - compare-pair-defaults, benchmark-transform, evaluation/chart-data,
    unofficial-run-provider: add `techniques: {}` to BenchmarkRow /
    EvalRow stubs so they satisfy the updated row shape

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@aistackdev
aistackdev marked this pull request as ready for review May 26, 2026 04:19
@aistackdev
aistackdev merged commit 175c394 into master May 26, 2026
9 of 15 checks passed
@aistackdev
aistackdev deleted the feat/skinny-configs-techniques branch May 26, 2026 04:19
aistackdev added a commit that referenced this pull request May 26, 2026
PR #1 left benchmark_results_unique and eval_results_unique unchanged
when it demoted spec_method to a per-measurement jsonb. With two MTP
variants (mtp_layers=4 vs 6) now sharing one config_id, rows with the
same (workflow_run, config, isl, osl, conc) but different techniques
collided on the existing constraint and the later insert overwrote the
earlier one. First post-deploy ingest of gemma4 run 26387431571 landed
6 of ~24 expected rows for exactly this reason.

Migration 007 adds techniques to both unique constraints and re-
truncates so re-ingest from scratch is clean. The bulk insert paths
also need techniques in their ON CONFLICT clause and in their in-batch
dedup discriminator — fixed in benchmark-ingest.ts and eval-ingest.ts.

Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aistackdev added a commit that referenced this pull request May 26, 2026
PR #1 added `gemma4: 'Gemma-4-31B-it'` to DB_MODEL_TO_DISPLAY in
constants/, but missed the parallel registration in the app's own
Model enum + MODEL_CONFIG (data-mappings.ts). Result: MODEL_OPTIONS
doesn't include Gemma-4-31B-it, so:

  GlobalFilterContext.availableModels =
    MODEL_OPTIONS.filter(... matches availability ...)
  // → [] because gemma4 isn't in MODEL_OPTIONS to begin with

The PR #4 fallback effect saw `availableModels.length === 0` and
bailed out, leaving the user stuck on the hardcoded DeepSeek_R1
default which has no data → "No data for DeepSeek-R1-0528" empty
state on /evaluation.

Add Gemma_4_31B to the enum + MODEL_CONFIG so the model surfaces in
the dropdown and the fallback effect can land on it.

Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aistackdev pushed a commit that referenced this pull request Jul 15, 2026
Pin MiniMax first and OpenAI second in the landing page carousel so
MiniMax is the quote shown on initial load. Also reorder the
CAROUSEL_ORGS whitelist (MiniMax, then OpenAI) so the FAQ supporter
listing stays consistent.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant