Name the model rungs for the job, and make Auto actually dispatch by rbuergi · Pull Request #992 · Systemorph/MeshWeaver

rbuergi · 2026-08-09T07:39:50Z

One tiering system, named for the job

Two half-finished mechanisms sat where one belonged: agents asked for a tier through the ModelTier__{Heavy,Standard,Light,Utility} config keys, while model nodes carried a parallel Size label (S/M/L/XL). This deletes the size vocabulary and makes the tier the one system.

The tiers, and why these names

A size tells you how big a model is and never when to pick one — you still need the house convention to read L at a call site. Sizes also age: this year's large is next year's medium, so every label drifts while the work it was chosen for stays identical. Coding is still coding.

Tier	Rank	What belongs here
`utility`	0	Cheap background micro-jobs the user never waits for — thread auto-naming, node icons and descriptions, classification, triage.
`chat`	10	Ordinary interactive conversation — the everyday round.
`reasoning`	20	Multi-step analysis, planning, synthesis — the hard asks that are not code.
`coding`	30	Writing and changing code, on the strongest model the deployment has.

coding is the top rung deliberately: it is the one job where a weaker model does not merely answer less well — it produces work that looks finished and is wrong, and that cost lands downstream, on a human, later. utility is the bottom rung and is exactly the auto-naming case.

Tiers are nodes, not an enum. They ship as nodeType:ModelTier under Provider/Tier, listed under Tiers in the AI menu beside Models and Providers, seeded create-if-absent (ExcludeThisAndChildren) so an operator can rename a rung, re-rank the set, rewrite a purpose or add a fifth — no code change, no config key, no redeploy, and the edit survives every update.

Auto is the default, and now actually dispatches

ModelDefinition.IsRouter documented Auto as "inspects the ask and dispatches to a real model", but no dispatcher existed — the flag was only ever read to exclude the entry, and its own doc said Auto was reachable "only ever EXPLICITLY, by a user picking it". There was also no Auto node: nothing emitted one.

Now: BuiltInLanguageModelProvider emits a credential-less Auto provider + router model at Order = -10 (below the -1 "pin the default" convention, so it sorts ahead of a deliberately-pinned default), ModelInfo.IsRouter carries through the projection so the composer's "lowest Order whose credentials resolve" rule does not skip the one entry with no credential of its own, and ValidMasterModel stops resetting a stored Auto selection.

Dispatch rule — stated plainly: Auto runs the tier the agent that serves the round declared. Agent declares none (or none is selected and the default agent declares none) → the deployment default. Nothing usable → the round fails audibly. No classification call, no second model deciding what the first should be: a router nobody can predict is worse than a fixed default, and dispatching on the agent's tier is the reading consistent with "tiers are assigned to agents".

The router check runs before the usability check, because a router parked on a keyed provider still names no wire model — "it has a key" is the wrong reason to let it reach a factory. That ordering is what the discriminating test below pins.

Lookup cannot fail — it degrades

ModelTierCatalog.Resolve: tier label on a model node → deprecated ModelTier:* config → deployment default. Every rung skips models whose credentials do not resolve, and skips routers. An unknown label, an unpopulated tier, no tier at all, a catalog with no labels anywhere, or every tier node deleted each cost a rung, never a round. The single null outcome — an entirely unusable catalog — is reported as ModelTierSource.None so the caller fails loudly instead of running on nothing.

…but falling back stays visible (#961 / #476)

The round already records the model that actually served. The resolution now also carries which rung answered, so an Auto dispatch that had to fall past an unpopulated tier logs a warning naming the tier and telling the operator to label a model — the difference between "my coding agent runs on the coding model" and "…runs on whatever sorts first". A tier that was never requested is not reported as a fallback, so the warning that matters keeps its signal.

Migration: the old config keys are deprecated, not removed

The live memex portal is steered by those keys right now, so they are still read — below the node labels, and only ever for a tier no model node carries, and only when they name a model whose credentials resolve. They map by rank (Heavy→coding, Standard→reasoning, Light→chat, Utility→utility) through each tier's aliases rather than a hard-coded switch, so the bridge survives a rename and an operator can point a new tier at an old key by adding an alias. Legacy heavy/standard/light and S/M/L/XL spellings resolve identically on both the agent's ask and the model node's label.

Nothing a deployment has today changes behaviour. To retire the keys: put tier on the model nodes, confirm no resolution reports LegacyConfig, delete the keys.

Tests

ModelTierCatalogTest (28) — pure functions over the catalog. Every stripped-catalog shape asserts a real model still comes out: no labels, router only, unknown label, empty tier registry, unusable models, empty catalog. Plus alias equivalence in both directions and the three legacy-config rungs.
AutoRouterDispatchTest (3) — real rounds through ThreadExecution over a real seeded catalog. Auto selected → the round records a real, usable, non-router model; every tier shape resolves to a usable non-router model off the live snapshot; and a declared tier beats the pinned default while an unpopulated one degrades and reports it.
BuiltInLanguageModelProviderTest (+6) — Auto ships with no catalog source configured at all, carries no credential, outranks even a pinned -1, does not collide with a provider model called auto; the shipped tiers arrive as create-if-absent nodes with coding on top.

Verified the key test bites. With the router check disabled (isRouter = false, i.e. the pre-change behaviour) ARouterOnAKeyedProviderIsStillDispatched_NotServed fails — the round runs on the router's own id instead of dispatching. Restored and re-verified green.

Also green locally: DocumentationLinkIntegrityTest + embed integrity (11), LocalizationTest (33 — both new menu keys present in en and de), and -c Release -warnaserror clean on MeshWeaver.AI, MeshWeaver.AI.Test, MeshWeaver.Documentation.Test, MeshWeaver.Messaging.Hub.Test, Memex.Portal.Shared.

🤖 Generated with Claude Code

Two half-finished mechanisms sat where one belonged. Agents asked for a tier through `ModelTier__{Heavy,Standard,Light,Utility}` config keys, while model nodes carried a parallel `Size` label (S/M/L/XL) — and a size says nothing about when to pick one. Sizes also age: this year's large is next year's medium, so every label drifts while the work it was chosen for stays the same. Tiers are now named for the JOB, and they are DATA: utility (0) background micro-jobs — thread auto-naming, icons, triage chat (10) the everyday interactive round reasoning (20) multi-step analysis and planning that is not code coding (30) writing and changing code, on the strongest model there is `coding` is the top rung because code is where a weaker model costs the most: a wrong patch looks finished and lands on a human later. They ship as ordinary `nodeType:ModelTier` nodes under `Provider/Tier`, listed in the AI menu beside Models and Providers, seeded create-if-absent — so a deployment can rename a rung, re-rank the set or add a fifth with no code change and no config key. Auto was documented as a router that "inspects the ask and dispatches", but `IsRouter` was only ever read to EXCLUDE the entry, and its own doc said Auto was reachable "only ever EXPLICITLY, by a user picking it". Nothing dispatched. It is now the default selection for a new thread and it dispatches on the tier the agent that serves the round declared — the deployment default when it declares none. No classification call: a router nobody can predict is worse than a fixed default. The router check runs BEFORE the usability check, because a router parked on a keyed provider still names no wire model. Lookup cannot fail, it degrades: tier label on a node → the deprecated config → the deployment default, skipping anything whose credentials do not resolve and skipping routers at every rung. An unknown label, an unpopulated tier, no labels at all, or every tier node deleted each cost a rung, never a round. The one null outcome — an entirely unusable catalog — is reported as `ModelTierSource.None` so the round fails audibly rather than running on nothing. Falling back stays VISIBLE. The round already records the model that actually served; the resolution now also carries WHICH rung answered, so an Auto dispatch that had to fall past an unpopulated tier logs a warning naming the tier and telling the operator to label a model. Silently substituting and saying nothing is the #476 defect. Nothing existing breaks. `heavy`/`standard`/`light` and S/M/L/XL resolve through each tier's aliases, matched via the tier registry rather than a hard-coded switch — so the bridge survives a rename, and an operator can point a new tier at an old config key by adding an alias. The `ModelTier:*` keys are still read, below the node labels, and only for a model whose credentials resolve; they are deprecated, not removed, so a portal cannot lose its model mapping on upgrade. Verified the Auto test bites: with the router check disabled (the pre-change behaviour) `ARouterOnAKeyedProviderIsStillDispatched_NotServed` fails, because the round runs on the router's own id instead of dispatching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR unifies language-model selection around usage tiers (utility/chat/reasoning/coding) instead of size labels, introduces tiers as editable mesh nodes, and makes Auto a first-class router/default selection that reliably dispatches to a concrete model based on the serving agent’s declared tier (with visible degradation/fallback behavior and legacy-config compatibility).

Changes:

Replaces size-based resolution with tier-based resolution (ModelTierCatalog), including fallback/degradation semantics and legacy ModelTier:* config shims.
Seeds Auto (router provider + model) and the tier registry as nodes; updates resolver/composer logic so Auto can be the default selection and still dispatch correctly.
Updates portal UI/menu, docs, helm/AKS guidance, and agent frontmatter; adds/updates tests to pin dispatch and degradation behavior.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/MeshWeaver.AI.Test/ModelTierCatalogTest.cs	New unit coverage for tier resolution, degradation, router exclusion, and legacy mappings.
test/MeshWeaver.AI.Test/AutoRouterDispatchTest.cs	New integration-style tests asserting Auto dispatch and router handling in real rounds.
test/MeshWeaver.AI.Test/ModelSubstitutionTest.cs	Updates warm-up/read path from size candidates to tier candidates.
test/MeshWeaver.AI.Test/ModelSizeCatalogTest.cs	Removes obsolete size-based catalog tests.
test/MeshWeaver.AI.Test/BuiltInLanguageModelProviderTest.cs	Adds coverage for Auto/router emission and tier-node seeding/ordering/projection.
src/MeshWeaver.AI/ModelTierNodeType.cs	Introduces `ModelTier` node type and registration under `Provider/Tier`.
src/MeshWeaver.AI/ModelTierDefinition.cs	Adds tier content model + shipped defaults (with aliases) for fallback behavior.
src/MeshWeaver.AI/ModelTierConfiguration.cs	Refactors legacy `ModelTier:*` config into a deprecated shim mapped via tier aliases.
src/MeshWeaver.AI/ModelTierCatalog.cs	New pure selection logic for tier resolution + degradation + router exclusion.
src/MeshWeaver.AI/ModelSizeCatalog.cs	Removes obsolete size-based catalog resolution implementation.
src/MeshWeaver.AI/ModelSize.cs	Removes obsolete size enum/parsing in favor of tier labels + aliases.
src/MeshWeaver.AI/ModelInfo.cs	Adds `IsRouter` projection so the composer can default-select Auto.
src/MeshWeaver.AI/ModelDefinition.cs	Replaces `Size` with `Tier`; clarifies router semantics; projects `IsRouter` into `ModelInfo`.
src/MeshWeaver.AI/LanguageModelNodeType.cs	Adds router constants (provider/id/path/order) and wires tier node type registration.
src/MeshWeaver.AI/ChatClientCredentialResolver.cs	Switches to tier candidates, adds tier registry reads, router-selection detection, and tier resolution API.
src/MeshWeaver.AI/ChatClientAgentFactory.cs	Updates agent-tier resolution to use tier registry + deprecated config shim.
src/MeshWeaver.AI/BuiltInLanguageModelProvider.cs	Seeds Auto router nodes unconditionally and seeds tier nodes create-if-absent.
src/MeshWeaver.AI/AiCatalogLayoutAreas.cs	Adds a new “Tiers” catalog area under the AI menu.
src/MeshWeaver.AI/AgentPickerProjection.cs	Ensures Auto can be default-selected and stored Auto selections remain valid.
src/MeshWeaver.AI/AgentConfiguration.cs	Updates `ModelTier` semantics/documentation to usage-tier language.
src/MeshWeaver.AI/AgentChatClient.cs	Implements Auto dispatch and tier-based stale selection fallback with operator-visible reporting.
src/MeshWeaver.Messaging.Hub/Localization/strings.en.json	Adds menu strings for the new Tiers entry.
src/MeshWeaver.Messaging.Hub/Localization/strings.de.json	Adds German equivalents for the new Tiers menu entry.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-09-model-tiers-and-auto.md	Adds “What’s New” entry describing tiers + Auto behavior.
src/MeshWeaver.Documentation/Data/Architecture/EmailIngestionAndNotifications.md	Updates triage agent tier wording and config guidance to tier-node labeling.
src/MeshWeaver.Documentation/Data/Architecture/DeploymentOptions.md	Updates deployment guidance to node-based tiers with legacy-config shim note.
src/MeshWeaver.Documentation/Data/AI/ProviderConfiguration.md	Updates documented model-selection precedence (Auto routing + tiers + legacy shim + default).
src/MeshWeaver.Documentation/Data/AI/ModelTiers.md	New documentation for tier concepts, setup, resolution chain, Auto dispatch, and migration.
src/MeshWeaver.Documentation/Data/AI/ModelSizes.md	Removes obsolete size-based documentation.
src/MeshWeaver.Documentation/Data/AI/ModelProviderSetup.md	Updates tier table and guidance to usage-tier naming and node labeling.
memex/Memex.Portal.Shared/MemexConfiguration.cs	Adds the Tiers entry to the AI menu (with localization keys).
deploy/helm/templates/memex-portal/config.yaml	Clarifies `ModelTier__*` values as deprecated compatibility shim in Helm config.
deploy/aks/values.aks.yaml	Marks `ModelTier__*` keys deprecated and points to node-based tier labels.
content/ai/Agent/NotificationTriage.md	Updates agent tier from `light` to `chat` (usage-tier naming).
content/ai/Agent/LogTriage.md	Updates agent tier from `standard` to `reasoning` (usage-tier naming).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Adding Auto and the tier registry to the static-node emission broke 16 tests that were never about either. They asked "what does THIS config section emit" but counted every node the provider returned, so the platform's own entries — the Auto router, its pseudo-provider, the four tier nodes — landed in their totals: `HaveCount(3)` found 4, `ContainSingle` found 2, "no models" found one called `auto`. `SourceNodes()` draws the line the tests always meant: everything a configured catalog source produced, minus what ships regardless of configuration. The platform-owned entries keep their own coverage in BuiltInLanguageModelProviderTest, which selects them deliberately. The five query-shape assertions spelled `LanguageModel|ModelProvider` as a literal, so adding ModelTier to the registry filter read as a regression in a query builder nobody had touched. They now build the expected filter from the same constants production does, so the next node type added to the registry cannot fail them. ObserveDefaultComposer_StaleMasterModel is a real behaviour change, not a stale expectation: the composer's fallback is now Auto, which sorts ahead of even a deliberately-pinned Order -1. Its point — a gone pinned model is sanitised away without throwing — is intact, and the test now also asserts the model Auto dispatches to, because the default SELECTION and the model that RUNS are two different things and both have to be right. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-08-09T08:07:08Z