ADE router: collect per-turn usage data from users, then build the router from it #1298

Goal

Build an "ADE router" that picks the harness, model, effort, and account for each task, in a cache-aware and quota-aware way (idea from Cognition's Devin Fusion + "more efficient Devin"). Beat Devin by routing across every harness and plan ADE supports, not one.

Status (2026-09-30)

Phase 1 (measure the router) is MERGED in PR arul28/ADE#1398 (merge b3b429c4a). It releases with the next desktop release after v1.2.83.

Earlier: steps 1 and 2 (model registry + shadow router) merged in PR arul28/ADE#1383 (merge adb5ced3e, in v1.2.83). Groundwork: PR arul28/ADE#1300 (per-turn usage ledger).

Current plan: release, update all machines, collect about 1 week of data (shadow log + efficiency report), review the picks with the owner, then fix plan burn rates, then live routing for low-risk subagents only.

What exists now

Model registry (Cloudflare account-directory Worker)

• A daily cron reads Artificial Analysis (2 page fetches; ~684 model records + the Coding Agent Index) and models.dev prices.
• GET /router/registry: signed-in ADE accounts only (Clerk bearer), ETag/304. D1 table model_registry_snapshots (migration 0013, keeps 7). Kill switch: MODEL_REGISTRY_REFRESH=0.
• Contract: apps/desktop/src/shared/routerRegistry.ts (NOT shared/modelRegistry.ts).

Shadow router (desktop + runtime) in apps/desktop/src/main/services/router/

• routeCatalog.ts: route = harness × model × effort from ADE's own model list, rated from the registry. The reference lookup prefers the exact model id over its family, so an OpenCode Go model keeps Go-plan billing (fixed in arul28/ADE#1398).
• routerCore.ts: task kind, plan state from live quota windows, pick = cheapest trusted route within the kind's quality tolerance and at most 1.5× slower. New kind lead (tolerance 0.02) for main threads.
• modelRouterService.ts: logs a pick for each subagent to ~/.ade/usage/router-shadow-YYYY-MM.jsonl. The model list is now the union of every open project (arul28/ADE#1398). Before, only the newest project counted, so the MacBook saw 0 OpenCode routes.
• routerEfficiency.ts (new in arul28/ADE#1398): replays every chat thread in the turn ledger at its free switch points (thread start, after compaction, after cache expiry: 5 min for Claude threads that write only 5-minute cache, else 1 h; route changed by the user). One pick per segment, priced against what ran. It also sums the shadow-logged subagents (token-weighted). A per-billing view shows list dollars and plan-window percent side by side. History comes from the ledger, so it has numbers on day one.
• CLI: ade router routes | pick | shadow | efficiency | refresh. Action usage.getRouterEfficiency.

Findings

30-day research (3 machines, list price): ~$27k; Claude 98%. Subagents 53% of Claude cost (2,218 runs, almost all Opus). Cache rebuilds ~14% of Claude cost (ADE restart/resume 4.2%, idle > 1h 3.3%, MCP tool-list change 2.1%). Cost per merged PR: median $163. Backtest of subagent re-routing: −63% to −66% (predicted).

First efficiency replay (2026-09-30, ledger since 09-24/25):

• Mac Studio: 92 threads, 569 turns, 169 segments, $1,309 list. MacBook: 44 threads, 172 turns, 62 segments, $203.
• Main threads: same-harness saving ≈ 0%. No Claude route is within 0.02 of Opus 5.5 and cheaper. Savings must come from subagents and cheaper worker tiers, not from switching the lead thread.
• Cross-harness picks are not trustworthy yet. ADE measures Claude at $10.93 per 1% of the weekly window, Codex at $0.09 per 1% (Mac Studio). ADE sees $6.82 of Codex work while the Codex window moved 72%, mostly from use outside ADE. So the router moves all Codex and DeepSeek work onto the Claude plan. This is the burn-rate bias below.

Research data

All research data and scripts are on the Mac Studio at /Users/admin/Projects/ADE/.ade/research/ade-159/ (git-ignored, private; start with its README.md). It holds the 30-day extracts of three machines, the Artificial Analysis pull, the route fits, and the phase 1 efficiency reports. Keep it off GitHub and Linear: the extracts contain prompt text, and the AA pages may not be redistributed. The Claude transcripts it came from expire after 30 days, so it cannot be made again.

Known limits

• Burn rates count ADE turns only. A plan also used outside ADE looks dearer. The efficiency report's per-billing view now shows this. Fix it before any cross-plan pick goes live.
• Quality is a benchmark prediction, not measured on the owner's tasks. More shadow data does not fix this; live low-risk routing with verify-then-escalate does.
• Claude reports no per-subagent price, so subagent savings are token-weighted.
• Cursor has no public per-token prices; 10 models have no public score.
• Claude's Agent tool can only switch among Claude models; cross-harness workers need ADE subagent chats.

Next steps

1. Release and update every machine (Mac Studio still ran v1.2.82 on 09-30).
2. Collect about 1 week: ade router shadow --days 7 and ade router efficiency --days 7. Review the picks with the owner.
3. Fix plan burn rates from the full local provider-ledger scan + account rollups, not ADE turns only.
4. Cache guard PR (~6% of Claude cost): stop ADE-caused rebuilds (runtime restart/resume, MCP tool-list changes mid-session).
5. Live subagent routing, low-risk kinds first (read-only, search) behind a flag, verify-then-escalate. About 2 weeks to reach ~150 outcomes per kind and route. Edit tasks last.
6. Lead-thread switching only at free switch points, and only if the data shows a saving (phase 1 says it is near 0 today).
7. Learning loop: outcomes per route and task kind; later a private benchmark from merged PRs.