add recommend-only harness comparison - #969
Conversation
6272c61 to
5e929a2
Compare
|
Follow-up: rebased the PR head onto current The real-data smoke test now skips explicitly when a checkout has no live |
|
The CLI is well-built and I like that it's strictly read-only (never writes
Path forward: land the Phase-1 scorer tagging (or fold it into this PR) so the input schema actually exists, then wire the test in. Minor while you're here: the alias list at |
|
fixed in b91f738.
this remains forward-only: merging starts collecting attributable entries, but it does not backfill old history. the tool will continue reporting insufficient data until at least two harnesses accumulate five tagged runs for the same skill. |
…en row The alias map only special-cased codex/cursor/hermes/fx "-default" model rows, so kimi-default, vibe-default and pi-default fell into "unattributed" - 3 of the 10 harnesses under-attributed their token cost. Replace the four hardcoded checks with a single "<harness>-default" rule validated against the resolve-harness.sh allowlist (a non-harness "-default" still stays unattributed), and extend the test to cover kimi/vibe/pi plus a bogus-default rejection. Co-authored-by: Svector-anu <Svector-anu@users.noreply.github.com>
why
Phase 2 of measured harness routing. Phase 1 tags new skill-health history entries with the harness, optional model, and ISO timestamp. This adds a read-only comparison command so operators can review quality alongside token/cache usage before changing
aeon.yml.what
scripts/skill-health-routing.mjs <skill>memory/token-usage.csvconservatively using exact history model hints and documented native aliasesaeon.ymlor any repository stategithub-trendinghistory and missing-file behaviorverification
node --check scripts/skill-health-routing.mjsbash scripts/tests/test_skill_health_routing.shgit diff --checkgithub-trendinghas Codexn=1, no other tagged harness scores, so the command correctly reports insufficient data and makes no recommendationThe branch contains no synthetic health history and no configuration changes.