Skip to content

add recommend-only harness comparison - #969

Merged
aaronjmars merged 4 commits into
aeonfun:mainfrom
Svector-anu:feat/skill-health-harness-routing
Aug 27, 2026
Merged

add recommend-only harness comparison#969
aaronjmars merged 4 commits into
aeonfun:mainfrom
Svector-anu:feat/skill-health-harness-routing

Conversation

@Svector-anu

Copy link
Copy Markdown
Contributor

why

Phase 2 of measured harness routing. Phase 1 tags new skill-health history entries with the harness, optional model, and ISO timestamp. This adds a read-only comparison command so operators can review quality alongside token/cache usage before changing aeon.yml.

what

  • add scripts/skill-health-routing.mjs <skill>
  • group only harness-tagged history entries
  • require five tagged scores per harness before a recommendation
  • join model-keyed memory/token-usage.csv conservatively using exact history model hints and documented native aliases
  • report unmatched token rows instead of guessing
  • show input tokens, cache reads, cache ratio, and per-run averages
  • never write aeon.yml or any repository state
  • add a real-data smoke test for the current github-trending history and missing-file behavior

verification

  • node --check scripts/skill-health-routing.mjs
  • bash scripts/tests/test_skill_health_routing.sh
  • git diff --check
  • real current output: github-trending has Codex n=1, no other tagged harness scores, so the command correctly reports insufficient data and makes no recommendation

The branch contains no synthetic health history and no configuration changes.

@Svector-anu
Svector-anu force-pushed the feat/skill-health-harness-routing branch from 6272c61 to 5e929a2 Compare August 26, 2026 12:58
@Svector-anu

Copy link
Copy Markdown
Contributor Author

Follow-up: rebased the PR head onto current upstream/main so the diff contains exactly the two new scripts (the fork main carried unrelated auto-commit history).

The real-data smoke test now skips explicitly when a checkout has no live memory/skill-health/github-trending.json (as in upstream CI) rather than fabricating fixtures. In the operator fork checkout, the same test exercises the real file and reports Codex n=1 with no recommendation.

@aaronjmars

Copy link
Copy Markdown
Collaborator

The CLI is well-built and I like that it's strictly read-only (never writes aeon.yml). Two things block it as-is:

  1. It depends on a "Phase 1" that isn't in the repo. The PR body says Phase 1 tags skill-health history entries with harness / optional model / ISO timestamp, but the canon scorer in .github/workflows/aeon.yml appends only {date, score} (jq '. + [{"date": $d, "score": $s}] | .[-30:]') - no harness field. No merged or open PR adds that tagging. So taggedScores() drops every entry, groups is always empty, eligible.length < 2 always holds, and on canon (and every current live instance) the tool can only ever print "recommendation: none - insufficient data". The feature is inert until the tagging exists.

  2. The new test isn't run by CI. ci-tests.yml uses an explicit hand-listed set of test steps, not a glob, and this PR doesn't add test_skill_health_routing.sh to it - so it never executes in CI. And as written it self-skips on canon (the early exit 0 when memory/skill-health/github-trending.json is absent skips even the data-independent negative assertion), so register it in ci-tests.yml and move the missing-file negative test above the live-data gate.

Path forward: land the Phase-1 scorer tagging (or fold it into this PR) so the input schema actually exists, then wire the test in. Minor while you're here: the alias list at scripts/skill-health-routing.mjs:96-101 already omits cursor/hermes/fx.

@Svector-anu

Copy link
Copy Markdown
Contributor Author

fixed in b91f738.

  • folded the scorer tagging into this pr: new history entries now include the actual run harness, an iso timestamp, and the model only when explicitly resolved. old {date,score} entries are left untouched.
  • registered test_skill_health_routing.sh in ci-tests.yml and moved the missing-file negative assertion before the live-data gate, so canon ci always exercises it.
  • added cursor-default, hermes-default, and fx-default to the conservative model-to-harness aliases, with hermetic coverage for all three.
  • mutation check: changing the missing-file assertion to an impossible error string makes the test exit 1 at that assertion; restoring it passes.
  • real ci: https://github.com/aeonfun/aeon/actions/runs/33080339668 — the log shows the routing test invoked and data-independent assertions passed; shellcheck also passed in https://github.com/aeonfun/aeon/actions/runs/33080339674.

this remains forward-only: merging starts collecting attributable entries, but it does not backfill old history. the tool will continue reporting insufficient data until at least two harnesses accumulate five tagged runs for the same skill.

…en row

The alias map only special-cased codex/cursor/hermes/fx "-default" model rows,
so kimi-default, vibe-default and pi-default fell into "unattributed" -
3 of the 10 harnesses under-attributed their token cost. Replace the four
hardcoded checks with a single "<harness>-default" rule validated against the
resolve-harness.sh allowlist (a non-harness "-default" still stays
unattributed), and extend the test to cover kimi/vibe/pi plus a bogus-default
rejection.

Co-authored-by: Svector-anu <Svector-anu@users.noreply.github.com>
@aaronjmars
aaronjmars merged commit a59b691 into aeonfun:main Aug 27, 2026
2 checks passed
@Svector-anu
Svector-anu deleted the feat/skill-health-harness-routing branch August 27, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants