Koth ladder - #589
Conversation
the engine-lane winner (pr #567) moves in-package as vouch.strategies.provenance and becomes what challengers must beat: contrib/strategies/baseline.py now delegates to it instead of returning identity order. new KBs get it as the final reorder stage via the starter config (retrieval.strategy); existing KBs keep byte-identical ordering until they add the key, and strategy: null opts out. the bench is unaffected - its throwaway KBs write their own config, so kits and strategies keep being measured explicitly. rerank-isolation tests opt out of the strategy stage, since the champion is final-say and would re-sort the asserted window.
retrieval wraps query-matched terms in guillemets, so the substring checks broke exactly on query-relevant claims: expected values read as missing (a claim ranked #1 graded "not surfaced") and highlighted forbidden values slipped past the zeroing. both arms suffered equally, so paired verdicts were fair, but absolute scores were distorted in both directions. grading now strips the markers; the reference baseline table is refreshed on the fixed contract (stock composite 0.58 with the four zeros crisp on the real levers).
the dethrone band vs the reigning champion is the merge threshold; this makes it ratchet. when the ledger sweep finds a merged engine-lane pr with a dethrone scorecard, promote_champion.py copies the winner over contrib/strategies/baseline.py with a provenance header (source, pr, scored mean), so the next challenger is scored against the new champion automatically. idempotent per file content; delete-only and kit prs are untouched.
pull_request_target checks out the DEFAULT branch when no ref is given, so the engine gate scored from main - which carries the workflow but not the scorer or the strategies - and died on a missing file (pr #574's first run). the kit gate got this exact pin in 683e0a3; the engine gate now matches, and the seed identity comes from the checked-out base tip instead of github.sha for the same reason.
one uniform suspect penalty scaled by query relevance, so hearsay, possessive fragments, stored instructions, and stale conflicting values fall out of a tight candidate pool in damage order; conflict collapse marks values contradicted by a change-of-state memory; past-tense questions suspend collapse because history is the answer there. verified with the ci-exact loop over seeds 1-12: challenger 0.7083 vs champion 0.6583, diff +0.0500, band 0.0381 - dethroned.
retrieval.strategy_params turns the reigning engine-lane strategy's
decision points into a closed, pydantic-bounded config surface. a kit
can now tune suspect penalties, the danger factor, update boosts, and
collapse thresholds - and auto-merge through the kit gate - because the
score plus the schema bounds cover everything such a submission can do.
the kit validator delegates the subtree to the same schema the runtime
builds the strategy from, so the gate and the engine cannot disagree
about a legal kit. defaults reproduce the promoted champion exactly, so
strategy_params: {} scores identically to the shipped default. the
dotted code hook (retrieval.strategy) stays out of the allowlist -
naming code to import is not data.
new ranking capabilities still enter as code through the human-reviewed
engine lane once; every configuration of them competes - and merges -
as data thereafter.
feat(competition): expose the champion family as bounded kit data
|
Warning Review limit reached
Next review available in: 46 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
docs
github-actions
Bot
added
size: L
test had already landed an equivalent hermetic fix for test_handler_returns_degraded_when_absent in c89ca20 (via #589), so both sides edited the same test and the same changelog block. resolved to test's version of the test: it uses the file's existing `store` fixture, which is the idiom the sibling transcript tests already follow (test_session_transcript.py:145 points the locator at `store.kb_dir / "none"` the same way). this branch's `, resp` assertion message survives — it is the one piece not duplicated, and it puts the envelope in the failure output when this regresses again. the changelog keeps one entry instead of two, and drops the stray `)` line c89ca20 left behind under `### Fixed`.
1 participant
What changed
Why
What might break
VEP
Tests
make checkpasses locally (lint + mypy + pytest)CHANGELOG.mdupdated under## [Unreleased]