feat(agent): feed aggregate outcome quality into confidence - #332
Merged
JSONbored merged 2 commits intoJun 3, 2026
Merged
Conversation
Introduce aggregate outcome quality signals from repo outcome patterns into recommendation confidence and evidence in the agent orchestrator. - Add `aggregateOutcomeQuality()` helper that classifies a repo's outside-contributor merge rate as strong (≥60%), weak (30–60%), high_risk (≤30%), sparse (<5 decided PRs), or absent - Feed the signal into `confidenceForDecisionPack()`: high-risk lowers to "low", weak lowers to "medium"; sparse and absent are advisory only - Thread the result into `decisionPackEvidence()` as a new `aggregate_outcome_quality` evidence source with per-signal freshness, warnings (high-risk/weak), and assumptions (sparse/absent) - Expose `aggregateOutcomeQuality` via `__agentOrchestratorInternals` - Add tests for all five signal branches, public sanitizer, and evidence shape; all private aggregate quality stays out of public card text Closes JSONbored#304
JSONbored
approved these changes
Jun 3, 2026
JSONbored
left a comment
JSONbored
left a comment
Owner
There was a problem hiding this comment.
@web-dev0521 this is ready from my pass.
A few notes:
- Folding aggregate repo outcome quality into private recommendation evidence makes sense.
- The confidence lowering for weak/high-risk outcomes is conservative.
- The public card sanitizer assertion is the right guardrail.
No code changes requested.
Validation expected:
- Keep the current green CI.
Closed
12 tasks
JSONbored
added
gittensor:feature
This was referenced Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
aggregateOutcomeQuality()classifies a repo's outside-contributor merge rate into five signals:strong(≥60%),weak(30–60%),high_risk(≤30%),sparse(<5 decided PRs),absentconfidenceForDecisionPack()now lowers confidence to"low"for high-risk outcomes and"medium"for weak outcomes; sparse/absent are advisory (no confidence impact, but assumption added)decisionPackEvidence()gains a newaggregate_outcome_qualityevidence source with signal-appropriate freshness, warnings, and assumptions — all in the private payload, never in public card textCloses #304
Scope
CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally; global coverage stays at or above 97% for lines, statements, functions, and branches (Statements 99.08%, Branches 97.1%, Functions 98.37%, Lines 99.66%)npm run test:workers(running in CI — passes on Linux; local Windows environment has pre-existing UV_HANDLE_CLOSING failures in mcp-cli.test.ts unrelated to this change)npm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
test:cichain (actionlint,build:mcp, UI checks) not run locally as they require additional tooling (actionlint binary, UI build chain); these run in CI and are unaffected by the two changed files (src/services/agent-orchestrator.ts,test/unit/agent-orchestrator.test.ts)mcp-cli.test.tshas 12 pre-existing Windows-platform failures (UV_HANDLE_CLOSINGassertion inlibuv) present onmainbefore this branch; unrelated to these changesSafety