Evaluate sessions through the shared evidence contract

[realtonyyoung]

AI-2746 — no GitHub issue exists for this change; the server half is kurrent-io/kcap-server#2058.

What & why

kcap eval and the daemon fetched and kept a whole compacted trace, could not certify a citation, and reported budget stops as generic failures. When the server's eval catalog advertises evidence_retrieval, a single-session run now prepares from the viewer's evidence scope through bounded reads. It judges one-shot or through eight tools served by kcap mcp judge --run, budgeted at the MCP boundary and ledgered to an owner-only per-run directory. Its citations are certified in batches and persisted with coverage-v2. Without the advertisement, and always under --chain, requests and payloads are byte-identical to today.

Where to look

• Scope loss (a 404/409 on renewal, read or certification) ends the run with nothing certified or persisted.
• The daemon's run cache now owns each prepared run's HttpClient, holding it for up to 30 minutes idle.

Verification

From src/cli, dotnet run --project test/<Project>/<Project>.csproj -- --treenode-filter "/*/*/<Class>/*" at the head:

Class	Passed
EvalEvidenceRouteTests	32/32
McpEvidenceJudgeToolsTests	24/24
EvalRunnerEvidenceTests	16/16
EvalContextCacheTests	7/7
EvalServiceLegacyByteIdentityTests	4/4

dotnet publish -c Release of Capacitor.Cli and Capacitor.Cli.Daemon (osx-arm64): grep -E 'IL[23][01][0-9]{2}' finds nothing.

🤖 Generated with Claude Code

[linear-code]

AI-2746

[qodo-code-review]

PR Summary by Qodo

Evaluate single sessions through the shared evidence contract

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Routes advertised single-session evaluations through bounded, viewer-scoped evidence retrieval.
• Certifies citations and persists ledger-derived coverage-v2, strategy, and obligation results.
• Preserves legacy payload bytes while adding daemon lifecycle and budget safeguards.

Diagram

sequenceDiagram
    actor Reviewer
    participant Eval as Eval Service
    participant Scope as Evidence API
    participant Judge as Claude Judge
    participant MCP as Judge MCP
    participant Ledger as Run Ledger
    participant Persist as Eval Storage
    Reviewer->>Eval: Start single-session eval
    Eval->>Scope: Resolve viewer scope
    Scope-->>Eval: Manifest and budgets
    Eval->>Eval: Choose one-shot or retrieval
    Eval->>Judge: Prompt and bounded evidence
    Judge->>MCP: Read evidence pages
    MCP->>Scope: Scoped reads
    MCP->>Ledger: Record pages and stops
    Judge-->>Eval: Verdict and cite handles
    Eval->>Scope: Certify citation batch
    Scope-->>Eval: Refs and digests
    Eval->>Ledger: Calculate coverage-v2
    Eval->>Persist: Persist certified assessment

Loading

High-Level Assessment

The advertised capability gate with an unchanged legacy fallback is the strongest approach. It adopts the server-owned evidence contract instead of duplicating authorization or retaining compacted traces, while owner-only run files safely bridge the parent evaluator and MCP subprocess. In-memory token passing or per-call certification would increase exposure, coupling, and request overhead without improving correctness.

Files changed (133) +8035 / -237

Enhancement (83) +3693 / -198

BaselineObserver.csRetain evidence treatment and question ledgers +38/-1

Retain evidence treatment and question ledgers

• Records treatment metadata and safely copies each evidence ledger under a hashed, ordinal filename beside baseline output.

src/Capacitor.Cli.Core/Eval/BaselineObserver.cs

BaselineOutput.csAdd treatment metadata to baselines +5/-0

Add treatment metadata to baselines

• Adds an optional evidence treatment record without changing legacy serialization.

src/Capacitor.Cli.Core/Eval/BaselineOutput.cs

BaselineQuestionOutput.csRecord retained ledger paths +7/-2

Record retained ledger paths

• Adds optional per-question ledger paths and documents the expanded route vocabulary.

src/Capacitor.Cli.Core/Eval/BaselineQuestionOutput.cs

EvalEvidenceCoverage.csValidate evidence coverage records +32/-0

Validate evidence coverage records

• Adds contract limits and validation for policy versions, sources, citations, omissions, and stop reasons.

src/Capacitor.Cli.Core/Eval/Contracts/EvalEvidenceCoverage.cs

EvalFailureCodes.csAdd budget and iteration failure codes +8/-2

Add budget and iteration failure codes

• Introduces iteration-cap and spend-budget failures plus the accepted CLI failure vocabulary.

src/Capacitor.Cli.Core/Eval/Contracts/EvalFailureCodes.cs

EvalObligationResult.csModel certified obligation results +19/-0

Model certified obligation results

• Defines the persisted obligation contract with identity, status, anchor, citations, and optional note.

src/Capacitor.Cli.Core/Eval/Contracts/EvalObligationResult.cs

EvalOmissionKinds.csDefine coverage omission vocabularies +27/-0

Define coverage omission vocabularies

• Mirrors coverage-v1 and coverage-v2 omission kinds used to describe undelivered evidence.

src/Capacitor.Cli.Core/Eval/Contracts/EvalOmissionKinds.cs

EvalQuestionAssessment.csExtend and validate question assessments +110/-2

Extend and validate question assessments

• Adds trace coverage, strategy stamps, obligations, loss reasons, and server-aligned validation while omitting null fields.

src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionAssessment.cs

EvalQuestionFailure.csValidate coded question failures +20/-0

Validate coded question failures

• Enforces detail-field rules for iteration caps, chat errors, timeouts, parse failures, and spend stops.

src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionFailure.cs

EvalStopReasons.csDefine evidence stop reasons +17/-0

Define evidence stop reasons

• Adds versioned vocabularies for byte, iteration, tool-call, time, token, and judge stops.

src/Capacitor.Cli.Core/Eval/Contracts/EvalStopReasons.cs

EvalTraceCoverage.csModel one-shot and retrieval trace coverage +58/-0

Model one-shot and retrieval trace coverage

• Defines mode-specific coverage metrics, constructors, and validation for complete traces and bounded retrieval.

src/Capacitor.Cli.Core/Eval/Contracts/EvalTraceCoverage.cs

QuestionResultV2.csReport run-fatal evidence failures +7/-2

Report run-fatal evidence failures

• Adds an optional scope-moved run failure while preserving legacy hub payloads when absent.

src/Capacitor.Cli.Core/Eval/Contracts/QuestionResultV2.cs

QuestionRunResult.csReturn per-question judge usage +1/-1

Return per-question judge usage

• Carries optional token usage from in-process question execution to daemon responses.

src/Capacitor.Cli.Core/Eval/Contracts/QuestionRunResult.cs

EvalService.Evidence.csOrchestrate evidence-scoped evaluations +502/-0

Orchestrate evidence-scoped evaluations

• Implements preparation, route selection, question execution, scope renewal, certification, obligation reconciliation, retrospectives, coverage-v2, and safe finalization.

src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs

EvalService.csGate evidence routing and preserve legacy behavior +86/-76

Gate evidence routing and preserve legacy behavior

• Selects evidence evaluation only for advertised non-chain runs, classifies budget failures, carries strategies and usage, and protects producer-owned fields.

src/Capacitor.Cli.Core/Eval/EvalService.cs

EvalEvidenceAdvertisementDto.csMirror advertised evidence budgets +13/-0

Mirror advertised evidence budgets

• Models the catalog capability and server-owned retrieval, page, one-shot, retrospective, and policy limits.

src/Capacitor.Cli.Core/Eval/Evidence/EvalEvidenceAdvertisementDto.cs

EvalObligationContract.csDefine judge obligation contract +122/-0

Define judge obligation contract

• Mirrors obligation schemas, vocabularies, byte limits, and all-or-nothing parsing rules.

src/Capacitor.Cli.Core/Eval/Evidence/EvalObligationContract.cs

EvalObligationRules.csReconcile certified obligations deterministically +100/-0

Reconcile certified obligations deterministically

• Normalizes titles, derives stable IDs, merges duplicates, downgrades unsupported statuses, and requires certified evidence.

src/Capacitor.Cli.Core/Eval/Evidence/EvalObligationRules.cs

EvalReportedObligation.csRepresent unverified judge obligations +5/-0

Represent unverified judge obligations

• Models judge-reported obligations before cite handles are expanded and certified.

src/Capacitor.Cli.Core/Eval/Evidence/EvalReportedObligation.cs

EvalStrategiesMirror.csSelect the obligation reporting question +26/-0

Select the obligation reporting question

• Mirrors server ordering rules for choosing one completion question to report obligations.

src/Capacitor.Cli.Core/Eval/Evidence/EvalStrategiesMirror.cs

EvalTreatment.csCapture evidence evaluation treatment +28/-0

Capture evidence evaluation treatment

• Records capability state, effective budgets, token limits, and hashes of embedded prompt resources.

src/Capacitor.Cli.Core/Eval/Evidence/EvalTreatment.cs

EvidenceBodyChunkDto.csModel paged evidence bodies +12/-0

Model paged evidence bodies

• Mirrors body content, encoding, ordinal, and continuation offsets from the evidence API.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceBodyChunkDto.cs

EvidenceBodyDescriptorDto.csModel deferred evidence bodies +10/-0

Model deferred evidence bodies

• Defines body descriptors used when canonical content must be fetched separately.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceBodyDescriptorDto.cs

EvidenceCallEntryDto.csModel evidence tool calls +12/-0

Model evidence tool calls

• Represents inline and deferred call arguments and canonical call bodies.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCallEntryDto.cs

EvidenceCanonicalContent.csIdentify canonical event content +51/-0

Identify canonical event content

• Defines which bodies constitute delivered evidence and shares keys for equivalent call representations.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCanonicalContent.cs

EvidenceCertificationOutcome.csRepresent citation certification outcomes +7/-0

Represent citation certification outcomes

• Carries certified citations, dropped-reference counts, and scope-loss state.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCertificationOutcome.cs

EvidenceCitationClient.csCertify citations in bounded batches +59/-0

Certify citations in bounded batches

• Batches refs, retries server work-budget deferrals, enforces a shared timeout, and rejects changed scopes.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs

EvidenceCitationEntryDto.csModel citation certification entries +10/-0

Model citation certification entries

• Mirrors each reference's certification state, digest, and refusal code.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationEntryDto.cs

EvidenceCitationsRequestDto.csModel citation certification requests +8/-0

Model citation certification requests

• Defines token-bound batches of evidence references.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationsRequestDto.cs

EvidenceCitationsResponseDto.csModel citation certification responses +8/-0

Model citation certification responses

• Carries the answering scope version and per-reference certification results.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationsResponseDto.cs

EvidenceCoverageMeasure.csDerive coverage-v2 from ledger delivery +125/-0

Derive coverage-v2 from ledger delivery

• Calculates consulted sources, unread turns, pages and bodies, budget stops, trace metrics, and iteration-cap detail.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCoverageMeasure.cs

EvidenceEventEntryDto.csMirror evidence event entries +17/-0

Mirror evidence event entries

• Models event identity, canonical inline content, calls, and deferred body descriptors.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceEventEntryDto.cs

EvidenceEventPageDto.csMirror paged evidence events +10/-0

Mirror paged evidence events

• Defines scope-bound event pages and continuation cursors.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceEventPageDto.cs

EvidenceFirstView.csRepresent strategy first views +5/-0

Represent strategy first views

• Carries resolved strategy stamps, rendered orientation text, and seeded ledger pages.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceFirstView.cs

EvidenceFirstViewReader.csRead and seed strategy first views +55/-0

Read and seed strategy first views

• Fetches bounded strategy views, renders citable sections, tracks omissions, and distinguishes transient failures from scope loss.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceFirstViewReader.cs

EvidenceHttpResult.csRepresent evidence HTTP outcomes +6/-0

Represent evidence HTTP outcomes

• Normalizes HTTP status and body, including unreachable requests as status zero.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceHttpResult.cs

EvidenceOrientation.csRepresent run orientation +9/-0

Represent run orientation

• Carries seeded pages, outlined-turn counts, rendered text, and read failure status.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceOrientation.cs

EvidenceOrientationBuilder.csBuild bounded run orientation +76/-0

Build bounded run orientation

• Combines scope metadata, source manifest, root outline, and call summary without partially delivering pages.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceOrientationBuilder.cs

EvidencePageRenderer.csRender citable evidence pages +154/-0

Render citable evidence pages

• Adds page and cite handles, strips cursors, and records delivered revisions, turns, bodies, and byte spans.

src/Capacitor.Cli.Core/Eval/Evidence/EvidencePageRenderer.cs

EvidencePreambleHashes.csHash evidence prompt resources +15/-0

Hash evidence prompt resources

• Computes treatment hashes over exact embedded question, one-shot, and retrospective resources.

src/Capacitor.Cli.Core/Eval/Evidence/EvidencePreambleHashes.cs

EvidencePromptBlocks.csRender evidence prompts safely +34/-0

Render evidence prompts safely

• Loads preambles and performs single-pass placeholder replacement without reinterpreting inserted session content.

src/Capacitor.Cli.Core/Eval/Evidence/EvidencePromptBlocks.cs

EvidenceQuestionOutcome.csRepresent evidence question outcomes +9/-0

Represent evidence question outcomes

• Distinguishes assessments, coded failures, usage, and run-fatal scope movement.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceQuestionOutcome.cs

EvidenceReadClient.csAdd scoped evidence HTTP reads +19/-0

Add scoped evidence HTTP reads

• Builds encoded evidence requests, normalizes transport failures, and supports admission rechecks.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceReadClient.cs

EvidenceReadErrorDto.csModel evidence read errors +5/-0

Model evidence read errors

• Defines the shared evidence error code and detail response.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceReadErrorDto.cs

EvidenceRefForm.csClassify evidence reference forms +3/-0

Classify evidence reference forms

• Defines event, range, and turn reference variants.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRefForm.cs

EvidenceRefText.csParse canonical evidence references +74/-0

Parse canonical evidence references

• Implements non-throwing parsing and strict percent-decoding for session and subagent event, range, and turn refs.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRefText.cs

EvidenceRetrospectiveInputs.csBuild bounded retrospective evidence +99/-0

Build bounded retrospective evidence

• Re-reads cited refs in priority order and creates scoped excerpts plus completeness and budget summaries.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRetrospectiveInputs.cs

EvidenceRoute.csDefine evidence and legacy route names +15/-0

Define evidence and legacy route names

• Adds internal route variants and stable wire and observer names.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRoute.cs

EvidenceRouteChoice.csChoose one-shot or retrieval evaluation +14/-0

Choose one-shot or retrieval evaluation

• Caps one-shot size by both advertised and token budgets, selecting retrieval unless the full canonical trace fits.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRouteChoice.cs

EvidenceRunBudgets.csRepresent per-run retrieval budgets +3/-0

Represent per-run retrieval budgets

• Groups tool-call, delivered-byte, and page-size limits shared with the MCP subprocess.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunBudgets.cs

EvidenceRunContext.csSecure evidence run working state +114/-0

Secure evidence run working state

• Creates owner-only run directories, buffers retained facts, deletes artifacts on disposal, and sweeps stale safe directories.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunContext.cs

EvidenceRunFile.csSerialize MCP evidence handoff files +59/-0

Serialize MCP evidence handoff files

• Stores the scope token, manifest, budgets, deadlines, ledger path, and seeded pages in an owner-only versioned file.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunFile.cs

EvidenceRunSetup.csAggregate prepared evidence run state +43/-0

Aggregate prepared evidence run state

• Owns the bound scope, clients, route, orientation, questions, strategy cache, harness configuration, and timeouts.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunSetup.cs

EvidenceRunSource.csModel MCP-visible evidence sources +6/-0

Model MCP-visible evidence sources

• Defines the reduced source manifest handed to judge tools.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunSource.cs

EvidenceScopeClient.csResolve and renew viewer evidence scopes +111/-0

Resolve and renew viewer evidence scopes

• Reads paged manifests, uses server-relative lifetimes, rechecks admission, and treats version changes or revocation as scope movement.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceScopeClient.cs

EvidenceScopeErrorDto.csModel evidence scope errors +5/-0

Model evidence scope errors

• Defines scope error codes and optional current versions.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceScopeErrorDto.cs

EvidenceScopeManifestDto.csMirror evidence scope manifests +16/-0

Mirror evidence scope manifests

• Models completeness, sources, pagination, token, and optional server lifetime fields.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceScopeManifestDto.cs

EvidenceScopeState.csRepresent a bound evidence scope +5/-0

Represent a bound evidence scope

• Combines the complete manifest with its current token and client-relative deadline.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceScopeState.cs

EvidenceScopeStatus.csClassify scope operations +3/-0

Classify scope operations

• Defines successful, hidden, moved, unauthorized, and failed scope states.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceScopeStatus.cs

EvidenceSourceDto.csMirror evidence source descriptors +17/-0

Mirror evidence source descriptors

• Models root and subagent identity, relationships, revision bounds, turn counts, and availability.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceSourceDto.cs

EvidenceTraceAssembler.csAssemble bounded canonical one-shot traces +151/-0

Assemble bounded canonical one-shot traces

• Streams available sources and deferred bodies, stopping immediately when escaped canonical content cannot fit.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs

EvidenceTraceResult.csRepresent one-shot fit results +18/-0

Represent one-shot fit results

• Carries assembled trace data, cite mappings, entry spans, read counts, size estimates, and failures.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceResult.cs

EvidenceTurnEntryDto.csMirror evidence turn outlines +12/-0

Mirror evidence turn outlines

• Models turn and event refs, range state, indexes, and revision bounds.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTurnEntryDto.cs

EvidenceTurnPageDto.csMirror paged turn outlines +10/-0

Mirror paged turn outlines

• Defines scope-bound turn pages and continuation cursors.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTurnPageDto.cs

JudgeCiteHandles.csCreate and expand judge cite handles +26/-0

Create and expand judge cite handles

• Defines seeded, dynamic, row, and one-shot handles and expands only delivered evidence within caps.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeCiteHandles.cs

JudgeLedger.csDerive delivered evidence from ledgers +56/-0

Derive delivered evidence from ledgers

• Builds citation, event, turn, source, continuation, refusal, byte, and stop state from ledger records.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedger.cs

JudgeLedgerCall.csModel ledgered tool calls +3/-0

Model ledgered tool calls

• Records tool sequence, arguments, outcome, stop reason, error, and delivered bytes.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerCall.cs

JudgeLedgerFooter.csModel authoritative ledger totals +4/-0

Model authoritative ledger totals

• Defines repeatedly appended totals whose last complete line survives interrupted questions.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerFooter.cs

JudgeLedgerHeader.csModel versioned ledger headers +5/-0

Model versioned ledger headers

• Records run identity, question, scope, budgets, deadline, and start time.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerHeader.cs

JudgeLedgerOutcomes.csDefine ledger call outcomes +9/-0

Define ledger call outcomes

• Distinguishes executed, refused, unexecuted, and errored tool calls.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerOutcomes.cs

JudgeLedgerPage.csModel exact delivered evidence pages +25/-0

Model exact delivered evidence pages

• Stores judge-visible text and structured delivery metadata with byte-aware equality.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerPage.cs

JudgeLedgerReader.csRead interruption-tolerant ledgers +58/-0

Read interruption-tolerant ledgers

• Reconstructs headers, pages, calls, and the latest footer while ignoring torn JSON lines.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerReader.cs

JudgeLedgerWriter.csWrite durable per-call ledgers +114/-0

Write durable per-call ledgers

• Flushes owner-only JSONL records for seeded pages, delivered pages, calls, and authoritative running totals.

src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerWriter.cs

IEvalObserver.csObserve evidence treatment and ledgers +13/-3

Observe evidence treatment and ledgers

• Adds backward-compatible callbacks and expanded route documentation for evidence evaluation telemetry.

src/Capacitor.Cli.Core/Eval/IEvalObserver.cs

Models.csExtend shared evaluation wire models +53/-20

Extend shared evaluation wire models

• Adds strategy, advertisement, evidence DTO serialization, and null-omitted daemon route and scope fields.

src/Capacitor.Cli.Core/Models.cs

preamble-eval-oneshot.txtAdd complete-trace evidence preamble +1/-0

Add complete-trace evidence preamble

• Tells one-shot judges the trace is complete, scoped, and directly citable.

src/Capacitor.Cli.Core/Resources/preamble-eval-oneshot.txt

preamble-eval-retrospective.txtAdd bounded retrospective preamble +1/-0

Add bounded retrospective preamble

• Constrains retrospectives to verdicts and re-readable cited excerpts without rescoring.

src/Capacitor.Cli.Core/Resources/preamble-eval-retrospective.txt

prompt-eval-question-evidence.txtAdd retrieval judge prompt +80/-0

Add retrieval judge prompt

• Describes the eight evidence tools, budgets, citation handles, outcomes, and structured verdict contract.

src/Capacitor.Cli.Core/Resources/prompt-eval-question-evidence.txt

EvalContextCache.csOwn evidence runs and HTTP clients in cache +64/-41

Own evidence runs and HTTP clients in cache

• Caches either legacy or evidence contexts and disposes run directories and owned clients on replacement, expiry, removal, or shutdown.

src/Capacitor.Cli.Daemon/Services/EvalContextCache.cs

EvalRunner.csRun evidence evaluation phases in daemon +92/-37

Run evidence evaluation phases in daemon

• Prepares and caches evidence runs, applies phase deadlines, returns usage and run-fatal scope failures, and finalizes safely.

src/Capacitor.Cli.Daemon/Services/EvalRunner.cs

McpEvidenceJudgeTools.csServe eight budgeted evidence judge tools +314/-0

Serve eight budgeted evidence judge tools

• Implements scoped page reads, admission checks, continuations, budget enforcement, error handling, cite handles, and durable ledgering.

src/Capacitor.Cli/Commands/McpEvidenceJudgeTools.cs

McpJudgeServer.csAdd evidence mode to judge MCP server +47/-9

Add evidence mode to judge MCP server

• Selects the evidence tool surface from a validated run file while retaining the existing six legacy tools.

src/Capacitor.Cli/Commands/McpJudgeServer.cs

Program.csAccept evidence run files for MCP judge +3/-2

Accept evidence run files for MCP judge

• Parses and forwards the internal --run option to the judge MCP server.

src/Capacitor.Cli/Program.cs

Bug fix (1) +21 / -5

ClaudeCliRunner.csPreserve Claude failure envelope subtypes +21/-5

Preserve Claude failure envelope subtypes

• Retains result subtypes and classifies max-budget failures separately from generic process and parse errors.

src/Capacitor.Cli.Core/Harness/Claude/ClaudeCliRunner.cs

Tests (44) +4254 / -16

Capacitor.Cli.Core.Tests.Unit.csprojInclude evidence contract fixtures in unit tests +2/-0

Include evidence contract fixtures in unit tests

• Adds fixture assets needed by shared evidence and strategy contract tests.

test/Capacitor.Cli.Core.Tests.Unit/Capacitor.Cli.Core.Tests.Unit.csproj

BaselineObserverTests.csTest treatment and ledger baseline output +149/-0

Test treatment and ledger baseline output

• Covers safe ledger filenames, copies, failures, duplicate questions, optional fields, and observer forwarding.

test/Capacitor.Cli.Core.Tests.Unit/Eval/BaselineObserverTests.cs

EvalServiceEvidenceRequestTests.csTest evidence request and prompt construction +62/-0

Test evidence request and prompt construction

• Verifies evidence-specific request shapes, routing inputs, and bounded prompt configuration.

test/Capacitor.Cli.Core.Tests.Unit/Eval/EvalServiceEvidenceRequestTests.cs

EvalServiceLegacyByteIdentityTests.csPin legacy wire byte identity +42/-0

Pin legacy wire byte identity

• Ensures new optional evidence fields do not alter serialized legacy prepare and question messages.

test/Capacitor.Cli.Core.Tests.Unit/Eval/EvalServiceLegacyByteIdentityTests.cs

EvalServiceProducerFieldsTests.csTest producer-owned verdict fields +25/-0

Test producer-owned verdict fields

• Verifies judge-supplied strategy and persisted-obligation fields are ignored and producer-reconciled.

test/Capacitor.Cli.Core.Tests.Unit/Eval/EvalServiceProducerFieldsTests.cs

EvalContractValidateTests.csTest evidence contract validation +85/-0

Test evidence contract validation

• Exercises valid and invalid assessment, coverage, trace, and failure combinations.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvalContractValidateTests.cs

EvalObligationContractTests.csTest obligation parsing limits +107/-0

Test obligation parsing limits

• Covers schema vocabularies, encoded byte caps, required fields, unknown members, and all-or-nothing parsing.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvalObligationContractTests.cs

EvalObligationRulesTests.csTest deterministic obligation reconciliation +173/-0

Test deterministic obligation reconciliation

• Uses shared vectors to verify IDs, merging, certification, status downgrades, ordering, and outcome changes.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvalObligationRulesTests.cs

EvalStrategiesMirrorTests.csTest reporting-question selection +29/-0

Test reporting-question selection

• Pins marked and fallback completion strategy behavior against shared fixtures.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvalStrategiesMirrorTests.cs

EvalTreatmentTests.csTest evidence treatment records +37/-0

Test evidence treatment records

• Verifies gate state, advertised and local budgets, and prompt resource hashes.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvalTreatmentTests.cs

EvidenceBudgetsTests.csTest first-view and turn budgets +16/-0

Test first-view and turn budgets

• Covers section allocation boundaries and max-turn calculations.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceBudgetsTests.cs

EvidenceCitationClientTests.csTest batched citation certification +80/-0

Test batched citation certification

• Covers ordering, batching, retries, drops, timeouts, transport errors, and scope-version changes.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceCitationClientTests.cs

EvidenceContractFixtureTests.csPin the shared evidence contract fixture +81/-0

Pin the shared evidence contract fixture

• Ensures local vocabularies, limits, body rules, handles, and page-ref mappings match the shared contract.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceContractFixtureTests.cs

EvidenceCoverageMeasureTests.csTest ledger-derived coverage-v2 +195/-0

Test ledger-derived coverage-v2

• Covers every omission, continuation, body, source, stop, one-shot, and iteration-cap calculation.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceCoverageMeasureTests.cs

EvidenceFirstViewReaderTests.csTest strategy first-view reading +89/-0

Test strategy first-view reading

• Verifies seeded pages, guidance, omitted sections, size limits, transient errors, and scope loss.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceFirstViewReaderTests.cs

EvidenceOrientationBuilderTests.csTest bounded evidence orientation +90/-0

Test bounded evidence orientation

• Covers manifest, outline, summary, unavailable-index, page integrity, and budget behavior.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceOrientationBuilderTests.cs

EvidencePageRendererTests.csTest citable page rendering +131/-0

Test citable page rendering

• Verifies handles, cursor removal, continuations, refs, canonical bodies, and exact UTF-8 spans.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidencePageRendererTests.cs

EvidencePreambleHashesTests.csTest evidence prompt resource hashes +55/-0

Test evidence prompt resource hashes

• Pins exact embedded resource selection and deterministic SHA-256 treatment hashes.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidencePreambleHashesTests.cs

EvidenceRetrospectiveInputsTests.csTest bounded retrospective inputs +103/-0

Test bounded retrospective inputs

• Covers citation ordering, excerpt limits, turn fallbacks, unreadable refs, summaries, and moved scopes.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceRetrospectiveInputsTests.cs

EvidenceRouteChoiceTests.csTest one-shot fit routing +107/-0

Test one-shot fit routing

• Exercises advertised and token limits, lower-bound rejection, full assembly, and read failures.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceRouteChoiceTests.cs

EvidenceRunContextTests.csTest secure run-directory lifecycle +126/-0

Test secure run-directory lifecycle

• Covers owner-only modes, retained facts, disposal, stale sweeping, symlinks, caps, and failure handling.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceRunContextTests.cs

EvidenceScopeClientTests.csTest scope resolution and renewal +181/-0

Test scope resolution and renewal

• Covers server-relative deadlines, headroom boundaries, pagination, revocation, version changes, timeouts, and cancellation.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceScopeClientTests.cs

EvidenceTraceAssemblerTests.csTest canonical trace assembly +72/-0

Test canonical trace assembly

• Verifies source ordering, deferred bodies, size cutoffs, escaping, paging, and no-fit behavior.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceTraceAssemblerTests.cs

EvidenceWireMirrorTests.csTest evidence wire DTO mirrors +92/-0

Test evidence wire DTO mirrors

• Pins JSON field names, required values, null handling, and fixture compatibility.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceWireMirrorTests.cs

JudgeCiteHandlesTests.csTest cite-handle expansion +51/-0

Test cite-handle expansion

• Covers handle forms, delivered literal refs, deduplication, unknown handles, and citation caps.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/JudgeCiteHandlesTests.cs

JudgeLedgerTests.csTest ledger and run-file durability +91/-0

Test ledger and run-file durability

• Verifies round trips, latest-footer authority, torn-line recovery, delivery sets, and owner-only handoff files.

test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/JudgeLedgerTests.cs

ClaudeCliEnvelopeTests.csTest Claude failure subtype classification +45/-0

Test Claude failure subtype classification

• Covers max-budget, max-turn, generic process, and unparseable output envelopes.

test/Capacitor.Cli.Core.Tests.Unit/Harness/Claude/ClaudeCliEnvelopeTests.cs

ClaudeCliRunnerDetailedTests.csUpdate detailed runner expectations +13/-16

Update detailed runner expectations

• Adapts detailed outcome tests to preserved envelope subtypes and spend-budget failures.

test/Capacitor.Cli.Core.Tests.Unit/Harness/Claude/ClaudeCliRunnerDetailedTests.cs

EvalContextCacheTests.csTest evidence cache ownership +158/-0

Test evidence cache ownership

• Ensures cancellation, finalization, expiry, replacement, disposal, and partial cleanup failures release run resources.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalContextCacheTests.cs

EvalRunnerEvidenceTests.csTest daemon evidence phases +361/-0

Test daemon evidence phases

• Covers preparation metadata, cached clients, usage, scope loss, facts, phase budgets, mixed results, and legacy isolation.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalRunnerEvidenceTests.cs

Capacitor.Cli.Tests.Integration.csprojReference evidence test helpers +1/-0

Reference evidence test helpers

• Adds the shared helper dependency required by evidence route integration tests.

test/Capacitor.Cli.Tests.Integration/Capacitor.Cli.Tests.Integration.csproj

EvalEvidenceRouteTests.csTest end-to-end evidence evaluation +714/-0

Test end-to-end evidence evaluation

• Exercises route selection, certification, coverage-v2, failures, scope loss, obligations, ledgers, cleanup, and legacy byte identity.

test/Capacitor.Cli.Tests.Integration/EvalEvidenceRouteTests.cs

McpEvidenceJudgeToolsTests.csTest all evidence MCP tools +396/-0

Test all evidence MCP tools

• Covers request shapes, pages, budgets, admission, retries, scope movement, ledgering, malformed calls, and tool registration.

test/Capacitor.Cli.Tests.Unit/Commands/McpEvidenceJudgeToolsTests.cs

Capacitor.Tests.Helpers.csprojAdd WireMock dependency for evidence helpers +1/-0

Add WireMock dependency for evidence helpers

• Enables the shared test project to host evidence API stubs.

test/Capacitor.Tests.Helpers/Capacitor.Tests.Helpers.csproj

EvidenceServerStub.csAdd reusable evidence API stub +129/-0

Add reusable evidence API stub

• Provides scoped manifests, events, turns, bodies, summaries, catalogs, and configurable route responses for tests.

test/Capacitor.Tests.Helpers/EvidenceServerStub.cs

FakeClaudeOnPath.csExtend fake Claude harness support +22/-0

Extend fake Claude harness support

• Adds helper behavior needed to script and inspect evidence-route Claude invocations.

test/Capacitor.Tests.Helpers/FakeClaudeOnPath.cs

RecordingEvalObserver.csRecord evidence observer callbacks +36/-0

Record evidence observer callbacks

• Captures treatment, ledger, route, failure, fact, and completion events for assertions.

test/Capacitor.Tests.Helpers/RecordingEvalObserver.cs

contract.jsonAdd shared evidence contract fixture +35/-0

Add shared evidence contract fixture

• Pins coverage-v2 vocabularies, failure codes, handle grammar, canonical bodies, limits, and row refs.

test/fixtures/eval-evidence/contract.json

catalog-payload.jsonAdd strategy catalog fixture +12/-0

Add strategy catalog fixture

• Provides completion and general strategy metadata, guidance, versions, and reporting markers.

test/fixtures/eval-strategies/catalog-payload.json

derive-id.jsonAdd obligation identity vectors +7/-0

Add obligation identity vectors

• Pins stable IDs across whitespace normalization, anchors, titles, and turn refs.

test/fixtures/eval-strategies/derive-id.json

first-view-response.jsonAdd strategy first-view fixture +13/-0

Add strategy first-view fixture

• Represents a built completion view with rendered and omitted sections.

test/fixtures/eval-strategies/first-view-response.json

parse.jsonAdd obligation parsing vectors +13/-0

Add obligation parsing vectors

• Covers accepted entries and invalid members, vocabularies, titles, token sizes, citation limits, and shapes.

test/fixtures/eval-strategies/parse.json

reconcile.jsonAdd obligation reconciliation vectors +15/-0

Add obligation reconciliation vectors

• Pins certification, merging,

[Comment truncated to fit github's 65,536-char limit.]

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Deferred tool arguments disappear ✓ Resolved 🐞 Bug ≡ Correctness ⭐ New

Description

IsAbsent now correctly schedules an arguments_body read when inline arguments is JSON null,
but WriteEntry treats that same null element as present and writes it instead of the retrieved
body. This produces a successful one-shot trace containing "arguments": null, withholding the
actual tool arguments from the judge.

Code

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCanonicalContent.cs[47]

+    static bool IsAbsent(JsonElement e, string property) => e.Prop(property) is not { } v || v.IsNull;

Relevance

●●● Strong

Clear null-versus-deferred-body correctness bug; repository history accepts defensive JSON-shape
fixes.

PR-#134
PR-#546

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The updated absence check treats JSON null as requiring deferred retrieval, and the wire DTO permits
nullable arguments alongside an arguments-body descriptor. After that body is fetched, the trace
writer tests only whether the property exists, so a present null element is emitted before the
resolved-body branch can run.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCanonicalContent.cs[31-47]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[140-145]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCallEntryDto.cs[7-10]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Explicit JSON null arguments trigger deferred body retrieval but later take precedence over the retrieved content during trace rendering.

## Fix Focus Areas
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCanonicalContent.cs[31-47]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[140-145]

## Recommended Fix
In `WriteEntry`, treat a missing or JSON-null `arguments` property as deferred and prefer the resolved arguments body when available. Add coverage for a call containing both `"arguments": null` and a valid `arguments_body` descriptor.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Some session events go unjudged ✓ Resolved 🐞 Bug ≡ Correctness ⭐ New

Description

AssembleAsync breaks from a source when an events page has no entries because last remains null,
even if that response supplies next_cursor. An empty intermediate page with a continuation
consequently returns a successful, fitting trace without reading the remaining events.

Code

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[R69-72]

+                }
+                // Advance past the last event returned: the page's to_revision echoes the requested end.
+                if (doc.RootElement.Str("next_cursor") is null || last is null || last.Value == source.RevisionCutoff) break;
+                from = last.Value + 1;

Relevance

●●● Strong

Clear pagination correctness bug where continuation data is ignored; repository history accepts
defensive evaluation-path fixes.

PR-#200
PR-#134

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

An empty entries array leaves last null, and the continuation condition explicitly breaks on
that null value. The wire model independently permits an empty entries list and a non-null cursor,
after which the assembler closes the trace and reports success if its size fits.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[46-52]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[69-83]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceEventPageDto.cs[5-9]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

Issue description
An events response with an empty `entries` array and a non-null continuation is accepted as the end of a source, causing trace assembly to omit later evidence.

Fix Focus Areas
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[46-72]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceTraceAssemblerTests.cs[73-129]

Recommended Fix
When `next_cursor` is non-null, require a valid last revision that advanced within the requested range; otherwise return a failed trace read rather than ending the source. Add a test for an empty events page with a continuation and assert that assembly fails rather than reporting a fitting trace.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Deferred citations disappear ✗ Dismissed 🐞 Bug ≡ Correctness ⭐ New

Description

CertifyAsync adds work_budget refusals to resend without marking the batch as progressed, then
drops that entire list in the !progressed branch instead of putting it back in pending. A
successful certification response containing only deferred references therefore loses all of those
citations and downstream evidence coverage records them as uncertified.

Code

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs[R49-52]

+                dropped += inFlight.Count(r => !answered.Contains(r));
+                inFlight.Clear();
+                if (!progressed) { dropped += resend.Count; continue; }
+                pending.InsertRange(0, resend);

Relevance

●●● Strong

Clear retry-state loss in an all-deferred path; repository history accepts fail-closed and
malformed-response correctness fixes.

PR-#200
PR-#1016

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The implementation promises that work-budget deferrals are resent, but the all-deferred control flow
clears the only retry list before it can be queued. The existing test exercises only a mixed
certified/deferred batch, which masks this path.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs[7-9]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs[43-52]
src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[248-257]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceCitationClientTests.cs[31-47]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

Issue description
`work_budget` is a retryable response, but a batch made entirely of such responses has no `progressed` entry and is dropped before its refs can be requeued.

Fix Focus Areas
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs[43-52]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceCitationClientTests.cs[31-47]

Recommended Fix
Treat retryable `work_budget` entries as sufficient progress to reinsert `resend` into `pending`, while retaining a bounded/no-progress guard that only drops refs when no answer entry was returned or the shared time budget expires. Add a test where every requested ref is initially refused with `work_budget` and is certified by the following request.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View high (2)

4. Bad cursors stall evidence preparation ✓ Resolved 🐞 Bug ☼ Reliability

Description

EvidenceTraceAssembler accepts event and body pagination positions without verifying that they
advance, including overflowing last + 1 when the revision is long.MaxValue. A malformed
successful response can therefore repeat reads indefinitely, and daemon preparation supplies only
the shutdown token rather than a phase timeout.

Code

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[R67-68]

+                if (doc.RootElement.Str("next_cursor") is null || last is null) break;
+                from = last.Value + 1;

Relevance

●●● Strong

Non-advancing pagination and missing bounds can hang evaluation preparation, matching accepted
reliability fixes.

PR-#173

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The event loop continues whenever next_cursor exists and computes the next revision with unchecked
addition, while body pagination assigns next_offset directly. Evidence preparation awaits this
assembler using a token that, in the daemon preparation path, is canceled only during daemon
shutdown.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[39-68]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[90-106]
src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[62-79]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Evidence pagination trusts server cursors and offsets without proving forward progress, allowing preparation to loop indefinitely or overflow a revision.

## Fix Focus Areas
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[39-68]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[90-106]

## Recommended Fix
Before continuing, require the last event revision to be within the requested range and strictly advance `from`, and require `next_offset` to be greater than the current offset. Convert invalid pagination into a controlled failed trace rather than issuing another request.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Cancelling an evaluation breaks judging ✓ Resolved 🐞 Bug ☼ Reliability

Description

EvalContextCache.Remove immediately disposes an EvidenceRunSetup, and Touch can still return
that setup after its refresh loses a removal race. Concurrent cancellation, finalization, expiry,
replacement, or idle sweeping can therefore delete the active run directory and dispose the shared
HTTP client while question or finalize handlers—and Claude's MCP process—still need the run file,
ledger, and network resources.

Code

src/Capacitor.Cli.Daemon/Services/EvalContextCache.cs[R40-42]

+    public void Remove(string evalRunId) {
+        if (_entries.TryRemove(evalRunId, out var entry)) Release(evalRunId, entry);
+    }

Relevance

●●● Strong

Disposal races affecting active operations match the team’s recent acceptance of lifecycle
concurrency fixes.

PR-#508
PR-#1096

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Touch ignores a failed TryUpdate and returns the old context, while Remove synchronously
disposes the setup and its owned HTTP client. SignalR cancellation independently removes the cache
entry without canceling the active operation, even though question and finalization handlers retain
the returned setup throughout their work and use its context for run files, ledgers, and MCP
configuration; disposing the setup recursively deletes that context directory.

src/Capacitor.Cli.Daemon/Services/EvalContextCache.cs[40-42]
src/Capacitor.Cli.Daemon/Services/EvalContextCache.cs[66-75]
src/Capacitor.Cli.Daemon/Services/EvalContextCache.cs[85-94]
src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[222-245]
src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[274-290]
src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[324-329]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunContext.cs[102-113]
src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[184-199]
src/Capacitor.Cli.Daemon/Services/EvalContextCache.cs[84-94]
src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[188-231]
src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[326-331]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunContext.cs[103-109]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Evidence cache entries can be removed and disposed while question or finalization handlers still retain and use their setups. Removal can delete the MCP run file and ledger directory and dispose the owned HTTP client during an active judge invocation, while `Touch` may return a stale setup when its compare-and-swap loses to removal.

## Fix Focus Areas
- src/Capacitor.Cli.Daemon/Services/EvalContextCache.cs[30-94]
- src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[188-245]
- src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[274-290]
- src/Capacitor.Cli.Daemon/Services/EvalRunner.cs[326-331]

## Recommended Fix
Introduce an operation lease or reference count for cached evidence setups. Acquiring a setup for a question or finalize phase must hold a lease until that operation completes; removal, expiry, replacement, and cache disposal should mark the entry unavailable to new work and signal cancellation where appropriate, but defer `EvidenceRunSetup.DisposeAsync()`, directory deletion, and owned-client disposal until all active leases are released. Coordinate cancellation with the active phase token if cancellation is intended to stop active work, and make `Touch` retry or fail when its compare-and-swap loses rather than returning the stale context.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. Seven test summaries duplicate cases ✓ Resolved 📘 Rule violation ⚙ Maintainability ⭐ New

Description

The <summary> blocks on seven test classes enumerate behavior already expressed by their test
method names and assertions rather than documenting a non-obvious constraint. Whenever cases are
added or changed, these parallel inventories require a second update and can leave later readers
with stale coverage claims.

Code

test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalContextCacheTests.cs[R13-15]

+/// read, the sweep, a second prepare under the same id, and host disposal with several runs cached. A leased run is cancelled
+/// when it leaves but disposed only when its last lease ends, and is never expired while leased. No test runs a question, so no
+/// harness is reachable.</summary>

Relevance

●●● Strong

Recent history repeatedly accepts removing verbose or duplicative test summaries under the comments
rule.

PR-#616
PR-#1024
PR-#385

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Compliance rule 2762993 permits comments only for non-obvious, behavior-critical constraints. Each
cited summary instead lists the outcomes covered by methods in its test class, including the newly
changed malformed-response, omitted-citation, pagination, ledger-recovery, cache-lifetime, and
cancellation cases.

Rule 2762993: Restrict comments to documenting non-obvious, behavior‑critical constraints
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceCitationClientTests.cs[6-7]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceOrientationBuilderTests.cs[6-8]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceRetrospectiveInputsTests.cs[6-7]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceTraceAssemblerTests.cs[6-9]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/JudgeLedgerTests.cs[6-7]
test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalContextCacheTests.cs[12-15]
test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalRunnerEvidenceTests.cs[13-16]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Seven test-class summaries restate the cases implemented by their methods instead of documenting non-obvious, behavior-critical constraints.

## Fix Focus Areas
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceCitationClientTests.cs[6-7]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceOrientationBuilderTests.cs[6-8]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceRetrospectiveInputsTests.cs[6-7]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceTraceAssemblerTests.cs[6-9]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/JudgeLedgerTests.cs[6-7]
- test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalContextCacheTests.cs[12-15]
- test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalRunnerEvidenceTests.cs[13-16]

## Recommended Fix
Remove these test inventory summaries. Retain or rewrite only text that explains a non-obvious constraint whose omission could cause incorrect test behavior.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

7. New JSON checks bypass shared helpers ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

ParseCitationTokens, TryParse, IsAbsent, Render, run-file readers, ledger readers, and four
tests inspect JsonElement.ValueKind directly instead of using the available IsString, IsArray,
IsObject, IsNumber, and IsNull helpers. These checks cover citations, obligations, evidence
pages, persisted runs, ledgers, and assertions, so later JSON-shape changes must now account for a
second checking style across production and test code.

Code

src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[496]

+                ? [.. citations.EnumerateArray().Where(c => c.ValueKind == JsonValueKind.String).Select(c => c.GetString()!)]

Relevance

●●● Strong

Recent JSON helper findings were accepted, including direct ValueKind comparisons in EvalService.

PR-#546
PR-#1016

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Rule 2270023 requires all changed JSON type and shape checks to use JsonElementExtensions. The
cited production parsers and tests directly compare or assert on ValueKind for strings, arrays,
objects, numbers, and null values.

Rule 2270023: Use JsonElementExtensions helpers instead of direct JsonValueKind comparisons
src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[496-496]
src/Capacitor.Cli.Core/Eval/Evidence/EvalObligationContract.cs[54-116]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCanonicalContent.cs[47-47]
src/Capacitor.Cli.Core/Eval/Evidence/EvidencePageRenderer.cs[56-62]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunFile.cs[54-54]
src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerReader.cs[37-48]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvalObligationRulesTests.cs[66-66]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidencePageRendererTests.cs[106-106]
test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceTraceAssemblerTests.cs[52-52]
test/Capacitor.Cli.Tests.Integration/EvalEvidenceRouteTests.cs[653-653]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
New production and test code directly compares `JsonElement.ValueKind` even though the shared JSON extensions expose equivalent shape helpers.

## Fix Focus Areas
- src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[496-496]
- src/Capacitor.Cli.Core/Eval/Evidence/EvalObligationContract.cs[54-116]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCanonicalContent.cs[47-47]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidencePageRenderer.cs[56-62]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRunFile.cs[54-54]
- src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerReader.cs[37-48]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvalObligationRulesTests.cs[66-66]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidencePageRendererTests.cs[106-106]
- test/Capacitor.Cli.Core.Tests.Unit/Eval/Evidence/EvidenceTraceAssemblerTests.cs[52-52]
- test/Capacitor.Cli.Tests.Integration/EvalEvidenceRouteTests.cs[653-653]

## Recommended Fix
Replace every direct `ValueKind` comparison with the corresponding `JsonElementExtensions` property, including `IsString`, `IsArray`, `IsObject`, `IsNumber`, or `IsNull`; update assertions to assert those boolean helpers directly.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

8. Malformed evidence aborts evaluations ✓ Resolved 🐞 Bug ☼ Reliability

Description

EvidenceTraceAssembler, EvidencePageRenderer, and retrospective readers parse successful server
bodies without translating JsonException into an evidence-read failure. Invalid JSON or a non-JSON
deferred arguments body therefore escapes preparation or finalization and turns a recoverable
evidence response problem into a generic evaluation failure.

Code

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[45]

+                using var doc = JsonDocument.Parse(page.Body);

Relevance

●●● Strong

Malformed successful response handling is a concrete reliability gap; defensive malformed-input
fixes are consistently accepted.

PR-#200

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The HTTP client returns successful bodies unchanged, after which multiple evidence components call
JsonDocument.Parse directly. Their orchestration callers await these operations without a local
parse-error conversion, so the exception reaches the outer generic failure handler.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceReadClient.cs[4-13]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[41-54]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[133-138]
src/Capacitor.Cli.Core/Eval/Evidence/EvidencePageRenderer.cs[23-28]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRetrospectiveInputs.cs[74-96]
src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[91-99]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Several evidence consumers parse successful HTTP bodies without handling malformed JSON, so response corruption aborts evaluation orchestration.

## Fix Focus Areas
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[41-54]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[99-106]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceTraceAssembler.cs[133-138]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidencePageRenderer.cs[23-28]
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceRetrospectiveInputs.cs[74-96]

## Recommended Fix
Catch JSON and structural parsing failures at the evidence-response boundary and return a controlled transient read failure with diagnostics. Validate deferred `arguments` JSON before assembling the trace, and apply the same handling to orientation and retrospective responses.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (2)

9. Missing certifications vanish silently ✓ Resolved 🐞 Bug ≡ Correctness

Description

EvidenceCitationClient.CertifyAsync clears the in-flight batch even when a successful response
omits some or all requested references, but increments Dropped only for entries actually returned.
An empty or partial response therefore silently loses citations and prevents the caller from
emitting its certification-loss diagnostic.

Code

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs[R45-47]

+                inFlight.Clear();
+                if (!progressed) { dropped += resend.Count; continue; }
+                pending.InsertRange(0, resend);

Relevance

●●● Strong

The finding identifies a concrete dropped-item accounting bug in newly added certification flow.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The batch is removed from pending before the request, response processing iterates only returned
entries, and then inFlight is cleared. For an empty response, progressed is false but
resend.Count is zero, so neither certification nor dropped accounting records the requested
references; the caller logs only when Dropped is positive.

src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs[26-47]
src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[248-258]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Successful certification responses that omit requested references remove those references from processing without counting them as dropped.

## Fix Focus Areas
- src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCitationClient.cs[26-47]
- src/Capacitor.Cli.Core/Eval/EvalService.Evidence.cs[248-258]

## Recommended Fix
Track the exact requested-reference set for each batch and remove only references explicitly certified, refused, or rejected by the response. Count every missing, duplicate, or unexpected response entry as a dropped requested citation, or retry it within the remaining certification budget.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

10. Crash windows undercount tool usage ✓ Resolved 🐞 Bug ≡ Correctness

Description

McpEvidenceJudgeTools.Finish flushes a completed call and its updated footer as separate ledger
records, while JudgeLedgerReader accepts calls after an older footer without invalidating that
footer. If the judge exits between those writes, coverage includes the call record but derives tool
calls, delivered bytes, stop reason, and refused sources from the stale checkpoint.

Code

src/Capacitor.Cli/Commands/McpEvidenceJudgeTools.cs[R251-253]

+    (string, bool) Finish(JudgeLedgerCall call, string text, bool isError) {
+        _ledger.Append(call);
+        _ledger.Append(new JudgeLedgerFooter(_charged, _delivered, _stop, [.. _refused], _time.GetUtcNow()));

Relevance

●●● Strong

Crash-window ledger undercount is a concrete correctness issue in newly added persistence logic.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Each append is independently serialized and flushed, so termination can persist the call without its
following footer. The reader retains both the earlier footer and later call, but JudgeLedger
always prefers footer totals, and coverage consumes those totals.

src/Capacitor.Cli/Commands/McpEvidenceJudgeTools.cs[251-254]
src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerWriter.cs[88-110]
src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerReader.cs[15-29]
src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedger.cs[39-41]
src/Capacitor.Cli.Core/Eval/Evidence/EvidenceCoverageMeasure.cs[57-65]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A crash between writing a call and its footer leaves a ledger whose final footer predates complete records that the reader still retains.

## Fix Focus Areas
- src/Capacitor.Cli/Commands/McpEvidenceJudgeTools.cs[251-254]
- src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedgerReader.cs[15-29]
- src/Capacitor.Cli.Core/Eval/Evidence/JudgeLedger.cs[39-41]

## Recommended Fix
Make reader accounting consistent with the records after the last footer: either ignore all records after the authoritative footer or recompute tool calls and delivered bytes from complete page/call records when later records exist. Add a crash-recovery test with a complete call line following an older footer.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

11. Four tests manage temporary dirs ✓ Resolved 📘 Rule violation ▣ Testability

Description

EvalContextCacheTests, EvalRunnerEvidenceTests, EvalEvidenceRouteTests, and
McpEvidenceJudgeToolsTests initialize TempDir fields with new() and dispose them manually
instead of exposing injected [TempDir] properties. TUnit creates these daemon, integration, and
command test classes, so their filesystem state bypasses the framework-managed lifecycle used by
their neighboring injected fixtures.

Code

test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalContextCacheTests.cs[20]

+    readonly TempDir _tmp = new();

Relevance

● Weak

Recent same-rule findings rejecting manual TempDir usage outweigh older accepted examples.

PR-#1138
PR-#976

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Rule 2808173 requires test classes to obtain TempDir through a public required property annotated
with [TempDir]. Each cited TUnit test class instead adds a manually constructed field and includes
it in its own disposal path.

Rule 2808173: Use injected [TempDir] public required property in test classes instead of manual fields
test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalContextCacheTests.cs[15-23]
test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalRunnerEvidenceTests.cs[17-25]
test/Capacitor.Cli.Tests.Integration/EvalEvidenceRouteTests.cs[17-25]
test/Capacitor.Cli.Tests.Unit/Commands/McpEvidenceJudgeToolsTests.cs[13-19]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Four new TUnit test classes manually construct and dispose `TempDir` fields instead of using the required framework-injected property pattern.

## Fix Focus Areas
- test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalContextCacheTests.cs[20-23]
- test/Capacitor.Cli.Daemon.Tests.Unit/Services/EvalRunnerEvidenceTests.cs[23-25]
- test/Capacitor.Cli.Tests.Integration/EvalEvidenceRouteTests.cs[23-25]
- test/Capacitor.Cli.Tests.Unit/Commands/McpEvidenceJudgeToolsTests.cs[16-19]

## Recommended Fix
Replace each manually initialized field with `[TempDir] public required TempDir Tmp { get; init; }`, update usages to `Tmp`, and remove manual disposal of the injected directory.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 64 rules

✅ Cross-repo context — repo relationships

  Explored: repo: kurrent-io/kcap-server (sha: 3d909619) — View relationship

Review mode: ⚖️ Balanced: The push makes localized but behavior-affecting changes to evidence pagination and argument resolution, with failure-state semantics and persisted trace correctness warranting a careful review.

Tip of the day

💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[realtonyyoung]

Qodo item 7 (summary-only): Fixed in 93aa9c0: EvalContextCacheTests, EvalRunnerEvidenceTests, EvalEvidenceRouteTests and McpEvidenceJudgeToolsTests now take [TempDir] public required TempDir Tmp { get; init; }, and each class disposes only its own stub and client. The kcap-cli rule: "When every test in a class needs one, inject it instead of holding a field" (CLAUDE.md, the TempDir paragraph).

[realtonyyoung]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit 80eafbf

[realtonyyoung]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit f2b68df

[realtonyyoung]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit ab6b33f