[realtonyyoung]

AI-2787 (no GitHub issue)

What & why

Users copy eval reports from the UI into Codex or Claude by hand, because no agent could ask for one. Two read tools on the kcap-sessions MCP server fix that. get_eval_report(session_id?, repo?) renders a session's latest completed eval as Markdown: the overall score and summary, each category, every question's finding, evidence and recommendation, and the retrospective's strengths, issues and suggestions. With no session it reports on the repository's most recently evaluated session. list_eval_runs(repo?, limit?) lists a repository's evaluated sessions, newest first, with their category scores, so the agent can pick one.

Where to look

The issue sketched a new kcap mcp evals server. These tools join kcap-sessions instead, so every harness kcap setup already wires gets them with no install or config change. Both read existing public routes gated by session visibility alone: /eval-progress already carries the full completed result, and /eval-trends (asked for its widest one-year window) names a repository's evaluated sessions. So the change needs no server work and works on every plan. Both tools are read-only and are added to the review-flow allowlist.

Verification

Unit tests cover the Markdown for a completed run, the state answered for each non-report case (not found, not evaluated, running, failed, unreadable), the size cap, runs grouped per session newest first, and limit validation. An integration test runs the MCP server against a stub: with no session id it reads the cwd repo's trends and reports the newest session's findings.

🤖 Generated with Claude Code

[linear-code]

AI-2787

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 001c482d-c3f5-4410-8d30-07548692013c

📥 Commits

Reviewing files that changed from the base of the PR and between f3a3067 and f74621c.

📒 Files selected for processing: 9

• README.md
• kcap/README.md
• src/Capacitor.Cli.Core/KcapMcpRegistry.cs
• src/Capacitor.Cli.Core/Resources/help-mcp.txt
• src/Capacitor.Cli/Commands/EvalReportTool.cs
• src/Capacitor.Cli/Commands/McpSessionsServer.cs
• src/Capacitor.Cli/Commands/SessionEvalsTool.cs
• test/Capacitor.Cli.Tests.Integration/McpSessionsServerTests.cs
• test/Capacitor.Cli.Tests.Unit/Commands/EvalReportToolTests.cs

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qodo-code-review]

PR Summary by Qodo

Expose eval reports and run listings through kcap-sessions

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 20-40 Minutes

AI Description

• Add read-only MCP tools to retrieve actionable eval reports and discover recently evaluated
 sessions.
• Reuse visibility-gated session and repository routes, so existing harness setups need no
 configuration changes.
• Document the tools and test report rendering, run ordering, validation, and default repository
 selection.

Diagram

graph TD
  A["Agent"] --> B["kcap-sessions"] --> C{"Session ID?"} -->|No| D["Eval trends"] --> E["Eval progress"] --> F["Markdown report"]
  C -->|Yes| E
  B -->|List runs| D
  D --> G["Runs list"]

Loading

High-Level Assessment

Adding read-only tools to the already configured kcap-sessions server is the best fit. A separate eval MCP server would require harness setup changes, while the existing visibility-gated routes supply both the report and run listing without server work.

Files changed (9) +395 / -4

Enhancement (3) +256 / -0

EvalReportTool.csDefine and format eval report and run tools +190/-0

Define and format eval report and run tools

• Defines both read-only tools, groups trend points into newest-first session runs, and renders completed evals as capped Markdown. It also returns readable states when no completed report is available.

src/Capacitor.Cli/Commands/EvalReportTool.cs

McpSessionsServer.csRoute eval tool calls through kcap-sessions +64/-0

Route eval tool calls through kcap-sessions

• Registers and handles both tools using existing eval-trends and eval-progress routes. Report calls without a session select the current repository’s latest evaluated session; handlers retain authentication and request-error responses.

src/Capacitor.Cli/Commands/McpSessionsServer.cs

SessionEvalsTool.csShare session ID validation +2/-0

Share session ID validation

• Exposes the existing session ID check for use by get_eval_report.

src/Capacitor.Cli/Commands/SessionEvalsTool.cs

Tests (2) +131 / -1

McpSessionsServerTests.csVerify MCP registration and default report lookup +32/-1

Verify MCP registration and default report lookup

• Checks that all twelve tools are advertised and that a report call without a session reads repository trends before fetching the latest session’s findings.

test/Capacitor.Cli.Tests.Integration/McpSessionsServerTests.cs

EvalReportToolTests.csCover report rendering and run selection +99/-0

Cover report rendering and run selection

• Tests completed-report content, non-report states, truncation, run grouping and ordering, limit validation, and the one-year trends URL.

test/Capacitor.Cli.Tests.Unit/Commands/EvalReportToolTests.cs

Documentation (3) +7 / -2

README.mdDocument the two eval discovery tools +2/-0

Document the two eval discovery tools

• Describes full Markdown reports and repository run listings in the main MCP tool reference.

README.md

README.mdAdd eval tools to the package tool table +2/-0

Add eval tools to the package tool table

• Lists the new report and run tools with their default-repository behavior.

kcap/README.md

help-mcp.txtUpdate MCP sessions help +3/-2

Update MCP sessions help

• Names the two new tools and changes the advertised tool count from ten to twelve.

src/Capacitor.Cli.Core/Resources/help-mcp.txt

Other (1) +1 / -1

KcapMcpRegistry.csAllow the eval tools in review flows +1/-1

Allow the eval tools in review flows

• Adds both read-only tool names to the kcap-sessions unattended-safe allowlist.

src/Capacitor.Cli.Core/KcapMcpRegistry.cs

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Failed re-evaluations hide previous reports ✗ Dismissed

Description

get_eval_report treats an /eval-progress response without completed_result as a failed
evaluation, although kcap-server retains an earlier judged result when a later run fails without
judging anything. In that case list_eval_runs still offers the session from /eval-trends, but
selecting it cannot retrieve the report it lists.

Code

src/Capacitor.Cli/Commands/EvalReportTool.cs[R126-129]

+        if (root.Obj("completed_result") is not { } result || !Items(result, "categories").Any()) {
+            if (root.Bool("is_terminal") == true || root.Obj("failed_result") is not null || root.Obj("completed_result") is not null)
+                return $"The latest eval of session {sessionId} failed: {root.Str("failure_reason") ?? "every question failed to judge"}.";
+            return $"Session {sessionId} is being evaluated; no completed report yet.";

Relevance

●●● Strong

Accepted precedent treats failure-only evaluation payloads as distinct failures; preserving prior
judged reports is a clear correctness fix.

PR-#1359

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The server stores failure-only runs separately from judged summaries, so trends continue to show the
prior score. Its latest-run progress snapshot has no completed result and the public mapping omits
the retained judged result; the new CLI renderer consequently returns a failure message for a
session listed by its other new tool.

kcap-cli -> kcap-server
src/Capacitor.Cli/Commands/EvalReportTool.cs[78-87]
src/Capacitor.Cli/Commands/EvalReportTool.cs[126-132]
External repo: kurrent-io/kcap-server, src/Capacitor.Server.Services/Evals/EvalSummariesProjectorV4.cs [53-59]
External repo: kurrent-io/kcap-server, src/Capacitor.Server.Services/Evals/EvalTrendQueries.cs [79-108]
External repo: kurrent-io/kcap-server, src/Capacitor.Server.Services/Evals/PersistedEvalRunSnapshot.cs [28-41]
External repo: kurrent-io/kcap-server, src/Capacitor.Server.Services/Evals/PersistedEvalRunSnapshot.cs [65-82]
External repo: kurrent-io/kcap-server, src/Capacitor.Api.Public/Sessions/SessionEvalProgressMapping.cs [11-25]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
After a failure-only re-evaluation, kcap-server keeps the prior judged summary in eval trends but its public eval-progress response contains only the latest failed run. The CLI lists the session as evaluated yet cannot render its retained report.
## Fix Focus Areas
- src/Capacitor.Cli/Commands/EvalReportTool.cs[126-132]
- src/Capacitor.Cli/Commands/McpSessionsServer.cs[340-345]
- /cross_repos/kcap-server/src/Capacitor.Api.Public/Sessions/SessionEvalProgressMapping.cs[11-25]
## Recommended Fix
Coordinate a public, visibility-checked way to retrieve the latest judged result from kcap-server (for example, expose its already-loaded `LatestJudgedResult` in the progress contract). Have the CLI render that result when the latest run is failure-only, and test a successful run followed by a failure-only re-evaluation.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Failed questions disappear from reports ✗ Dismissed

Description

Markdown renders questions from categories but never renders the separate failed_questions
array. When some questions are judged and others fail, the result qualifies as completed, yet the
report omits the failed questions and their failure codes.

Code

src/Capacitor.Cli/Commands/EvalReportTool.cs[R144-146]

+        foreach (var category in Items(result, "categories")) {
+            md.Append("## ").Append(category.Str("name")).Append(" (").Append(category.Num("score")?.ToString(CultureInfo.InvariantCulture) ?? "?").Append("/5)\n\n");
+            foreach (var q in Items(category, "questions")) {

Relevance

●●● Strong

The report promises every question, and accepted evaluation-rendering precedents preserve separately
represented failure details.

PR-#1257
PR-#1359

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The completed payload stores failures separately, and aggregation counts them alongside judged
questions. The new report loop visits only category questions.

src/Capacitor.Cli.Core/Eval/Contracts/SessionEvalCompletedPayloadV4.cs[22-24]
src/Capacitor.Cli.Core/Eval/EvalService.cs[1507-1534]
src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionFailure.cs[7-18]
src/Capacitor.Cli/Commands/EvalReportTool.cs[126-132]
src/Capacitor.Cli/Commands/EvalReportTool.cs[144-156]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Completed evals can contain both judged category questions and separate failed questions. The report currently omits the latter.
## Fix Focus Areas
- src/Capacitor.Cli/Commands/EvalReportTool.cs[144-156]
- src/Capacitor.Cli.Core/Eval/Contracts/SessionEvalCompletedPayloadV4.cs[22-24]
## Recommended Fix
Render `failed_questions` with their category, question ID and failure code, and add a mixed-success-and-failure report test.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. JSON checks bypass shared helpers ✗ Dismissed

Description

The new Items method directly compares e.ValueKind with JsonValueKind.Object and
JsonValueKind.String instead of using the available JsonElementExtensions predicates. Every
report and trend parsing path that calls Items therefore retains direct JSON shape checks in the
changed code.

Code

src/Capacitor.Cli/Commands/EvalReportTool.cs[R188-189]

+    static IEnumerable<JsonElement> Items(JsonElement parent, string name, bool objects = true) =>
+        parent.Arr(name) is { } array ? array.EnumerateArray().Where(e => objects ? e.ValueKind == JsonValueKind.Object : e.ValueKind == JsonValueKind.String) : [];

Relevance

●●● Strong

A recent accepted precedent explicitly replaces direct ValueKind comparisons with shared
JsonElementExtensions helpers.

PR-#1559

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changed Items helper contains direct JsonElement.ValueKind comparisons for object and string
checks, while Compliance ID 2270023 requires equivalent JsonElementExtensions helpers for these
checks.

Rule 2270023: Use JsonElementExtensions helpers instead of direct JsonValueKind comparisons
src/Capacitor.Cli/Commands/EvalReportTool.cs[188-189]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The `Items` helper directly compares `JsonElement.ValueKind` against `JsonValueKind` values, violating the project convention that equivalent `JsonElementExtensions` helpers must be used.

## Fix Focus Areas
- src/Capacitor.Cli/Commands/EvalReportTool.cs[188-189]

## Recommended Fix
Replace the direct `ValueKind` comparisons with the corresponding `JsonElementExtensions` methods, such as `IsObject()` and `IsString()`, while preserving the existing array filtering behavior.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Reports omit obligation findings ✗ Dismissed

Description

Markdown emits each question's finding, evidence and recommendation but ignores its obligations
and obligations_not_reported fields. For evaluations that reconcile individual obligations, the
report leaves out their statuses, notes and citations, as well as any indication that obligations
could not be reported.

Code

src/Capacitor.Cli/Commands/EvalReportTool.cs[R151-153]

+                Line(md, "Finding", q.Str("finding"));
+                Line(md, "Evidence", q.Str("evidence"));
+                Line(md, "Recommendation", q.Str("recommendation"));

Relevance

●●● Strong

The renderer omits contract-defined obligation details, conflicting with the report’s stated purpose
and accepted structured-rendering expectations.

PR-#1257

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The question schema defines obligation results and an explicit loss indicator; each obligation has
its own status and citations. The new per-question rendering exhausts its output after three text
fields.

src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionAssessment.cs[43-50]
src/Capacitor.Cli.Core/Eval/Contracts/EvalObligationResult.cs[7-18]
src/Capacitor.Cli/Commands/EvalReportTool.cs[146-154]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Question assessments can contain separately reconciled obligations, but the Markdown report silently discards them.
## Fix Focus Areas
- src/Capacitor.Cli/Commands/EvalReportTool.cs[146-154]
- src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionAssessment.cs[43-50]
- src/Capacitor.Cli.Core/Eval/Contracts/EvalObligationResult.cs[7-18]
## Recommended Fix
Render each obligation's title, status, note and citations beneath its question. Also surface `obligations_not_reported` when present, with a test for both cases.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (1)

5. Readers cannot distinguish skipped questions ✗ Dismissed

Description

Markdown labels every question without a score as not scored and omits its outcome. When the
judge returns insufficient_evidence or not_applicable, both distinct outcomes become the same
label in the report.

Code

src/Capacitor.Cli/Commands/EvalReportTool.cs[R147-150]

+                md.Append("- **").Append(q.Str("question_id")).Append("**: ")
+                  .Append(q.Num("score") is { } score ? $"{score}/5" : "not scored");
+                if (q.Str("verdict") is { } verdict) md.Append(", ").Append(verdict);
+                md.Append('\n');

Relevance

●●● Strong

Outcome is distinct contract data; exposing insufficient-evidence and not-applicable states is a
straightforward correctness fix.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The question contract carries an outcome independent of score, and validation requires a null score
for non-assessed outcomes. The renderer uses that null score without displaying the outcome.

src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionAssessment.cs[18-25]
src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionAssessment.cs[69-80]
src/Capacitor.Cli/Commands/EvalReportTool.cs[147-150]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Non-assessed questions have distinct outcomes, but the report presents all of them as `not scored`.
## Fix Focus Areas
- src/Capacitor.Cli/Commands/EvalReportTool.cs[147-150]
- src/Capacitor.Cli.Core/Eval/Contracts/EvalQuestionAssessment.cs[18-25]
## Recommended Fix
Include the question's `outcome` in its Markdown heading, retaining the score and verdict for assessed questions. Test both `insufficient_evidence` and `not_applicable`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

⚠️ Tickets: not configured — No jira, linear integration is connected for the linked tickets

✅ Compliance rules (platform): 59 rules

✅ Cross-repo context — repo relationships

  Explored: repo: kurrent-io/kcap-server (sha: 4155f5b1) — View relationship

Review mode: Auto: ⚖️ Balanced: New public MCP tools add HTTP, parsing, rendering, and visibility-sensitive behavior.

Tip of the day

💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗