[rohitg00]

Summary

Fixes #796 (@RichardWang11 couldn't reproduce P@5 = 0.578 / 0.267 from the scorecard).

The original numbers in docs/benchmarks/2026-05-20-coding-agent-life-v1.md were generated before PR #562 changed the P@K denominator from topK.length to k. On the current scoring formula in eval/runner/score.ts, those numbers are mathematically impossible.

At K=5, with 12 single-gold and 3 double-gold questions, P@5 ceiling = (12 * 1/5 + 3 * 2/5) / 15 = 0.240. The published 0.578 / 0.267 numbers can't be reached under the current formula.

What changed

• Re-scored on main at v0.9.26: hybrid P@5 = 0.240, R@5 = 1.000, hit rate 15/15, p50 14ms
• grep baseline: P@5 = 0.227, R@5 = 0.967 (misses 1 of 2 gold on the temporal question)
• Added a Math ceiling on this dataset section so future repros land on the same numbers
• Replaced per-question-type table with both adapters' P@5 + R@5 columns; differentiator is temporal (grep 1/2, hybrid 2/2)
• Recall, hit rate, latency, and the qualitative finding (hybrid edges grep on recall + per-type, ties on aggregate at this corpus size) are unchanged

Why this is honest about the benchmark

Aggregate P@5 saturates at 0.240 on this corpus, so it can't differentiate top-tier adapters. The real signal is R@5 + per-type P@5. The benchmark is small (15 questions) and gold-sparse — tuned for fast iteration on the retrieval stack, not headline P@K claims. Harder corpora (paraphrased queries, distractor sessions, longer chains, LongMemEval _s) are the right next scorecard.

Test plan

• Doc-only change, no source/tests touched
• Math verified against eval/runner/score.ts (P@K = hits / k averaged across questions)
• Numbers verified against 2026-06-03 re-run on v0.9.26 main

Summary by CodeRabbit

• Documentation
  • Updated benchmark report with the latest performance metrics for the search service.
  • Added evaluation of temporal query capability, demonstrating improved retrieval for time-sensitive searches.
  • Simplified performance results to focus on core search adapters.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agentmemory	Ready	Preview, Comment	Jun 3, 2026 9:52am

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1f5ead94-97ad-496e-8b61-488b72ecc905

📥 Commits

Reviewing files that changed from the base of the PR and between 7446bde and e468a2c.

📒 Files selected for processing (1)

• docs/benchmarks/2026-05-20-coding-agent-life-v1.md

---

📝 Walkthrough

Walkthrough

The benchmark report for the coding-agent-life-v1 evaluation is updated from v0.9.21 to v0.9.26. Headline metrics, per-adapter results, and per-question-type performance narratives are revised to reflect re-scored results and a mathematical ceiling context. Methodology and reproduction steps remain unchanged.

Changes

Benchmark Scorecard Update

Layer / File(s)

Summary

Revised coding-agent-life-v1 benchmark results and narrative docs/benchmarks/2026-05-20-coding-agent-life-v1.md

Benchmark header updated to v0.9.26, headline metrics revised to P@5/R@5 ceiling values, per-adapter table simplified to grep and agentmemory-hybrid with updated scores, and per-question-type section rewritten to show near-saturation across types with temporal queries highlighted as the key performance differentiator where grep R@5 = 0.50 and hybrid R@5 = 1.00. Methodology and reproduction sections retained unchanged.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Poem

🐰 A scorecard hops to v0.9.26,
With metrics now fixed and precise,
Temporal queries reveal the mix—
Grep hops, but hybrid thinks twice! 🔍

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/796-benchmark-scorecard-correction

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}