Skip to content

feat(benchmarks): LongMemEval failure-mode diagnosis harness + judge quota preflight - #183

Merged
jack-arturo merged 3 commits into
developfrom
feat/longmemeval-failure-diagnosis
Jun 11, 2026
Merged

feat(benchmarks): LongMemEval failure-mode diagnosis harness + judge quota preflight#183
jack-arturo merged 3 commits into
developfrom
feat/longmemeval-failure-diagnosis

Conversation

@jack-arturo

@jack-arturo jack-arturo commented Jun 11, 2026

Copy link
Copy Markdown
Member

Summary

  • tests/benchmarks/longmemeval/diagnose_failures.py: two-stage classifier for failed-but-retrieved questions (is_correct=false AND recall_hit_at_5=true). Stage 1 is pure code (answer rank, abstention-despite-hit, stale-candidate-above-answer, noise ratio, date-math detection → documented 8-rule mode ladder); stage 2 (--llm) labels each failure with the pinned judge and reports a stage-1/stage-2 agreement matrix (transport errors counted and excluded).
  • tests/benchmarks/judge_preflight.py: one minimal judge call before any judged run; exits non-zero with an actionable message on 429/insufficient_quota or auth failure. Wired into test-longmemeval-benchmark.sh (only when --llm-eval). Would have caught the June 6 quota-compromised runs before they started.
  • Harness instrumentation: retrieved_session_ids_full (all 10 recalled memories + scores) recorded per question — closes the rank-6-10 blind spot for future runs.

Findings on the canonical run (87.0% accuracy, recall@5 97.2%)

58 of 69 failures had the answer retrieved in the top 5. LLM labels (judge gpt-5.4-mini-2026-03-17): answer-construction 42 (27 of them literal "I don't know" abstentions with the answer in context), missing-date-use 7, ranking 4, conflict-resolution 2, retrieval-gap 2, outdated-fact 1. This reprioritized the release: the harness answer-assembly path (see feat/date-aware-ranking) is the biggest benchmark lever; pure ranking fixes are the production-quality lever.

Report artifact: benchmarks/results/failure_modes_canonical_llm_20260611.json (local, gitignored).

Testing

31 new tests (synthetic fixtures, mocked clients, no network); full suite 518 passed, 12 skipped; black + flake8 clean.

Refs #158, #159 (both issues require this failure-mode classification as their first acceptance criterion).

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings June 11, 2026 01:45
@jack-arturo jack-arturo mentioned this pull request Jun 11, 2026
Merged
Copilot AI reviewed Jun 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a LongMemEval “failed-but-retrieved” diagnosis harness (for #158/#159) plus a lightweight judge quota/auth preflight to prevent wasted benchmark runs when the pinned judge is unavailable.

Changes:

  • Record retrieved_session_ids_full (rank-ordered session_id + score for all recalled memories) in LongMemEval result artifacts to remove the rank-6–10 visibility gap.
  • Introduce tests/benchmarks/longmemeval/diagnose_failures.py (stage-1 deterministic evidence + optional stage-2 judge labeling with an agreement matrix) and comprehensive unit tests.
  • Add tests/benchmarks/judge_preflight.py and wire it into test-longmemeval-benchmark.sh when --llm-eval is enabled.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/benchmarks/longmemeval/test_longmemeval.py Adds retrieved_session_ids_full to persisted results and implements helper to capture session IDs with scores.
tests/benchmarks/longmemeval/diagnose_failures.py New two-stage failure-mode diagnosis CLI (deterministic evidence + optional LLM labeling).
tests/benchmarks/longmemeval/test_diagnose_failures.py New unit tests covering stage-1 evidence/labeling, stage-2 parsing/errors, and judge preflight behavior.
tests/benchmarks/judge_preflight.py New minimal judge call preflight with classified exit codes and actionable messages.
test-longmemeval-benchmark.sh Runs judge preflight before benchmark ingestion/question loop when --llm-eval is requested.

Comment thread test-longmemeval-benchmark.sh Outdated
Comment on lines +276 to +279
if ! (cd "$SCRIPT_DIR" && "${PREFLIGHT_CMD[@]}"); then
echo -e "${RED}Judge preflight failed — aborting before the question loop${NC}"
exit 1
fi

@jack-arturo jack-arturo Jun 11, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserved the judge preflight exit status by capturing $? in the failure branch and exiting with it — test-longmemeval-benchmark.sh:276. GitHub marked the original line outdated after the push.

jack-arturo added a commit that referenced this pull request Jun 11, 2026
@jack-arturo @claude
feat(recall): date-aware ranking + latest-fact selection (#158, #159) (…
a6ed945 
…#187)

## Summary
Stacked on #186. Driven by the failure-mode diagnosis in #183 (58
failed-but-retrieved LongMemEval questions: answer-construction 42,
missing-date-use 7, ranking 4).

Server (production + benchmark):
- **Timestamp tiebreak** (always-on): exact score ties now order
newest-first deterministically (`_score_sort_key`).
- **`recency_bias=auto|on|off`** (ships `off` via
`RECALL_RECENCY_BIAS`): after dedup/state-filter and before the adaptive
floor, candidate timestamps are min-max normalized and
`SEARCH_WEIGHT_TEMPORAL` (default 0.1) × relative recency is added — so
the newest version of a conflicting fact can outrank an older, heavier
one. `auto` triggers on temporal intent ("latest", "current", "what
changed", …; word-boundaried, "currency"/"nowhere" safe).
- **Supersession chain-walk**: `current` state mode now resolves
INVALIDATED_BY/EVOLVED_INTO chains to their head (A→B→C surfaces C,
provenance still points at A; depth-bounded at 5, cycle-safe, batched).
*Honesty note: benchmark corpora carry no supersession edges — this is a
production-correctness fix, not a score mover.*

Harness (benchmark-only, flag-gated for methodology reproducibility):
- **`temporal_answer_hint`** config flag (default off; new
`temporal-answer` preset for A/B): chronological memory rendering with
scores + conflict-recency guidance + anti-overabstention guidance (27 of
the 58 failures were literal "I don't know" with the answer retrieved;
abstention remains possible). Flag-off prompt is byte-identical to the
canonical methodology (equality-tested).

## Testing
40 new tests (tiebreak, bias flip with the shipped default weight,
auto-detection, chain-walk depth/cycle/query-count guards, #158
preference latest-wins acceptance, prompt byte-identity). Full suite 563
passed, 12 skipped; black + flake8 clean.

## Validation plan before enabling anything
Lab IR A/B → automem-evals 22-probe zero-delta gate (recency_bias=off
default keeps it zero-delta) → LongMemEval mini judge-off (recall@5 ≥
97.2% floor) → judged mini → full run; `temporal-answer` preset A/B for
the harness flag, with server-vs-harness deltas reported separately in
EXPERIMENT_LOG.

Refs #158, #159

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
jack-arturo and others added 3 commits June 11, 2026 21:05
@jack-arturo @claude
feat(benchmarks): add LongMemEval failure-mode diagnosis harness and …
21399af 
…judge preflight

Implements PR-2 of the LongMemEval failure-diagnosis plan (issues #158/#159):

- tests/benchmarks/longmemeval/diagnose_failures.py: two-stage CLI that
  classifies failed-but-retrieved questions (is_correct=false AND
  recall_hit_at_5=true). Stage 1 is pure code: joins failures to dataset
  haystack sessions/dates and emits per-question evidence (answer_rank,
  abstained_despite_hit, stale_candidate_above_answer, noise_ratio,
  date_arithmetic_needed, answer_coverage_top5) plus a deterministic
  suggested_mode documented in the module docstring. Stage 2 (--llm) asks
  the pinned benchmark judge for an independent label and records the
  stage1-vs-stage2 agreement matrix. Default type filter is all types
  (58 questions on the canonical full run; the four weak categories
  cover 54 of them).

- tests/benchmarks/judge_preflight.py: one minimal completion against the
  pinned judge model before benchmark runs; exit 0 ok, 2 quota/429,
  3 auth, 1 other, with actionable one-line messages. Wired into
  test-longmemeval-benchmark.sh before the question loop when --llm-eval
  is set, so quota exhaustion aborts before any ingestion work.

- test_longmemeval.py instrumentation: additive details key
  retrieved_session_ids_full with ALL recalled memories' session ids in
  rank order and per-memory score (existing top-5-unique key unchanged).

- tests/benchmarks/longmemeval/test_diagnose_failures.py: synthetic-only
  unit tests for evidence extraction, the suggested_mode heuristic, the
  type filter, stage-2 with a mocked client, and preflight error
  classification. No network calls in tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jack-arturo @claude
fix(benchmarks): exclude stage-2 transport errors from agreement metrics
931451c 
Review fixes for the failure-diagnosis harness:

- run_stage2: records with llm_error set (transport/parse failures) no
  longer pollute the agreement matrix or exact_agreement; they are
  counted in a new agreement.llm_errors field and reported in the
  stdout summary. Per-record llm_error is unchanged.
- Import _result_details from analyze_results instead of duplicating it
  (module is side-effect-free at import).
- Comment the deliberate over-trigger in _DATE_MATH_RE (stage 2
  cross-checks it).
- main(): only print the exact-agreement line when a rate exists (no
  stray blank line).
- _session_ids_with_scores docstring: note the full ranked pool is
  recorded for future rank-6-10 depth analysis, not yet consumed by
  diagnose_failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jack-arturo
fix(benchmarks): preserve judge preflight exit status
bbabb3b
@jack-arturo
jack-arturo force-pushed the feat/longmemeval-failure-diagnosis branch from c526bd4 to bbabb3b Compare June 11, 2026 19:06
@jack-arturo
jack-arturo changed the base branch from main to develop June 11, 2026 19:06
@jack-arturo
jack-arturo merged commit f99bece into develop Jun 11, 2026
5 checks passed
@jack-arturo
jack-arturo deleted the feat/longmemeval-failure-diagnosis branch June 11, 2026 19:06
This was referenced Jun 12, 2026
Merged
Closed
Closed
jack-arturo added a commit that referenced this pull request Jun 12, 2026
@jack-arturo
feat(recall): ranking release — recency config, tag-score cap, releva…
337fe98 
…nce gate, date-aware ranking (#182, #193, #186, #187, #183, #184, #188) (#194)

## Release: ranking & recall series (develop → main)

⚠️ **Merge with a MERGE COMMIT — do not squash.** release-please needs
the individual conventional commits below to compute the version and
changelog for PR #154.

### What's in this release

| PR | Change | Default behavior |
|---|---|---|
| #182 | `feat(recall)`: configurable recency decay window/curve |
unchanged (env-gated) |
| #193 (replaces #185) | `feat(recall)`: tag-score denominator cap fixes
query-length bias | unchanged (`SEARCH_TAG_SCORE_TOKEN_CAP=0`) |
| #186 | `fix(recall)`: relevance gate — query-independent scoring gated
on topical evidence (#130) | unchanged (gate off) |
| #187 | `feat(recall)`: date-aware ranking,
`recency_bias=off\|on\|auto`, latest-fact selection (#158, #159) |
`RECALL_RECENCY_BIAS=off`; adds deterministic timestamp tiebreak for
near-ties |
| #183 | `feat(benchmarks)`: failure-mode diagnosis harness + judge
quota preflight | tooling only |
| #184 | `fix(mcp)`: surface stored metadata + `updated_at` in detailed
recall format (#111) | additive |
| #188 | `feat(enrichment)`: classification fallback-rate metrics in
`/enrichment/status` | additive |

Plus: CI now runs on `develop` pushes/PRs; benchmark experiment log +
README contribution-policy note.

### Verification evidence

- **Unit/lint/npm**: 625 pytest + 16 mcp-sse-server tests green on
develop head; CI green.
- **Default-preserve**: recall-lab baseline on the 10k-memory production
snapshot — develop defaults vs main pooled baseline identical aggregates
(R@5 0.655 / R@10 0.710 / MRR 0.434 / NDCG@10 0.501). Two-stack probe
run (main vs develop, defaults): 11/12 preserve-exact, remaining diffs
are near-tie reorders (top-1 score deltas ≤ 5.4e-5, the #187 timestamp
tiebreak).
- **Full judged 500q LongMemEval** (ship config:
`RECALL_RECENCY_BIAS=auto` + `temporal-answer` harness): recall@5 96.6%
(483/500), accuracy 86.0% (430/500), `judge_errors=0`,
`memory_ingest_failures=0`.
- **Churn attribution** (targeted re-runs of all 17 churned questions on
current-main-at-defaults and develop-at-defaults): 15/17 moved with #191
(already on main) — the April canonical 97.2% floor is stale; current
main measures ~97.0%. Develop-at-defaults differs from current main by
**1 question in 500** (a near-tie rank-5/6 flip from #187's
deterministic tiebreak). Accuracy is within answerer replicate noise
(identical-config reference runs flip 28/500 answers).
- Full detail: `benchmarks/EXPERIMENT_LOG.md` (2026-06-11 entry) and
`benchmarks/results/lme_churn17_*` + `analyze_churn17.py`.

### Opt-in features shipped OFF

`RECALL_RELEVANCE_GATE` (validated at 0.40 on lab corpus; improves
negative-probe precision) and `RECALL_RECENCY_BIAS=auto` (current-state
query re-ranking). Neither affects default behavior; see
`docs/ENVIRONMENT_VARIABLES.md`.

### After merging

release-please will update PR #154 (v0.16.0); merging *that* cuts the
tag and publishes the `:stable` image — the actual user-facing deploy
event for Railway template users.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@jack-arturo jack-arturo mentioned this pull request Jun 11, 2026
Merged
@jack-arturo jack-arturo mentioned this pull request Jun 26, 2026
Open
3 tasks
jack-arturo added a commit that referenced this pull request Jun 26, 2026
@jack-arturo
chore(main): release 0.16.0 (#154)
8c67c16 
🤖 I have created a release *beep* *boop*
---


##
[0.16.0](v0.15.2...v0.16.0)
(2026-06-26)


### Features

* **api:** add admin backup endpoint
([#162](#162))
([8b1f264](8b1f264))
* **api:** support bulk memory associations
([1221e36](1221e36))
* **api:** support bulk memory associations
([#198](#198))
([28eb916](28eb916))
* **benchmarks:** LongMemEval failure-mode diagnosis harness + judge
quota preflight
([#183](#183))
([f99bece](f99bece))
* **consolidation:** expose cluster threshold and min size as env vars
([#163](#163))
([7e731f3](7e731f3))
* **enrichment:** expose classification fallback-rate metrics in
/enrichment/status
([#188](#188))
([0b522a9](0b522a9))
* **entity:** harden identity cleanup and repair tooling
([#176](#176))
([827dfbc](827dfbc))
* **eval:** recall-quality optimization harness — lab foundation +
design ([#197](#197))
([431433e](431433e))
* **graph:** support unbounded visualizer snapshots
([#141](#141))
([c730128](c730128))
* **lab:** add aged labelled distractor injection
([cc5d546](cc5d546))
* **lab:** add config_complexity simplicity metric
([dfb10d9](dfb10d9))
* **lab:** add distractor_rate_at_k precision guardrail metric
([872eab2](872eab2))
* **lab:** add lab_corpus with parameterized recall
([5e1e071](5e1e071))
* **lab:** add pick_winner scorecard decision rule
([3187eac](3187eac))
* **lab:** add real consolidation pass helper
([48a7d4a](48a7d4a))
* **lab:** isolate production clone restores
([#171](#171))
([aef90c0](aef90c0))
* **lab:** wire scorecard, distractors, recall params, consolidation
into runner
([589ec30](589ec30))
* **recall:** add metadata sidecar search
([#177](#177))
([4e7956e](4e7956e))
* **recall:** add state_mode=current|history recall alias
([#173](#173))
([b1df86c](b1df86c))
* **recall:** cap tag-score denominator to fix query-length bias
([#193](#193))
([cefa516](cefa516))
* **recall:** date-aware ranking + latest-fact selection
([#158](#158),
[#159](#159))
([#187](#187))
([a6ed945](a6ed945))
* **recall:** make recency decay window and curve configurable
([#182](#182))
([dbb933f](dbb933f))
* **recall:** ranking release — recency config, tag-score cap, relevance
gate, date-aware ranking
([#182](#182),
[#193](#193),
[#186](#186),
[#187](#187),
[#183](#183),
[#184](#184),
[#188](#188))
([#194](#194))
([337fe98](337fe98))
* **scripts:** safer reclassify_with_llm.py with provider flags +
tighter prompt
([#164](#164))
([a742602](a742602))


### Bug Fixes

* **api:** address copilot review on PR
[#198](#198)
([0466a1e](0466a1e))
* **api:** handle grouped association write failures
([cd93df9](cd93df9))
* **backup:** make backup_automem.py runnable as `python
scripts/backup_automem.py`
([#175](#175))
([edd9742](edd9742))
* **benchmarks:** add publication verification bundle
([#166](#166))
([420d721](420d721))
* **consolidation:** skip eager first tick at startup to avoid FalkorDB
load race
([#165](#165))
([1b812cf](1b812cf))
* **docs:** keep dispatch payload arrays stable
([df6e9e8](df6e9e8))
* **embedding:** fall back to per-item real embeddings before
placeholders in batch path
([#189](#189))
([6e9c62c](6e9c62c))
* **entity:** restore person-shape exemption on the slug validation path
([#179](#179))
([5e29960](5e29960))
* **entity:** stop validator over-rejecting real people, code tools, and
event categories
([#178](#178))
([193b730](193b730))
* **lab:** address copilot review on PR
[#197](#197)
([45f80d6](45f80d6))
* **lab:** align scorecard key contract (build_scorecard -&gt;
pick_winner)
([7d91530](7d91530))
* **mcp-sse:** decouple /health liveness from upstream readiness
([#151](#151))
([5bcfb8b](5bcfb8b))
* **mcp:** cap association failure summary
([ea4e08f](ea4e08f))
* **mcp:** surface stored metadata and updated_at in detailed recall
format ([#184](#184))
([230416e](230416e))
* **recall:** address copilot review on PR
[#194](#194)
([50b1647](50b1647))
* **recall:** canonicalize / and : separators in context_tag matching
([3afd9d3](3afd9d3))
* **recall:** canonicalize / and : separators in context_tag matching
([#203](#203))
([ba5e9ff](ba5e9ff))
* **recall:** gate query-independent scoring on topical evidence within
tag scope
([#130](#130))
([#186](#186))
([c11b594](c11b594))
* **recall:** hydrate semantic recall summaries
([#192](#192))
([76e845d](76e845d))
* **recall:** normalize graph keyword scores into the 0-1 component
range ([#191](#191))
([3653ddf](3653ddf))
* **recall:** respect current memory state
([#170](#170))
([ed36b98](ed36b98)),
closes [#169](#169)
[#158](#158)
[#159](#159)
* **scripts:** add sys.path guard to reembed_embeddings.py
([d333cf0](d333cf0))


### Documentation

* add scripts catalog, recall-quality-lab guide, and 0.16.0 migrations
([f20c664](f20c664))
* **bench:** log full judged 500q LongMemEval ship-config run with churn
attribution
([41bf8d0](41bf8d0))
* **eval:** Plan A — lab metric foundation (TDD, 9 tasks)
([0087dda](0087dda))
* **eval:** Plan B — parallel matrix harness (TDD, 9 tasks)
([c8ddfb2](c8ddfb2))
* **evals:** mark Memora/FAMA/WRIT lifecycle diagnostics as
diagnostic-only
([#174](#174))
([e8a3285](e8a3285))
* **eval:** spec for recall-quality optimization harness
([b1a1995](b1a1995))
* fix stale claims and document gated flags for 0.16.0
([b152d64](b152d64))
* note develop-branch contribution policy in README
([ccf02dd](ccf02dd))
* **positioning:** add scout reference
([#168](#168))
([922d23b](922d23b))
* refresh benchmark currency for the neutral AMB run and prune stale
archive docs
([3ff95bd](3ff95bd))
* refresh benchmark currency for the neutral AMB run and prune stale
archive docs
([#204](#204))
([89c30e0](89c30e0))
* refresh README and benchmark guidance
([#157](#157))
([bba31cc](bba31cc))
* **runtime:** align Docker viewer paths and setup guidance
([#155](#155))
([bbda79b](bbda79b))
* scripts catalog, recall quality lab guide, and 0.16.0 migration
runbook ([#199](#199))
([f190ae5](f190ae5))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jack-arturo