Skip to content

fix(entity): stop validator over-rejecting real people, code tools, and event categories - #178

Merged
jack-arturo merged 2 commits into
mainfrom
fix/entity-validator-overrejection
Jun 10, 2026
Merged

fix(entity): stop validator over-rejecting real people, code tools, and event categories#178
jack-arturo merged 2 commits into
mainfrom
fix/entity-validator-overrejection

Conversation

@jack-arturo

@jack-arturo jack-arturo commented Jun 10, 2026

Copy link
Copy Markdown
Member

Summary

Production Gate-3 review of the entity-tag repair rollout (the human review of rejected-tags.csv before executing repair_entity_tags.py against production) surfaced three over-rejection classes in the deployed validator. Of 7,677 planned tag removals, ~1,600 were legitimate entities.

What was wrong

  1. Real people rejected by the context-hint branch. _looks_tool_or_org_like rejected any person whose memory content contained generic words (data, project, platform, tool...). In an engineering corpus that's nearly every memory — 725 distinct multi-token person names were condemned, including the corpus owner's own canonical entity.
  2. code as a primary fragment token rejected real tool entities: claude-code (80 occurrences), vs-code.
  3. events and opportunities categories missing from _CATEGORY_ALIASES — every such tag was dropped as unknown_category.

The fix

  • Person-shaped multi-token slugs skip the context-hint branch. CamelCase and tool/org suffix signals still apply (growthmath-style names are still rejected); single-token brand-like people (automem, claude) are still rejected contextually.
  • code demoted to _MARKDOWN_OR_CODE_SECONDARY_TOKENS (needs a second code-ish token to reject). People slugs containing code and path/markdown fragments remain rejected via existing checks.
  • events/opportunities (+ singular aliases) added to the category map.
  • Backstop for noise the context branch used to catch: _NON_PERSON_COMMON_TOKENS (bottom-line / deck-today / email-highlights / claude-desktop class) and pipeline added to _NON_PERSON_TECH_TOKENS.

Empirical validation on a production clone (10,061 memories)

deployed validator this PR
planned rejections 7,677 6,106
freed (all person/tool/event entities) — 1,603 across 743 distinct tags
newly caught (all generated noise) — 32

Every freed tag inspected by category: person names, claude-code/vs-code, entity:events:*, entity:opportunities:*. Every newly-rejected tag is generated noise (good-plugins, chrome, claude-desktop, stream-deck-as-person).

This also stops enrichment-time over-stripping: the same validator gates _validated_entities(), so new memories were silently losing these entities on every store since #176 deployed.

Test Plan

  • pytest tests/ → 487 passed, 12 skipped (env-dependent)
  • make lint → clean
  • New tests: person names in technical context, code-suffixed tools, event/opportunity categories, common-word-pair people noise — all using synthetic names per the no-real-fixtures rule

Refs #72. Part of the staged production entity repair (Gate 3 of the rollout runbook).

🤖 Generated with Claude Code

@jack-arturo @claude
fix(entity): stop validator over-rejecting real people, code tools, a…
31526f6 
…nd event categories

Production Gate-3 review of the entity-tag repair plan surfaced three
over-rejection classes in the deployed validator (~1,600 of 7,677
planned removals were legitimate entities):

- The _TOOL_OR_ORG_CONTEXT_HINTS branch rejected any person whose memory
  mentioned generic words like data/project/platform/tooling — in an
  engineering corpus that is nearly every memory, so 725 distinct
  multi-token person names were condemned. Person-shaped multi-token
  slugs now skip the context branch; camelCase and tool/org suffix
  signals still apply, and single-token brand-like people are still
  rejected contextually.
- "code" as a primary markdown/code-fragment token rejected real tool
  names (claude-code, vs-code, code-server). It is now a secondary
  signal; people slugs containing "code" and path/markdown fragments
  remain rejected.
- events and opportunities categories were missing from
  _CATEGORY_ALIASES, so every such tag was dropped as unknown_category.

Backstop: a small common-word token set (bottom-line, deck-today,
email-highlights, claude-desktop class) plus "pipeline" in the tech
tokens keeps the generated-noise people slugs out now that the context
branch no longer catches them.

Empirical diff on a production clone (10,061 memories): rejections
7,677 -> 6,106; all 1,603 freed rejections are person/tool/event
entities; all 32 newly-caught rejections are generated noise.

Refs #72.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 10, 2026 21:15
Copilot AI reviewed Jun 10, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the entity-quality validator to reduce false rejections discovered during the production “Gate-3” review of the entity-tag repair rollout, specifically for real people names in technical context, tool names containing code, and missing events / opportunities categories.

Changes:

  • Add event(s) and opportunity/opportunities to the entity category alias map so they validate instead of being rejected as unknown_category.
  • Refine the “markdown/code fragment” heuristic by demoting code to a secondary token (so *-code tools aren’t rejected solely due to code).
  • Reduce context-hint over-rejection for multi-token person names and add a backstop token set for common non-name word pairs; add tests covering these cases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
automem/utils/entity_quality.py Updates category normalization and validation heuristics (people/tool/code/category handling).
tests/test_entity_quality.py Adds regression tests for multi-token people-in-tech-context, code-suffixed tools, new categories, and common-word-pair noise.

Comment thread automem/utils/entity_quality.py
@jack-arturo
Potential fix for pull request finding
e6876b0 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@jack-arturo
jack-arturo merged commit 193b730 into main Jun 10, 2026
7 checks passed
@jack-arturo
jack-arturo deleted the fix/entity-validator-overrejection branch June 10, 2026 21:24
This was referenced Jun 10, 2026
Merged
Merged
jack-arturo added a commit that referenced this pull request Jun 10, 2026
@jack-arturo @claude
fix(entity): restore person-shape exemption on the slug validation pa…
5e29960 
…th (#179)

## Why

PR #178's applied review suggestion (commit e6876b0, "Potential fix for
pull request finding") gated the person-shape exemption on the display
value containing a space:

```python
if " " in (value or "").strip() and len(parts) >= 2 and _has_person_name_shape(parts):
```

Stored entity tags only retain the slug (`entity:people:jack-arturo`),
so `validate_entity_tag(context=...)` — the path
`scripts/lab/repair_entity_tags.py` uses — never satisfies the guard,
and the context-hint branch re-rejects every real person mentioned
alongside data/projects/tooling.

**Empirical impact (prod dry-run, read-only):** 7,494 planned rejections
with the guard vs ~6,106 expected with the exemption — ~1,390 legitimate
person tags (jack-arturo ×51, zack-katz ×27, jason-coleman ×25,
katie-keith ×14, ...) wrongly planned for removal. CI stayed green
because every existing test exercised the spaced-value path
(`validate_entity_value("people", "Mara Quinn", ...)`), never the slug
path with context.

## What

- Restore the plain person-shape exemption (no space guard) in
`_looks_tool_or_org_like`.
- Address the original Copilot concern (`entity:people:data-dog`)
deterministically: add `"data"` to `_NON_PERSON_TECH_TOKENS`, so
brand-like person-shaped pairs are rejected by the per-token vocabulary
check on **every** path (with or without context) — strictly stronger
than the context-hint rejection the guard tried to preserve.
- Regression tests for the slug path: real-person tags survive technical
context via `validate_entity_tag`; `data-dog` is rejected with
`low_signal_people_slug` without needing context.

## Verification

- `pytest tests/`: 490 passed, 12 skipped
- Spot checks: `jack-arturo`/`zack-katz`/`jason-coleman`/`katie-keith`
accepted with technical context on the tag path; `data-dog`,
`growthmath`, `claude-code`-as-people still rejected

Part of the issue #72 production repair rollout (follow-up to #178).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This was referenced Jun 10, 2026
Open
Closed
Closed
jack-arturo added a commit that referenced this pull request Jun 26, 2026
@jack-arturo
chore(main): release 0.16.0 (#154)
8c67c16 
🤖 I have created a release *beep* *boop*
---


##
[0.16.0](v0.15.2...v0.16.0)
(2026-06-26)


### Features

* **api:** add admin backup endpoint
([#162](#162))
([8b1f264](8b1f264))
* **api:** support bulk memory associations
([1221e36](1221e36))
* **api:** support bulk memory associations
([#198](#198))
([28eb916](28eb916))
* **benchmarks:** LongMemEval failure-mode diagnosis harness + judge
quota preflight
([#183](#183))
([f99bece](f99bece))
* **consolidation:** expose cluster threshold and min size as env vars
([#163](#163))
([7e731f3](7e731f3))
* **enrichment:** expose classification fallback-rate metrics in
/enrichment/status
([#188](#188))
([0b522a9](0b522a9))
* **entity:** harden identity cleanup and repair tooling
([#176](#176))
([827dfbc](827dfbc))
* **eval:** recall-quality optimization harness — lab foundation +
design ([#197](#197))
([431433e](431433e))
* **graph:** support unbounded visualizer snapshots
([#141](#141))
([c730128](c730128))
* **lab:** add aged labelled distractor injection
([cc5d546](cc5d546))
* **lab:** add config_complexity simplicity metric
([dfb10d9](dfb10d9))
* **lab:** add distractor_rate_at_k precision guardrail metric
([872eab2](872eab2))
* **lab:** add lab_corpus with parameterized recall
([5e1e071](5e1e071))
* **lab:** add pick_winner scorecard decision rule
([3187eac](3187eac))
* **lab:** add real consolidation pass helper
([48a7d4a](48a7d4a))
* **lab:** isolate production clone restores
([#171](#171))
([aef90c0](aef90c0))
* **lab:** wire scorecard, distractors, recall params, consolidation
into runner
([589ec30](589ec30))
* **recall:** add metadata sidecar search
([#177](#177))
([4e7956e](4e7956e))
* **recall:** add state_mode=current|history recall alias
([#173](#173))
([b1df86c](b1df86c))
* **recall:** cap tag-score denominator to fix query-length bias
([#193](#193))
([cefa516](cefa516))
* **recall:** date-aware ranking + latest-fact selection
([#158](#158),
[#159](#159))
([#187](#187))
([a6ed945](a6ed945))
* **recall:** make recency decay window and curve configurable
([#182](#182))
([dbb933f](dbb933f))
* **recall:** ranking release — recency config, tag-score cap, relevance
gate, date-aware ranking
([#182](#182),
[#193](#193),
[#186](#186),
[#187](#187),
[#183](#183),
[#184](#184),
[#188](#188))
([#194](#194))
([337fe98](337fe98))
* **scripts:** safer reclassify_with_llm.py with provider flags +
tighter prompt
([#164](#164))
([a742602](a742602))


### Bug Fixes

* **api:** address copilot review on PR
[#198](#198)
([0466a1e](0466a1e))
* **api:** handle grouped association write failures
([cd93df9](cd93df9))
* **backup:** make backup_automem.py runnable as `python
scripts/backup_automem.py`
([#175](#175))
([edd9742](edd9742))
* **benchmarks:** add publication verification bundle
([#166](#166))
([420d721](420d721))
* **consolidation:** skip eager first tick at startup to avoid FalkorDB
load race
([#165](#165))
([1b812cf](1b812cf))
* **docs:** keep dispatch payload arrays stable
([df6e9e8](df6e9e8))
* **embedding:** fall back to per-item real embeddings before
placeholders in batch path
([#189](#189))
([6e9c62c](6e9c62c))
* **entity:** restore person-shape exemption on the slug validation path
([#179](#179))
([5e29960](5e29960))
* **entity:** stop validator over-rejecting real people, code tools, and
event categories
([#178](#178))
([193b730](193b730))
* **lab:** address copilot review on PR
[#197](#197)
([45f80d6](45f80d6))
* **lab:** align scorecard key contract (build_scorecard -&gt;
pick_winner)
([7d91530](7d91530))
* **mcp-sse:** decouple /health liveness from upstream readiness
([#151](#151))
([5bcfb8b](5bcfb8b))
* **mcp:** cap association failure summary
([ea4e08f](ea4e08f))
* **mcp:** surface stored metadata and updated_at in detailed recall
format ([#184](#184))
([230416e](230416e))
* **recall:** address copilot review on PR
[#194](#194)
([50b1647](50b1647))
* **recall:** canonicalize / and : separators in context_tag matching
([3afd9d3](3afd9d3))
* **recall:** canonicalize / and : separators in context_tag matching
([#203](#203))
([ba5e9ff](ba5e9ff))
* **recall:** gate query-independent scoring on topical evidence within
tag scope
([#130](#130))
([#186](#186))
([c11b594](c11b594))
* **recall:** hydrate semantic recall summaries
([#192](#192))
([76e845d](76e845d))
* **recall:** normalize graph keyword scores into the 0-1 component
range ([#191](#191))
([3653ddf](3653ddf))
* **recall:** respect current memory state
([#170](#170))
([ed36b98](ed36b98)),
closes [#169](#169)
[#158](#158)
[#159](#159)
* **scripts:** add sys.path guard to reembed_embeddings.py
([d333cf0](d333cf0))


### Documentation

* add scripts catalog, recall-quality-lab guide, and 0.16.0 migrations
([f20c664](f20c664))
* **bench:** log full judged 500q LongMemEval ship-config run with churn
attribution
([41bf8d0](41bf8d0))
* **eval:** Plan A — lab metric foundation (TDD, 9 tasks)
([0087dda](0087dda))
* **eval:** Plan B — parallel matrix harness (TDD, 9 tasks)
([c8ddfb2](c8ddfb2))
* **evals:** mark Memora/FAMA/WRIT lifecycle diagnostics as
diagnostic-only
([#174](#174))
([e8a3285](e8a3285))
* **eval:** spec for recall-quality optimization harness
([b1a1995](b1a1995))
* fix stale claims and document gated flags for 0.16.0
([b152d64](b152d64))
* note develop-branch contribution policy in README
([ccf02dd](ccf02dd))
* **positioning:** add scout reference
([#168](#168))
([922d23b](922d23b))
* refresh benchmark currency for the neutral AMB run and prune stale
archive docs
([3ff95bd](3ff95bd))
* refresh benchmark currency for the neutral AMB run and prune stale
archive docs
([#204](#204))
([89c30e0](89c30e0))
* refresh README and benchmark guidance
([#157](#157))
([bba31cc](bba31cc))
* **runtime:** align Docker viewer paths and setup guidance
([#155](#155))
([bbda79b](bbda79b))
* scripts catalog, recall quality lab guide, and 0.16.0 migration
runbook ([#199](#199))
([f190ae5](f190ae5))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jack-arturo