Skip to content

fix(extract): treat any non-whitespace-flanked period as non-boundary - #703

Merged
plind-junior merged 5 commits into
vouchdev:testfrom
philluiz2323:fix/extract-segment-boundary-non-digit-periods
Jul 31, 2026
Merged

fix(extract): treat any non-whitespace-flanked period as non-boundary#703
plind-junior merged 5 commits into
vouchdev:testfrom
philluiz2323:fix/extract-segment-boundary-non-digit-periods

Conversation

@philluiz2323

Copy link
Copy Markdown
Contributor

What changed

extract.py's _SEGMENT_RE now treats any period flanked by non-whitespace
on both sides as a non-boundary, instead of only one flanked by digits.

Why

The old lookaround, (?<=\d)\.(?=\d), correctly kept decimal/version
numbers intact ("6.8.3") but did nothing for the far more common case of
periods inside file paths (src/vouch/cli.py), module/line references
(cli.py:2550), or domains (github.com) — every one of those still
split the sentence. segment_source has no coherence check afterward,
only a minimum-length and letter-ratio filter, so the resulting
mid-sentence shards survived and got filed via extract_receipt_claims
propose_quoted_claim, which auto-approves anything that verifies as
a byte-exact quote of the source (review.auto_approve_on_receipt
defaults to True).

Confirmed with a repro: ingesting "the parser bug lives in
src/vouch/cli.py:2550 near the top. see the changelog at
github.com/vouchdev/vouch/blob/main/CHANGELOG.md for details." produced
four durably-approved claims, two of them nonsensical fragments
("see the changelog at github.", "py:2550 near the top.").

Fixes #702

What might break

Nothing for users with an existing .vouch/ directory — no on-disk
shape, kb.* method, or object model change. Behaviorally: sentences
containing a file path, module reference, or domain now stay intact
instead of fracturing, which is strictly the intended, documented
behavior for decimals extended to a broader (and more common) class of
non-boundary periods. No legitimate sentence-ending period changes
behavior, since a true sentence end is followed by whitespace or
end-of-string, which still terminates the segment.

VEP

Not applicable — no object model, kb.* method, on-disk layout, bundle
format, or audit-log shape change. A regex-correctness fix inside the
ingest segmenter.

Related prior work

A related PR, #518 (fix(extract): stop fragment claims and stopword-matched injection), touched adjacent parts of this file but
was closed unmerged with an unresolved CodeRabbit nit (an unrelated
markdown-bullet-prefix gap) and predates select_spans's
density-selection logic added since, so its diff no longer applies. This
PR targets a narrower, independently-verified defect in the current
code — the segmentation regex's digit-only lookaround — unrelated to
what #518 was reviewing.

Tests

  • Local make check-equivalent: ruff clean (src + tests); mypy
    clean on extract.py; all 14 tests/test_extract.py cases pass
    (13 pre-existing + 1 new); tests/test_bench.py,
    tests/test_capture_answer.py, tests/test_capture_scope.py, and
    tests/test_health.py (all consume extract.py) pass with one
    pre-existing, unrelated failure
    (test_finalize_supersedes_updated_claims), confirmed present
    identically on test HEAD before this change via git stash
    comparison
  • New / changed behaviour has a test —
    test_segment_source_keeps_file_paths_and_urls_intact
  • CHANGELOG.md updated under ## [Unreleased]

_SEGMENT_RE only skipped a "." as a sentence boundary when it was
flanked by digits on both sides, correctly keeping decimal/version
numbers like 6.8.3 intact. every other non-whitespace-flanked period -
file paths (src/vouch/cli.py), module/line refs (cli.py:2550), domains
(github.com) - still split the sentence, since the lookaround only
special-cased digits.

segment_source has no coherence check afterward, only a min-length and
letter-ratio filter, so the resulting mid-sentence shards ("see the
changelog at github.") survived and got filed via
extract_receipt_claims -> propose_quoted_claim, which auto-approves
anything that verifies as a byte-exact quote of the source -
review.auto_approve_on_receipt defaults to true, so these fragments
landed as durable, approved claims indistinguishable from real facts.

broaden the lookaround from digit-flanked to non-whitespace-flanked:
(?<=\S)\.(?=\S). this subsumes the existing decimal case (digits are
non-whitespace) while covering file paths, module refs, and domains -
a period only ends a segment when followed by whitespace or
end-of-string.

Fixes vouchdev#702
@github-actions github-actions Bot added docs documentation, specs, examples, and repo guidance tests tests and fixtures size: XS less than 50 changed non-doc lines labels Jul 31, 2026
@plind-junior

Copy link
Copy Markdown
Member

the bug is real and the repro is convincing — "see the changelog at github." reaching decided/ as an auto-approved claim is exactly the failure mode auto_approve_on_receipt makes expensive, because a byte-exact quote of a garbage shard still verifies. no argument that (?<=\d)\.(?=\d) was too narrow.

my concern is that the replacement is very broad in the other direction, and the PR only tests the direction it is trying to fix. (?<=\S)\.(?=\S) says a period is a boundary only when whitespace or end-of-string follows it, so every one of these now merges what used to be two segments:

  • a missing space after a full stop — "ship it.Then we deploy." — which is ordinary typing noise in the chat transcripts this segmenter runs on
  • a period before a closing quote, paren or bracket — "we agreed (see docs.)", he said "done."
  • an ellipsis — "wait... then go"

the result is a longer run, and segment_source's only downstream filters are min length, max length and letter ratio. so an over-merged run either survives as one run-on claim or trips DEFAULT_MAX_CHARS = 320 and is dropped entirely — a fact silently lost rather than fractured. under auto_approve_on_receipt the first outcome is durable without review, same as the bug you are fixing.

concretely: the three new assertions are all "these stay intact", and none is "these still split". worth adding the negative half — a two-sentence string with a missing space after the period, and one ending docs.) — and deciding explicitly which behaviour you want. if merging on a missing space is not acceptable, the narrower rule that still fixes #702 is to exclude whitespace and sentence-case starts, e.g. requiring the following character to be lowercase or a digit: (?<=\S)\.(?=[a-z0-9/]). that keeps cli.py, github.com and 6.8.3 intact while ship it.Then still breaks.

the CHANGELOG wording is accurate about what the regex does, so no issue there.

philluiz2323 and others added 4 commits July 31, 2026 00:41
…-boundary-non-digit-periods

# Conflicts:
#	CHANGELOG.md
a6c6862 (vouchdev#686) fixed capture.load_config's min_observations and
dedup_window_seconds to fall back to their defaults on a malformed
config value via the coerce_numeric() helper, instead of raising
ValueError straight out of load_config. 47eaf56 (vouchdev#645, realtime
opt-in) branched off the pre-fix capture.py and reintroduced the bare
int()/float() calls when it merged into test - the coerce_numeric
import survived (nothing else referenced it), but the two call sites
it fed didn't, silently reverting the fix and breaking ruff's
unused-import gate for every PR built on top of `test`.

restore the coerce_numeric() calls, matching recall.load_config's
still-intact equivalent.

unrelated to this PR's own change (extract.py segmentation); needed
only to get CI green on top of a currently-broken `test`.
…periods

changelog union only; no source conflict.
…periods

changelog union only; no source conflict.
@plind-junior
plind-junior merged commit 755427f into vouchdev:test Jul 31, 2026
11 checks passed
@github-actions github-actions Bot added the ci: passing ci is green label Jul 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

diff coverage: n/a — this PR changes no python under src/vouch/, so there is nothing for the gate to measure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: passing ci is green docs documentation, specs, examples, and repo guidance size: XS less than 50 changed non-doc lines tests tests and fixtures

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(extract): sentence segmenter fractures file paths and URLs into auto-approved garbage claims

2 participants