Read dupword from the comments, not from the source lines

[ptr727]

Finding 3 of #489, reported by the Blog agent while carrying the baseline into the first Hugo repo.

The defect

dupword is one of three rules in the blocking CI step (--check charset --check dupword --check spelling). Outside markdown it read the raw source line, so it judged code as prose and rejected correct work.

The reported case has no legal fix:

<div class="gallery gallery-cols-1">

Two class names sharing a prefix is the ordinary CSS idiom. Editing it changes the rendered page, so the only way to pass the gate was to break the styling.

The fix

spelling already solved this in the same function: outside markdown it reads the extracted comments rather than the source line. dupword now selects its text the same way, reusing the same extraction.

Narrowing to the comment is preferred over exempting the class attribute, since rel, srcset, sizes and the data-* attributes all take value lists of the same shape, and an exemption list covers only what its author thought of.

Each comment on a line is judged on its own rather than joined with its neighbors, because two comments are two sentences and joining them reads the second's opening word as a repeat of the first's last.

Measured, both directions

Every case below was run against the rule before and after. Nothing else changed.

Case	Before	After
<div class="gallery gallery-cols-1">	FIRES	quiet
<a rel="nofollow nofollow-ugc" href="#">	FIRES	quiet
x = "the the" (Python string literal)	FIRES	quiet
key: the the (YAML scalar)	FIRES	quiet
{ "a": "the the" } (JSON string value)	FIRES	quiet
<p>the the</p> (HTML body text)	FIRES	quiet
<!-- The the thing. --> (HTML comment)	FIRES	FIRES
# The the thing. (Python, shell, YAML)	FIRES	FIRES
// The the thing. and /* ... */ (C#)	FIRES	FIRES
x = 1 # The the thing. (trailing comment)	FIRES	FIRES
the the thing (markdown prose)	FIRES	FIRES

The cost, stated rather than hidden

A duplicated word in HTML body text, or in a YAML or JSON string value, is no longer caught. That trade was accepted deliberately when the approach was chosen. The narrower alternative, stripping only HTML attribute values, was considered and declined for the reason above.

Asserting the floor

This narrows a blocking gate, so the floor is asserted rather than assumed.

• Tree-wide dupword was clean before the change and is clean after.
• Six new test cases fail against the old rule and pass against the new one, so they are not vacuous. The comment-syntax cases pass against both, which is the floor holding.
• test_the_repo_is_clean_of_duplicated_words pins the tree-wide floor the way the spelling rule already does, so the gate cannot silently stop gating.
• DUP_ALLOW (that that, had had) is untouched. It is the pre-existing band-aid, still needed for markdown prose, and now the only remaining allowlist in the rule.

Test bait for the attribute cases is assembled from two literals, following the convention this module already states, so the file never holds the pattern it feeds the gate.

Verification

Every step of the CI job run locally:

python3 scripts/test_prose_lint.py     136 tests, OK  (130 before, 6 added)
python3 scripts/test_repo_gate.py       23 tests, OK
python3 scripts/test_pr_review.py       27 tests, OK
python3 spec/audit.py --selftest        SELFTEST PASS
python3 scripts/repo_gate.py            eol 0, sha-pin 0
prose_lint --check charset --check dupword --check spelling    clean
prose_lint . --diff HEAD                clean, all default rules

The warn-only step reports the same 5 pre-existing comment-wrap findings in prose_lint.py as before, and no new ones. Two comments this PR touches were rewritten to one sentence per line so the change adds no backlog.

Docs

scripts/README.md documented the comments-only scope for spelling alone. It now covers both rules, states why an attribute value is not prose, and states the cost.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes a false positive in the blocking dupword prose gate by aligning it with the existing spelling rule behavior: outside Markdown, both rules now analyze extracted comments rather than raw source lines. This prevents valid code/content constructs (notably repeated tokens in HTML attribute values and other non-prose syntax) from being incorrectly rejected by CI, while preserving enforcement for actual prose in Markdown and in comments across supported syntaxes.

Changes:

• Update dupword to scan extracted comments (per-comment) instead of scanning raw source lines in non-Markdown files.
• Add targeted tests that prove the prior false positives (attributes / literals) are fixed while comment-based enforcement remains intact.
• Update documentation to describe the shared “comments-only outside Markdown” scope for both spelling and dupword, including the intentional tradeoff.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
scripts/prose_lint.py	Changes dupword to read extracted comments outside Markdown (per comment), avoiding false positives on code/attribute values.
scripts/test_prose_lint.py	Adds regression tests covering the reported attribute cases and the new comments-only semantics for dupword.
scripts/README.md	Documents the updated scope and its deliberate cost for the CI-gating dupword rule.