Tier the charset rule, invert the prose rules, and gate comments in every language

[ptr727]

The flat non-ASCII ban could not express context. reports/homeassistant-purpleair/audit.md classified U+2264 and U+2265 as the scientific carve-out while the rule named both must-replace. That is not a wrong reading so much as a rule with no vocabulary for the distinction.

Three tiers

Tier	Meaning	Behavior
1	Typography carrying no meaning its ASCII form loses	always flags
2	Relational and arithmetic operators	kept beside a number, unit, or another operator; replaced between words
3	Unit and scientific symbols	never flags
none	Unclassified	reported, not passed

Measured behavior:

Range <=2264> 50 <B5>g/m<B3> at 25 <B0>C <B1> 2.   -> clean
0 <=2264> x <=2264> 100                            -> clean
the check <=2264> the threshold                    -> flag, use `<=`
a <=2014> clause break                             -> flag, restructure
a <=2603> appeared                                 -> charset-unknown, classify it

Clean tree-wide, so charset gates. The audit report is corrected in the same pass.

Two prose rules inverted

The pronoun-keyed splice pattern found 170 of 493 and missed every imperative one, so any prose semicolon now flags and the exception is the list the rule already names. The em-dash replacement becomes "restructure the sentence" rather than a spaced hyphen, and that hyphen is banned in its own right: allowing the ASCII spelling would keep the shape and change only the character.

Both are markdown-only until a comment can be told from code. repo-config/configure.sh alone carries 78 statement separators that are not prose. Both are warn-only.

Comments, in every language the fleet declares

Driven by the declared project types (csharp, cpp, python, docker, homeassistant, eda, docs, nuget, pypi) rather than by whichever file was open:

Marker	Languages
//, /* */	C#, C, C++, JSONC, JSON, .code-workspace, JS/TS, CSS
#	Python via tokenize, shell, YAML, TOML, .gitattributes
<!-- -->	XML, .csproj, .props, markdown
<# #>	PowerShell
;, #	INI, .editorconfig
///, /**, docstrings	exempt, CODESTYLE governs those

JSON is read as JSONC, because that is what ships. Four tracked files were invisible before, including catalog/snippets/configs/vscode-tasks.json and both devcontainer.json snippets, which downstream repos copy. Their 37 malformed comments are fixed here so the shape stops propagating.

A marker inside a string literal is not a comment, verified in C#, shell, JSON, and Python.

Two rules come out of it: one sentence per comment line, never wrapped and never two on a line, and a sentence that starts on a line starts with a capital. Where the first word is a tool whose own casing is lowercase, the fix is to restructure rather than to capitalize against CODESTYLE tooling-casing.

Verification Discipline

The mechanisms behind the above land as fleet law: a gate has to be watched failing, driving the real table rather than a restatement of it, and a heredoc consumes a backslash escape and leaves an invisible control character behind.

Watched failing

• Drop U+2264 from tier 2 -> 2 cases fail, naming the codepoint and tier.
• Drop U+2126 from tier 3 -> 2 cases fail.
• Drop the .json mapping -> 5 cases fail, naming the file type.
• Disable the capitalization check -> 1 case fails.
• Break the workflow glob -> repo_gate prints 0 issue(s) and reads clean, while 3 cases object.

51 self-test cases. Whole gate green: mypy, markdownlint, actionlint, editorconfig-checker, cspell.

Posture

Gating: self-tests, both repo_gate checks, charset, dupword.
Warn-only in a continue-on-error step so the backlog is visible without blocking: charset-unknown, semicolon 323, dash 963, comment-wrap 432, comment-case 56. Going green needs the sweep the prose rules themselves forbid, so each is corrected as its file is next edited.

The 247-instance GOVERNANCE.md dash sweep is deliberately a separate PR, where the diff is reviewable as prose rather than buried under tooling.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR updates the repo’s governance-driven prose linting to support tiered charset rules, invert/reshape prose punctuation checks, and expand comment-format checks across the fleet’s declared language/file types, with CI updated to gate only the now-clean checks and surface the rest as warn-only backlog.

Changes:

• Replaces the prior flat ASCII/typography rule with a 3-tier charset policy (charset, charset-unknown) and corresponding self-tests and governance coupling.
• Makes semicolon and dash checks “construction-banning” (with named exceptions) and scopes them to markdown-only until code-comments can be cleanly separated.
• Adds cross-language comment extraction and comment style checks (comment-wrap, comment-case), plus CI reporting for backlog rules.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
scripts/test_prose_lint.py	Expands self-tests to cover charset tiers, punctuation rules, and multi-language comment extraction behavior.
scripts/README.md	Updates documentation of prose_lint.py behavior, including tiering and new comment rules.
scripts/prose_lint.py	Implements tiered charset scanning, markdown-only punctuation rules, and multi-syntax comment extraction with wrap/case checks.
host-setup/agent-safety/.markdownlint-cli2.jsonc	Reflows/clarifies markdownlint configuration comments (no functional change).
GOVENANCE.md	Updates governance text for comment capitalization, tiered charset policy, and punctuation rules.
docs/token-cost.md	Updates analysis text to reflect which rules gate vs. warn.
catalog/snippets/vscode/docker.jsonc	Reflows comments to match comment style rules.
catalog/snippets/vscode/base.jsonc	Reflows comments to match comment style rules.
catalog/snippets/devcontainer/python/devcontainer.json	Reflows comments to match comment style rules.
catalog/snippets/devcontainer/dotnet/devcontainer.json	Reflows comments to match comment style rules.
catalog/snippets/configs/vscode-tasks.json	Reflows comments to match comment style rules.
catalog/snippets/configs/vscode-tasks-python.json	Reflows comments to match comment style rules.
.markdownlint-cli2.jsonc	Reflows/clarifies markdownlint configuration comments (no functional change).
.github/workflows/validate-task.yml	Updates CI to gate charset+dupword and add warn-only reporting for backlog rules.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

docs/token-cost.md:92

• This bullet refers to multiple rules, but the follow-up sentence uses the singular pronouns "its" and "it", which reads ambiguously and is grammatically incorrect in context.

- **The semicolon and dash rules are warn-first, and the charset and duplicate-word rules gate, being clean tree-wide.** Compare their hits against the next batch of review findings before enforcing them, and do not claim the mechanical share until that comparison exists.

scripts/prose_lint.py:452

• Comment scanning runs before the markdown code-fence logic, so comment-wrap / comment-case will treat <!-- --> sequences inside fenced code blocks as real markdown comments. That can produce false positives when a fenced snippet contains HTML/XML (or literal <!-- ... -->) that is not a prose comment.

        raw = path.read_bytes().decode('utf-8')
    except (UnicodeDecodeError, OSError):
        return out
    lines = raw.split('\n')

scripts/prose_lint.py:477

• The semicolon exception currently skips any semicolon if there is any comma earlier on the line. That can miss real semicolon splices in sentences that happen to contain a comma before the splice (e.g. "When X, do Y; it gates the merge"). If the intent is to only exempt list-style semicolons, the heuristic should be constrained to list contexts (for example, after a colon that introduces the list).

        # Both prose rules are markdown-only until a comment can be told from code.
        # A shell script carries 78 statement separators that are not prose at all.
        if path.suffix == '.md':
            if 'semicolon' in rules:
                listish = prose.count(';') > 1 or ':' in prose.split(';')[0]

scripts/prose_lint.py:404

• extracted_comments scans for block-comment openers before it considers line comments. In C-like syntaxes this can mis-parse a /* ... */ sequence that appears inside a // ... line comment as the start of a real block comment, causing subsequent lines to be treated as "inside a block comment" until a closing token is found.

                out.append((n, body, True))
            closing = '' if end >= 0 else closing
            continue
        masked = strip_strings(line, spec['quotes'])
        cut = len(line)
        leading = True
        for opener, closer in spec['block']:
            at = masked.find(opener)
            if 0 <= at < cut:

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

scripts/README.md:48

• This list of warn-only checks is incomplete: the workflow's warn-only step also runs comment-wrap and comment-case, so the README should include them to match CI behavior.

`charset` and `dupword` are clean tree-wide and gate CI. `charset-unknown`, `semicolon`, and `dash` run as a warn-only CI step so their backlog is visible without blocking, and are corrected as each file is next edited.

scripts/prose_lint.py:455

• comment_wrap_findings() runs before fenced code blocks are detected, so comment-wrap / comment-case findings can be reported from inside ```/~~~ fenced regions even though the rest of check_file() skips fenced blocks. This makes markdown code examples and other fenced text eligible for comment findings while other rules intentionally ignore them.

    lines = raw.split('\n')
    if {'comment-wrap', 'comment-case'} & rules:
        out.extend(f for f in comment_wrap_findings(path, raw, lines) if f[1] in rules)
    in_fence = False

scripts/README.md:42

• This sentence says CSS uses // line comments, but the implementation explicitly treats CSS as block-comment-only (see scripts/prose_lint.py CSS: Syntax = {'line': (), ...}) and the tests assert that // in CSS is a URL scheme separator, not a comment marker.

The `comment-wrap` rule covers comments in every syntax the fleet's project types carry, not only the hash ones: `//` and `/* */` for C#, C, C++, JSONC and CSS, `<!-- -->` for XML, csproj and markdown, `<# #>` for PowerShell, `;` for INI, and `#` for Python, shell, YAML and TOML.

scripts/README.md:40

• The earlier paragraph about whole-tree runs still refers to the old ascii rule and CI gating ascii + dupword. With this PR, CI gates charset + dupword and reports the other checks warn-only, so that paragraph should be updated to avoid misleading local runs.

This issue also appears on line 48 of the same file.

The `semicolon` and `dash` rules ban a construction rather than a detectable subset of it, so each flags by default and the exceptions are the ones the rule names: a semicolon inside a list that already carries commas, and for the dash a compound word, a leading list marker, a range, and the `- **Label** - explanation` separator that opens a governed bullet.

**Both are markdown-only for now.** A shell script carries 78 statement separators that are not prose at all, so telling a comment from code is a precondition for reaching source files. Until then a semicolon or dash in a code comment is missed, which reading the diff by eye still catches.

[ptr727]

Worked the low-confidence findings from the review body, which I had not opened. All four held.

1. Comment rules ignored fenced blocks (prose_lint.py:455). The real bug of the four. comment_wrap_findings() ran before the fence loop, so a comment inside a fenced markdown block was reported while every other rule skipped fences. Reproduced first:

Prose.

```html
<!-- One thing. Another thing. -->

-> flagged comment-wrap

A fenced example is quoted code, so its comments belong to whatever is being shown. Fixed in `cc1446d`, with a case asserting the fenced form is clean and the unfenced form still flags.

Worth noting my first attempt to reproduce this was wrong. I put a `//` comment in a markdown fence and saw nothing, and nearly declined the finding on that basis. Markdown maps to `<!-- -->` only, so `//` was never a comment there. The finding was right and my test was measuring nothing.

**2, 3, 4. Three stale claims in `scripts/README.md`.** All mine. It said CSS uses `//` line comments, which the previous round disproved when it turned out `//` in a URL was reading as a comment marker. The whole-tree paragraph still named the pre-tier `ascii` rule. The warn-only list omitted `comment-wrap` and `comment-case`, which the CI step does run, so the doc understated what CI reports.

## The loop is the durable fix

These findings have landed several times now, and a loop that polls `reviewThreads` never sees them. Two changes:

- The **runbook** now requires reading the body of every review and investigating each suppressed finding on the same footing as an inline one, answering it in the PR conversation because there is no thread to reply on or resolve. It carries the query.
- The **merge gate** named "all review threads resolved" and "issue-level comments triaged", which silently excluded these. Condition 3 now names them.

So the gap was in the contract, not just in my attention. 56 self-test cases, whole gate green.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

scripts/prose_lint.py:461

• comment-wrap continuation detection only tracks prev_body for comments that start the line. That misses wrapped sentences that begin in a trailing inline comment and continue on the next (leading) comment line (still a cross-line wrapped sentence per the rule), because prev_body is cleared when leading is false.

        # A continuation is the very next line: two comments with code between them are separate.
        adjacent = n == prev_no + 1
        continuation = (adjacent and leading and prev_body
                        and not SENT_END.search(prev_body) and body[:1].islower())
        if continuation:
            out.append((prev_no, 'comment-wrap',
                        'comment sentence wraps into the next line -> one sentence per line'))
        # A lowercase opening that is not a continuation is a sentence that failed to start.
        elif leading and body[:1].islower():
            out.append((n, 'comment-case',
                        'comment sentence opens in lowercase -> capitalize, or restructure so it '
                        'does not open on a lowercase name'))
        prev_body = body if leading else ''
        prev_no = n

[ptr727]

The one low-confidence finding on the previous head held, and it was worse than a miss.

prev_body was cleared whenever a comment did not start its line, so this pair was not read as wrapped:

x = 1  # a sentence that keeps
# going onto the next line.

It reported comment-case instead, advising to capitalize the continuation, when the correct fix is to un-wrap the sentence. Reporting the wrong rule is worse than reporting nothing, because the fix it asks for is wrong.

Fixed in abcfe8c. A trailing comment is now remembered as a possible sentence start. The continuation still has to be a full-line comment, since a trailing one annotates its own line rather than continuing the line above, so two trailing annotations on consecutive lines stay separate. Verified across all six combinations:

Input	Result
trailing starts, leading continues	comment-wrap
leading starts, leading continues	comment-wrap
trailing label, then a new sentence	clean
two trailing annotations	clean
labels split by code	clean
lowercase leading, no predecessor	comment-case

Reverting the fix fails the new case, so the behavior is pinned rather than incidental.

58 self-test cases. Whole gate green.

That is two rounds in a row where the only real finding was in the suppressed block and the inline threads were empty, which is why the runbook and merge gate now name it.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

scripts/prose_lint.py:279

• comment-wrap is meant to catch any two sentences on one comment line, but RUN_ON only triggers when the second sentence starts with an uppercase letter and also suppresses matches when the sentence-ending punctuation is preceded by any uppercase letter. This misses real run-ons like One thing. another thing. and CI. Another thing.

# Two sentences on one line, guarded against an abbreviation or a dotted identifier.
RUN_ON = re.compile(r'(?<![A-Z])(?<!\be\.g)(?<!\bi\.e)(?<!\bvs)(?<!\betc)[.!?]\s+(?=[A-Z])')

.github/workflows/validate-task.yml:66

• The workflow comment says only the semicolon rule is warn-only, but this job also runs other warn-only prose rules (charset-unknown, dash, comment-wrap, comment-case) in the backlog step right below. Updating the comment avoids implying that only semicolon is non-gating.

      # The charset and duplicate-word rules are clean tree-wide, so they gate.
      # The semicolon rule stays warn-only, since going green needs a sweep the rule forbids.

[ptr727]

Both low-confidence findings on abcfe8c held. Fixed in f401f9c.

1. A sentence ending in an acronym was invisible. The initial guard was (?<![A-Z]), meant to keep J. Smith as one name. It suppressed a match whenever any capital preceded the period, and this codebase ends sentences in CI, PR, SHA, and JSON constantly.

# The check runs in CI. Another thing happens.   -> MISSED
# Pinned by SHA. Dependabot still bumps it.      -> MISSED

The guard now anchors on a word boundary, (?<!\b[A-Z]), so J. is still one name (the J opens a word) while CI. is two sentences (the I does not).

2. A lowercase second sentence escaped both rules. The run-on required (?=[A-Z]) and missed it for being lowercase; the capitalization rule missed it for not starting a line. # One thing happens. another thing happens. fell through the gap. Either case now counts.

3. The workflow comment named only the semicolon rule as warn-only while the step below reports four more.

Every guard re-verified after widening:

Input	Result
CI. Another thing.	flag
One thing. another thing.	flag
Reviewed by J. Smith today.	clean
Uses e.g. Docker and i.e. Podman.	clean
Bump to 3.13 for the runner.	clean
See audit.py and validate.py.	clean
Formats, then lints, etc. for each file.	clean

Reverting the guard fails 3 cases. comment-wrap rises 436 to 454, which is the under-reporting removed.

61 self-test cases. Whole gate green.

That is three rounds running where the inline threads were empty and the suppressed block held the only real findings, five of five across them. Worth saying plainly: had I merged on the first clean-looking round, the gate would have shipped silently under-reporting on exactly the sentence shapes this repo writes most.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.