Promote the tiered charset rule and the comment gate to main

[ptr727]

Promotes d4c4085 (#459), adaa068 (#461), e52c68a (#463), 7bf8a12 (#464), 269629a (#465), and 036e468 (#467). Conflict-free, six commits ahead.

What lands

The charset rule gets three tiers instead of a flat non-ASCII ban, which could not express context - an audit report classified U+2264 and U+2265 as the scientific carve-out while the rule named both must-replace. Tier 1 never survives ASCII, tier 2 is an operator kept beside a number and replaced between words, tier 3 is a unit symbol whose ASCII form would be a lie. A character in no tier is reported rather than passed, and the rule is clean tree-wide, so it gates.

Two prose rules invert from detecting a subset to banning the construction. The pronoun-keyed splice pattern found 170 of 493 and missed every imperative one; the em-dash rule now says restructure the sentence, and the spaced hyphen is banned in its own right.

Comments get a gate covering every syntax the fleet types carry - //, /* */, <!-- -->, <# #>, ;, # - with JSON read as JSONC because that is what ships. That recovered four files the old discovery never saw, including the VS Code task and devcontainer snippets downstream repos copy; their 37 malformed comments are fixed here.

The Verification Discipline mechanisms land too: a gate has to be watched failing, and a check is scoped by what the project declares rather than by the file that prompted it.

The comment-extractor fix (#461)

Copilot's review of this promotion found a defect in the extractor #459 adds, so it was fixed at the source and this PR now carries it.

Block openers were searched before line markers against a ceiling of the whole line, so a /* inside a // comment opened a real block: the line's own comment was truncated at the opener, the closer carried into the lines below, and the code there was handed to the comment rules as prose. PowerShell failed identically with <# inside a # comment. Those are the two fleet syntaxes carrying both marker kinds.

Tree-wide counts are unchanged - dash 963, comment-wrap 454, semicolon 388, comment-case 56 - because nothing in this repository nests the markers that way. The exposure is downstream, in the C# and PowerShell repositories the extractor is aimed at, and it would surface the moment the comment rules gate rather than warn. That is the argument for fixing before promotion rather than after.

The carve-out contradiction (#463)

Copilot's second round found the same review's own subject matter contradicting itself. Two adjacent bullets disagreed on whether a developer-typed but un-tiered character is exempt from the gate or reported by it. The implementation already reported it as charset-unknown while the doc read as an exemption. The carve-out now states that it governs what an agent may rewrite rather than what the gate reports, which leaves both bullets true and changes neither rule.

Worth blocking a promotion for, because main is what the audit reads as ground truth, and this contradiction had already produced a real misreading - the audit report that classified U+2264 and U+2265 as the scientific carve-out, which is what motivated tiering the rule in #459 to begin with.

The scanner rewrite (#464)

The third round found a fourth defect in the same function, so the structure went rather than the instance. The extractor searched each marker kind from column 0 against a ceiling, and a ceiling can only describe the first comment on a line. That one shape produced every extractor defect in this promotion: a marker quoted inside the first comment read as real, a doc marker excluded from the ceiling unbounded it, and anything after the first comment was unreachable.

A single left-to-right pass replaces it. Several blocks on one line, a comment trailing the line where a block closes, and a multi-line documentation block that no longer leaks its prose into the scan all come with it.

#462 is fixed there too rather than deferred, since a defect left in the promoted diff keeps being found and the loop cannot reach a clean round while it stands. C# verbatim strings read correctly in both directions and in all three spellings, and C# took its own syntax entry so the rest of the C-like family does not inherit a form it lacks.

Multi-line strings (#465)

The fourth round found that masking runs per line while a C# verbatim string spans them, so a marker on any later line of one was reported as a comment. A false positive, and the direction that is worse than a miss, since it asks a reader to edit text that is data.

Reviewing that fix found two more in the same area: a quote in comment text opened a phantom string that blanked the markers after it, within a line and then across lines, and a line-skip meant as an optimization dropped a string that closed and reopened around real code. Masking now runs from the scan position and only code advances the string state.

#466 records what is still not carried. C# is the only syntax whose strings are tracked across lines, and the README says so rather than leaving a reader to find out.

The continuation asterisk (#467)

The fifth round found the extractor damaging the prose it then judged. A leading * was taken off every block comment body, which is the /* */ convention for continuing a line and ordinary text everywhere else, so <!-- *emphasis* leads here --> became emphasis* leads here and comment-case reported the lowercase opening it had just created. Worse than a plain false positive, since the prose reported is not what the file holds.

Stripping is now one * against whitespace, on a line continuing a /* */ block. This defect dates to d4c4085 rather than to any fix made during this review, so the promotion is where it would first reach main.

Why promote now

The snippets fixed here are the ones a new or realigning repo copies, and the reviewer-facing rules - the merge gate and the runbook now require investigating the low-confidence block - only bind downstream once main carries them.

That requirement earned itself twice more on #461. Round one's single inline comment read as a wording nit and was a live defect; round two had no inline comments at all and both findings sat in the suppressed block, one of them the reopened bug. A loop polling reviewThreads would have reported a clean pass on both rounds.

Fidelity note

GOVERNANCE.md and .github/copilot-instructions.md sections declared verbatim changed, so downstream copies are stale until re-vendored. spec/fidelity_honesty.py separates stale from modified by hash, so the audit reports it correctly.

Warn-only backlog, reported but not gating: dash 963, comment-wrap 454, semicolon 388, comment-case 56.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Promotes governance and linting changes to main by replacing the flat “ASCII-only” prose lint with a tiered charset model, expanding prose/comment lint coverage, and wiring the updated gates into CI and repo guidance docs.

Changes:

• Replace the prior ascii rule with tiered charset / charset-unknown handling, and add new prose/comment rules (dash, comment-wrap, comment-case) with expanded test coverage.
• Update CI (validate-task.yml) to gate on charset + dupword and report remaining prose/comment backlogs as warn-only.
• Refresh governance + snippets documentation/comments to reflect the new rules and review-loop requirements.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/prose_lint.py	Implements tiered charset checks plus new dash/semicolon/comment linting and comment extraction across file syntaxes.
scripts/test_prose_lint.py	Updates and significantly expands unit tests for tiered charset behavior, dash/semicolon policy, and comment linting.
scripts/README.md	Documents updated prose_lint.py rule set, tiers, scope, and CI posture (gate vs warn-only).
.github/workflows/validate-task.yml	Switches gating checks to charset + dupword and adds a warn-only backlog reporting step.
GOVERNANCE.md	Updates comment-style guidance and replaces the charset/semicolon/dash rules with tiered charset language + new constructions.
.github/copilot-instructions.md	Adds instruction to review “low confidence” suppressed findings in Copilot review bodies.
.markdownlint-cli2.jsonc	Comment wording improvements in markdownlint config.
host-setup/agent-safety/.markdownlint-cli2.jsonc	Comment wording improvements in nested markdownlint config.
docs/token-cost.md	Updates statement about which prose rules are warn-first vs gating.
catalog/snippets/vscode/base.jsonc	Reformats snippet comments to comply with the new comment-shape rules.
catalog/snippets/vscode/docker.jsonc	Reformats snippet comments to comply with the new comment-shape rules.
catalog/snippets/devcontainer/python/devcontainer.json	Reformats snippet comments to comply with the new comment-shape rules.
catalog/snippets/devcontainer/dotnet/devcontainer.json	Reformats snippet comments to comply with the new comment-shape rules.
catalog/snippets/configs/vscode-tasks.json	Reformats snippet comments to comply with the new comment-shape rules.
catalog/snippets/configs/vscode-tasks-python.json	Reformats snippet comments to comply with the new comment-shape rules.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

scripts/prose_lint.py:399

• In extracted_comments(), block comment openers are searched before line comment markers. As written, a /* (or <#) that appears inside a // ... (or # ...) line comment can be misinterpreted as starting a real block comment, causing subsequent code lines to be linted as comment prose until a */ (or #>) appears.

                body = (line[at + len(opener):end if end >= 0 else None]).strip().lstrip('*').strip()
                if body:
                    out.append((n, body, leading))
                closing = '' if end >= 0 else closer
        if closing:

scripts/test_prose_lint.py:295

• There is no regression test covering comment-marker precedence (e.g., a /* sequence inside a // ... line comment). Without a test, extracted_comments() can regress back into treating that /* as a real block opener and linting subsequent code lines as comment prose.

    def test_a_marker_inside_a_string_is_not_a_comment(self) -> None:

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

GOVERNANCE.md:186

• The docs say developer-typed Unicode (e.g. emoji) is preserved regardless of tier, but the implementation and tests treat an un-tiered emoji (U+2603) as a charset-unknown finding. Clarify the doc so it matches the current behavior (preserved in-place but still reportable as charset-unknown).

- **Unicode the developer deliberately typed** stays regardless of tier - emoji used for emphasis or as callout markers, for example the warning markers a maintainer placed in `README.md`. Never strip the developer's own characters. This carve-out is for developer-authored text, not a license for the agent to add emoji.
- **An unrecognized non-ASCII character is reported, not allowed.** Classify it into a tier above before using it. A gate that passes whatever it does not recognize stops gating as the character set grows, which is the silent-narrowing failure named under "Verification Discipline".

scripts/README.md:24

• This sentence says developer-typed characters such as emoji are preserved regardless of tier, but prose_lint currently reports an un-tiered emoji (U+2603) as charset-unknown. Consider rewording to reflect that developer Unicode is preserved in-place but can still be surfaced as charset-unknown when not tiered.

Enforces the [`GOVERNANCE.md`](../GOVERNANCE.md) "Documentation Style Conventions" rules that no linter checks: non-ASCII judged against the charset rule's three tiers, a semicolon in prose, a spaced hyphen joining or interrupting a sentence, a duplicated consecutive word, and the shape of a comment's prose.

The tiers decide by context rather than by a flat ban. Tier 1 carries no meaning its ASCII form loses and always flags. Tier 2 is an operator, kept next to a figure or another operator and replaced between words, so a threshold table reads as the range it is. Tier 3 is a unit or scientific symbol whose ASCII form would be a lie and never flags. Developer-typed characters such as emoji are preserved regardless of tier.

A character in no tier is a `charset-unknown` finding rather than a silent pass, since a gate that allows whatever it does not recognize stops gating as the character set grows. Classifying one is a fleet-law edit, so CI surfaces it without blocking on it.

scripts/prose_lint.py:270

• strip_strings() treats backslashes as escapes in all quoted spans, which breaks C# verbatim strings (e.g. @"C:\tmp") where backslash is literal. That can cause the closing quote to be masked and make trailing // comments invisible to the comment-wrap/comment-case extractor.

        if ch == '\\' and quote:
            escaped = True
            out[i] = ' '
            continue
        if quote:

scripts/test_prose_lint.py:278

• Comment extraction has a regression risk for C# verbatim strings: strip_strings() currently treats backslash as an escape inside any quoted span, which can hide a trailing // comment after a verbatim string ending in a backslash (e.g. @"C:\tmp"). Adding a case here would lock in the intended behavior for C# syntax.

        for name, text in (
            ('a.cs', f'// {self.RUN_ON}\n'),
            ('a.cs', f'/* {self.RUN_ON} */\n'),
            ('a.cpp', f'// {self.RUN_ON}\n'),
            ('a.c', f'/* {self.RUN_ON} */\n'),

[ptr727]

Answering the four findings in the round-two review's low-confidence block, since a suppressed finding has no thread to reply on. All four are right. They are two defects seen twice each, one fixed here and one deferred with a reason.

The carve-out contradiction (GOVERNANCE.md:186, scripts/README.md:24) - fixed

Confirmed against the implementation before acting on it:

un-tiered emoji: [(1, 'charset-unknown', 'SNOWMAN (U+2603) is in no tier - classify it in GOVERNANCE.md')]

An emoji is both developer-typed and un-tiered, so the two adjacent bullets pointed opposite ways: one said it stays and is never stripped, the next said an unrecognized character is reported rather than allowed. The code already had the coherent answer - preserved in place, still reported until classified - and the doc never said the second half.

Fixed in #463, squashed to develop as e52c68a and now carried here. The carve-out is stated as governing what an agent may rewrite rather than what the gate reports, which leaves both bullets true and changes neither rule. The spaced hyphen in the edited bullet is recast at the same time, under the rule three bullets below it.

This one was worth blocking the promotion for. main is what the audit reads as ground truth, and the contradiction had already produced a real misreading - the audit report that classified U+2264 and U+2265 as the scientific carve-out, which is what motivated tiering the rule in #459 to begin with.

The C# verbatim string (prose_lint.py:270, test_prose_lint.py:278) - deferred to #462

Also confirmed:

LINE : var p = @"C:\tmp\"; // Two things. Here.
MASK : var p = @"
EXTR : []

A verbatim string ending in a backslash consumes its own closing quote, the masker never leaves the string, and the trailing comment is invisible to every comment rule. The related "" escape form is mis-paired too.

Deferred rather than fixed here, with the scope that justifies it recorded in #462: false negatives only (the opposite direction from the defect fixed in #461), line-scoped so it cannot corrupt the lines below, warn-only in CI today, and C# alone among the fleet syntaxes has a verbatim form. Fixing it means teaching the masker a language-specific string form, which is a larger change than a promotion should absorb.

On this block

It has now been load-bearing on three of the four review rounds across this PR and #461, twice with no inline comments at all. A loop polling reviewThreads would have reported a clean pass every one of those times. That is the case the merge gate and the runbook were changed to require in #459, which is the commit being promoted here.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

scripts/prose_lint.py:270

• strip_strings() only treats backslash as an escape. For C# verbatim strings (e.g. @"..."), embedded quotes are doubled (""), so this loop will treat the first quote in the pair as closing the string and can mis-read later comment markers on the same line as real comments.

        if quote:
            out[i] = ' ' if ch != quote else ch
            if ch == quote:
                quote = ''
        elif ch in quotes:

scripts/prose_lint.py:399

• extracted_comments() can only extract at most one block comment per line, and it cannot see a line comment that appears after a block comment on the same line (e.g. code /* ... */ // tail). Because marker searches always start at column 0 (find() with no start index) and cut is set to the block opener position, any later comment on the line is skipped, reducing comment-wrap/case coverage.

        leading = True
        for opener, closer in spec['block']:
            at = masked.find(opener)
            if 0 <= at < min(cut, line_at):
                if any(line[at:].startswith(d) for d in spec['doc']):

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

scripts/test_prose_lint.py:452

• The PR description says the C# verbatim-string comment-masking defect (#462) is deferred, but this change set appears to include verbatim-string handling already (SYNTAX['.cs']['verbatim'] plus the new strip_strings() verbatim logic and the associated tests). This is a discrepancy between the PR description and the actual diff; please update the PR description (or clarify what part remains deferred) so reviewers and downstream consumers don’t misinterpret the promotion scope.

    def test_a_verbatim_string_keeps_its_own_closing_quote(self) -> None:
        """A backslash is ordinary inside one and a doubled quote is the escape.

        Read with C escape rules the string never closes, so the masker blanks the rest of the
        line and the trailing comment goes unseen.
        """
        # Ending in a backslash, the string swallows its closing quote and hides a real comment.
        self.assertEqual(['comment-wrap'],
                         self.flag('a.cs', 'var p = @"C:\\tmp\\"; // Two things. Here.\n'))
        # Reading a doubled quote as a close then a reopen puts string content outside the string.
        self.assertEqual([],
                         self.flag('a.cs', 'var s = @"a""// One thing. Another thing.""b"; // ok\n'))
        # An interpolated one is spelled either way round, and only one of them abuts the quote.
        for text in ('var s = $@"C:\\tmp\\"; // Two things. Here.\n',
                     'var s = @$"C:\\tmp\\"; // Two things. Here.\n'):
            with self.subTest(line=text.strip()):
                self.assertEqual(['comment-wrap'], self.flag('a.cs', text))

    def test_only_the_syntax_that_has_verbatim_strings_gets_them(self) -> None:
        """C shares the C-like spec without the form, so `@` there is an ordinary character."""
        self.assertTrue(prose_lint.SYNTAX['.cs']['verbatim'])
        self.assertFalse(prose_lint.SYNTAX['.c']['verbatim'])
        self.assertFalse(prose_lint.SYNTAX['.json']['verbatim'])
        # The C escape still hides a marker, which is what the verbatim rule must not undo.
        self.assertEqual(['comment-wrap'],
                         self.flag('a.cs', 'var s = "a\\"b"; // Two things. Here.\n'))

scripts/README.md:44

• The README states broadly that “A marker inside a string literal is not a comment”, but the implementation blanks quoted spans on a per-line basis (see scripts/prose_lint.py:253-295). For syntaxes that allow multi-line strings (notably C# verbatim strings), comment markers on subsequent lines inside the string can still be misread as comments. Tighten the wording here to reflect the line-scoped behavior so consumers don’t assume full multi-line string tracking.

JSON is treated as JSONC, because that is what ships: VS Code tasks, launch, devcontainer and workspace files all carry comments under a plain `.json` name. A marker inside a string literal is not a comment, so each line is scanned with quoted spans blanked first, and Python uses `tokenize` so a trailing comment is seen exactly. A documentation comment (`///`, `/**`, a docstring) is left to CODESTYLE, which permits the paragraphs this rule forbids.

[ptr727]

Answering the two findings in the round-four low-confidence block.

test_prose_lint.py:452, the description says #462 is deferred. This one does not hold against the current description. It was true of an earlier revision, and the body was updated when #464 landed the fix, before this review was requested. Line 33 now reads:

#462 is fixed there too rather than deferred, since a defect left in the promoted diff keeps being found and the loop cannot reach a clean round while it stands.

#462 is closed. Nothing to change, and I am noting it rather than silently ignoring it, since a stale finding is still worth answering.

scripts/README.md:44, the string-masking claim is broader than the behavior. Right, and the behavior was the larger half. A C# verbatim string spans lines while blanking is per line, so a marker on any later line of one was read as a comment:

var s = @"line one
// Two things. Here.      <- string content, reported as a comment
line three";

That is a false positive, the direction that is worse than a miss, since it asks a reader to edit text that is data. Fixed in #465 rather than reworded around: strip_strings now reports whether it ended inside a verbatim string and the extractor carries that state, as it already does for an unclosed block or documentation block.

The README takes the wording fix too, and goes further than the finding asked. It now says blanking is per line, that the verbatim string is carried because it spans lines, and names the heredoc and the YAML block scalar as forms that are not carried. Naming the remaining gap beats a claim a reader would have to disprove.

This PR picks both up when #465 squashes in.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

scripts/prose_lint.py:464

• When extracting block comments, the body is normalized with .lstrip('') for all block syntaxes. This is appropriate for C-style '/ ... /' where interior lines are often prefixed with '', but it should not be applied to other block comment syntaxes (e.g., ''), since it can alter the actual comment text and affect comment-wrap/comment-case detection.

            opener, closer = found
            end = line.find(closer, at + len(opener))    # a quote in the comment is prose
            body = (line[at + len(opener):end if end >= 0 else None]).strip().lstrip('*').strip()
            if body:
                out.append((n, body, leading))

scripts/prose_lint.py:417

• extracted_comments() unconditionally strips leading '' from block-comment bodies. That behavior is specific to C-style block comments (lines commonly prefixed with ''), but it will also mutate non-C block comment syntaxes like HTML comments ('') and PowerShell blocks ('<# ... #>'), which can change what the comment-prose rules see and lead to incorrect lint results.

This issue also appears on line 460 of the same file.

        elif closing:                                # carried in from an unclosed block
            end = line.find(closing)
            body = (line if end < 0 else line[:end]).strip().lstrip('*').strip()
            if body:
                out.append((n, body, True))

[ptr727]

Answering the two findings in the round-five low-confidence block. They are the same defect reported against both places that do it, and both are right. Fixed in #467.

A leading * was taken off every block comment body. That is the /* */ convention for continuing a line, and it is ordinary prose in every other block syntax:

<!-- *emphasis* leads here -->   ->  'emphasis* leads here'  ->  ['comment-case']
<#  *emphasis* leads here  #>    ->  'emphasis* leads here'  ->  ['comment-case']
/*  *emphasis* leads here  */    ->  'emphasis* leads here'  ->  ['comment-case']

The extractor removed the emphasis marker, that left a lowercase opening, and comment-case reported a finding the extractor had created. Worse than a plain false positive, because the prose it reports is not what the file contains.

Stripping now happens only on a line continuing a /* */ block, which is the line the convention describes. The opening line keeps its text in every syntax, C included, since a * directly after /* is content. A fourth case pins the convention where it does apply, so the fix cannot be read as removing it.

This one is pre-existing. .lstrip('*') dates to d4c4085, the commit being promoted, rather than to any of the fixes made during this review, so it would reach main for the first time with this promotion. That is the argument for taking it here rather than after.

This PR picks it up when #467 squashes in.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.