fix(lychee): exclude auth-walled URLs

[kyle-sexton]

Refs melodic-software/claude-code-plugins#18

Summary

• scope public bot-blocking exclusions to the five exact URL paths currently present downstream: one Medium article, one Miro help article, IsDown's Anthropic status page, and two npm package pages
• allow only optional trailing slash, query, or fragment variants of those exact paths; unrelated current or future URLs on the same hosts remain checked
• extend the exact private-repository inventory to matching raw.githubusercontent.com content URLs
• keep public sibling and stale-owner URLs checked
• expand fixtures and assertions for every excluded URL and same-host non-excluded boundary

Why

The downstream claude-code-plugins online lane reaches extraction successfully with Lychee 0.24.2, but current content exposes expected auth-boundary failures: five public paths that reject automated probes and raw-content URLs for private repositories. The shared Lychee ruleset is standards-managed, so the fix belongs here and will return through the governed sync workflow.

Lychee's current official documentation confirms that exclude entries are URL regular expressions and demonstrates exact-URL exclusions:

• https://lychee.cli.rs/guides/config/#excluding-links
• https://lychee.cli.rs/recipes/excluding-links/
• https://lychee.cli.rs/recipes/excluding-paths/

GitHub's repository contents documentation notes that private-repository download URLs are authenticated and temporary:

• https://docs.github.com/en/rest/repos/contents

Validation

• bash components/lychee/lychee.test.sh: 27 passed
• ShellCheck: passed
• shfmt diff check: passed
• Markdown lint on changed docs/fixtures: 0 errors
• downstream online check with proposed config: 1,655 links, 0 errors
• downstream offline check with proposed config: 1,655 links, 0 errors
• independent fresh-agent review: no correctness or security blockers

[claude]

Claude finished @kyle-sexton's task in 0s —— View job

---

I'll analyze this and get back to you.

[claude]

Claude finished @kyle-sexton's task in 2m 53s —— View job

---

PR Review

• Gather context (CLAUDE.md, REVIEW.md, diff)
• Review changed files
• Post findings

---

Verdict: Looks good with two minor observations.

The two-commit shape is clean: commit 1 added the raw.githubusercontent.com arm and initially used wide host exclusions for medium/miro/isdown/npmjs; commit 2 immediately narrowed those to exact path-scoped patterns. The final state is correct.

Regex correctness

All four path-scoped patterns use the right boundary:

/?([?#].*)?$

This correctly matches the bare path, an optional trailing slash, and an optional query/fragment — while rejecting longer paths or slugs that happen to share the prefix. Verified mentally for each of the four patterns.

The raw.githubusercontent.com pattern:

'^https?://raw\.githubusercontent\.com/melodic-software/(claude-code-plugins|dotfiles|github-iac|medley|provisioning|standards)/',

is symmetrical with the existing github.com inventory rule and the trailing / prevents it from accidentally matching a hypothetical standards-extra repo. Correct.

Test design

The structural grep -F / assert_eq assertions that pin the exact TOML line text are a good idea — they guarantee the two inventory alternation groups stay in sync and will catch a copy-paste drift before lychee even runs. The positive/negative split (assert_contains for siblings that must stay checked, assert_not_contains for excluded URLs) gives good boundary coverage.

Minor observations

1. Raw-content dump assertion only covers one of the six repos (lychee.test.sh:41)

The github.com exclusion is verified for all six repos via a for loop (lines 36–39), but raw.githubusercontent.com is only asserted for claude-code-plugins. Since the regex alternation is identical, this is not a correctness gap — if one matches, all six do — but the asymmetry with the for loop is slightly surprising.

2. README omits the pre-existing wide-host exclusions (README.md:14–23)

The README accurately describes the path-scoped exceptions (medium/miro/isdown/npmjs) and the private-repo inventory, but x.com, twitter.com, linkedin.com, and bsky.app (pre-existing wide host exclusions) go unmentioned. A future reader scanning the README will not find them documented. This pre-dates the PR, but since README.md was updated here, it would be a natural place to close the gap.

Neither observation blocks the merge. The raw.githubusercontent.com addition is correct and the path-narrowing in commit 2 is the right call.

[claude]

Claude finished @kyle-sexton's task in 1m 57s —— View job

---

PR Review

• Gather context (CLAUDE.md, REVIEW.md, diff)
• Review changed files
• Post findings

---

Verdict: Ready to merge. The regex patterns are correctly scoped, the test coverage is thorough, and the documentation is accurate.

lychee.toml

All four path-scoped patterns use the correct boundary:

/?([?#].*)?$

This matches the bare path and allows an optional trailing slash and query/fragment, while correctly rejecting any longer path or slug that shares the prefix. Spot-checked against the medium, miro, isdown, and npmjs patterns — all correct.

One non-obvious case worth verifying: the scoped npm package name @mirohq/miro-api contains a literal /. Inside the alternation group (firecrawl-cli|@mirohq/miro-api), the / matches a literal slash in the URL. After the group, /? can only consume one more /, and ([?#].*)?$ then requires end-of-string or a ?/# character. This means a URL like https://www.npmjs.com/package/@mirohq/miro-api/v/1.0.0 correctly falls through and stays checked. Pattern is right.

The raw.githubusercontent.com rule:

'^https?://raw\.githubusercontent\.com/melodic-software/(claude-code-plugins|dotfiles|github-iac|medley|provisioning|standards)/',

is symmetrical with the existing github.com inventory rule. The trailing / acts as a boundary guard — a hypothetical standards-extra repo would not be accidentally excluded.

lychee.test.sh

The assert_eq structural tests that pin the exact TOML lines are the right call — they catch alternation-group drift before lychee even runs. The assert_not_contains / assert_contains split on same-host boundary URLs gives solid coverage of what's excluded vs. what stays checked.

Minor observations (non-blocking)

1. Raw-content dump assertion covers only one of the six repos (line 41)

The github.com exclusion is verified for all six repos via a for loop (lines 36–39), but raw.githubusercontent.com is only asserted for claude-code-plugins. Not a correctness gap — the assert_eq structural test at lines 24–28 already pins the full alternation group in TOML — but the asymmetry with the for loop is mildly surprising.

2. README does not document the pre-existing wide-host social exclusions (README.md lines 14–23)

x.com, twitter.com, linkedin.com, and bsky.app are not mentioned in the README even though it was updated in this PR. A future maintainer reading only the README would not know those blanket exclusions exist. Pre-dates this change and does not block the merge.

Neither observation requires action before merging.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 97cd1037b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep placeholder boundary links out of online checks

The scheduled/on-demand online link check scans ./**/*.md and only excludes components/[^/]+/fixtures/bad in .github/workflows/link-check.yml, so this new good fixture is still crawled. This placeholder Miro article is intentionally left non-excluded, but it is not a valid article URL, so the advisory job will report a broken external link whenever it runs; use a real same-host URL or exclude this boundary-only fixture from the online glob.

Useful? React with 👍 / 👎.