Repository navigation
test: validate documentation against the skill catalog - #56
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c1940accb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Architecture diagram
sequenceDiagram
participant CLI as validate.py CLI
participant V as validate()
participant VC as validate_catalog()
participant FS as Checkout Filesystem
participant MD as Markdown Prose Extractor
participant F as Failure Collector
Note over CLI,F: Tool validation flow
CLI->>V: Run validation
V->>V: Validate structure (names, policies, links, manifests)
V->>VC: NEW: Call validate_catalog()
Note over VC,FS: Catalog Count Verification
VC->>FS: Count skills/*/SKILL.md
VC->>FS: Count principle-*/SKILL.md
VC->>FS: Count playbooks/*.md
VC->>FS: Count unique agent roles (codex/*.toml + claude/*.md)
FS-->>VC: Tree counts
VC->>MD: Extract prose from README.md and docs/index.md
MD-->>VC: Prose content
VC->>VC: Parse advertised counts (e.g., "53 skills")
alt Count mismatch
VC->>F: Record "Catalog says X, but tree has Y"
end
Note over VC,FS: Current-Source Link Verification
VC->>FS: Gather README.md, docs/guide/*.md, docs/reference/*.md
VC->>MD: Strip fenced code blocks
MD-->>VC: Prose with links
VC->>VC: Match fullerzz/agent-skills/blob/main/ links
VC->>FS: Check target file exists in checkout
FS-->>VC: File exists or missing
alt Broken source link
VC->>F: Record "Broken current-source link"
end
V-->>CLI: Validation result (failures)
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15756f50ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 275b192576
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fbf8f1cbf3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 20fea1fd44
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for file in files: | ||
| if not file.is_file(): | ||
| continue | ||
| text = file.read_text(encoding="utf-8") |
There was a problem hiding this comment.
When any guide or reference Markdown file contains invalid UTF-8, the main walk records the UnicodeDecodeError as a structural failure, but this unconditional second read_text() raises the same error outside the per-file handler and terminates validation with a traceback. Catch read/parse errors here and report them through fail so one malformed documentation file does not prevent the remaining checks from running.
Useful? React with 👍 / 👎.
| opening = re.match(r"^ {0,3}(`{3,}|~{3,})(.*)$", line) | ||
| if opening and (opening[1][0] == "~" or "`" not in opening[2]): |
There was a problem hiding this comment.
When README or docs/index.md contains a fenced example nested in a block quote or list, such as an opening line prefixed by > or -, this expression does not recognize the CommonMark fence and the catalog scanner interprets sample text like 99 skills as a real advertised total. Fresh evidence beyond the earlier top-level tilde-fence report is that container-prefixed fences remain in markdown_prose; account for Markdown container prefixes or derive prose from parser tokens.
Useful? React with 👍 / 👎.
1 participant
Upstream #702 | Upstream #705 | Upstream #704
Why the change
Documentation totals and source links can drift from the installed skill tree, so the existing validator now checks them against the checkout.
Special things to note
0.13.0.Change outline