GEO Improvement: Add robots.txt and llms.txt to docs site
Source audit: GEO Audit Report — 2026-07-09
Audit date: 2026-07-09
Run: §29035910430
Finding
"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)" — docs site robots_txt score: 0/18
"Create /llms.txt for AI indexing: geo llms --base-url https://github.github.com/gh-aw" — docs site llms_txt score: 0/18
Both checks currently score 0 on github.github.com/gh-aw/, costing the docs site 36 out of 100 possible points (36% of the maximum score). The sitemap average across all 20 pages is also 0 for both categories.
Why this matters
AI crawlers (GPTBot, ClaudeBot, PerplexityBot) look for explicit permission signals in robots.txt before deep-crawling a site, and use llms.txt to build a structured index of available content. Without these files, the docs site is invisible or deprioritized by all major AI engines, directly reducing citation coverage for gh-aw in AI-generated answers.
Suggested fix
-
robots.txt — Add a static/robots.txt file to the Docusaurus site (under website/static/ or equivalent) containing:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Allow: /
-
llms.txt — Run the geo CLI to generate the file and commit it under static/llms.txt:
geo llms --base-url https://github.github.com/gh-aw > website/static/llms.txt
Then add a link to it in the Docusaurus <head> if supported.
-
Re-run the GEO audit after deploying to verify both checks pass.
Generated by 🌍 GEO Optimizer Daily Audit · 35.9 AIC · ⌖ 12.4 AIC · ⊞ 4.9K · ◷
GEO Improvement: Add robots.txt and llms.txt to docs site
Source audit: GEO Audit Report — 2026-07-09
Audit date: 2026-07-09
Run: §29035910430
Finding
Both checks currently score 0 on
github.github.com/gh-aw/, costing the docs site 36 out of 100 possible points (36% of the maximum score). The sitemap average across all 20 pages is also 0 for both categories.Why this matters
AI crawlers (GPTBot, ClaudeBot, PerplexityBot) look for explicit permission signals in
robots.txtbefore deep-crawling a site, and usellms.txtto build a structured index of available content. Without these files, the docs site is invisible or deprioritized by all major AI engines, directly reducing citation coverage forgh-awin AI-generated answers.Suggested fix
robots.txt — Add a
static/robots.txtfile to the Docusaurus site (underwebsite/static/or equivalent) containing:llms.txt — Run the geo CLI to generate the file and commit it under
static/llms.txt:geo llms --base-url https://github.github.com/gh-aw > website/static/llms.txtThen add a link to it in the Docusaurus
<head>if supported.Re-run the GEO audit after deploying to verify both checks pass.