GEO Improvement: Create robots.txt and llms.txt for the docs site
Source audit: GEO Audit Report — 2026-07-08
Audit date: 2026-07-08
Run: §28960233276
Finding
Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)
Create /llms.txt for AI indexing: geo llms --base-url https://github.github.com/gh-aw. Note: llms.txt is an organizational signal, not a proven ranking factor. It helps structure content for AI systems.
The docs site (https://github.github.com/gh-aw/) scores 0/18 on robots.txt and 0/18 on llms.txt — neither file exists. This is the single largest combined gap in the audit: 36 raw points left unclaimed, affecting every one of the 20 audited pages universally.
Why this matters
Without a robots.txt, AI crawlers (GPTBot, ClaudeBot, PerplexityBot) receive zero explicit permission to index the docs site, reducing citation probability in ChatGPT, Claude, and Perplexity answers. Without llms.txt, LLM-based indexers cannot discover site structure, key resources, or navigate the documentation hierarchy — directly hurting AI-engine citation coverage for the project's primary documentation surface.
Suggested fix
- Create
static/robots.txt in the docs site source:
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://github.github.com/gh-aw/sitemap.xml
- Generate
llms.txt using the geo-optimizer CLI:
geo llms --base-url https://github.github.com/gh-aw
Place the output at static/llms.txt in the docs site source so it is served at https://github.github.com/gh-aw/llms.txt.
- Verify by re-running the audit:
geo audit https://github.github.com/gh-aw --output docs-site-audit.json
Expected score improvement: +28–36 points (Foundation → Good band).
Generated by 🌍 GEO Optimizer Daily Audit · 39.2 AIC · ⌖ 36 AIC · ⊞ 4.9K · ◷
GEO Improvement: Create robots.txt and llms.txt for the docs site
Source audit: GEO Audit Report — 2026-07-08
Audit date: 2026-07-08
Run: §28960233276
Finding
The docs site (
https://github.github.com/gh-aw/) scores 0/18 on robots.txt and 0/18 on llms.txt — neither file exists. This is the single largest combined gap in the audit: 36 raw points left unclaimed, affecting every one of the 20 audited pages universally.Why this matters
Without a
robots.txt, AI crawlers (GPTBot, ClaudeBot, PerplexityBot) receive zero explicit permission to index the docs site, reducing citation probability in ChatGPT, Claude, and Perplexity answers. Withoutllms.txt, LLM-based indexers cannot discover site structure, key resources, or navigate the documentation hierarchy — directly hurting AI-engine citation coverage for the project's primary documentation surface.Suggested fix
static/robots.txtin the docs site source:llms.txtusing the geo-optimizer CLI:Place the output at
static/llms.txtin the docs site source so it is served athttps://github.github.com/gh-aw/llms.txt.Expected score improvement: +28–36 points (Foundation → Good band).