Skip to content

Add full AI crawler allow-list to docs site robots.txt #49820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 2 commits into from
Aug 2, 2026
Merged

Add full AI crawler allow-list to docs site robots.txt #49820

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bd238f7
Initial plan
Copilot Aug 2, 2026
fe4c923
Add AI crawler rules to docs robots.txt
Copilot Aug 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,13 @@ All commands are run from the root of the project, from a terminal:

## ⚠️ Known Dev-Mode Limitations

### Sitemap not available in dev mode
### Sitemap behavior in dev and production

The sitemap (`/gh-aw/sitemap-index.xml`) is **only generated during a production build** (`npm run build`). It is not available when running the local development server (`npm run dev`).
The robots file references `/gh-aw/sitemap.xml`, which is the stable sitemap entrypoint for the docs site.

If a CI pipeline or automated tool checks for the sitemap URL during a local preview, it will receive a 404 response. To verify the sitemap, run `npm run build` followed by `npm run preview`.
During a production build (`npm run build`), Astro generates `/gh-aw/sitemap-index.xml`, and the static `/gh-aw/sitemap.xml` entrypoint points crawlers at that generated sitemap index. The generated sitemap index is not available when running the local development server (`npm run dev`).

If a CI pipeline or automated tool checks the generated sitemap index URL during a local preview, it will receive a 404 response. To verify the production sitemap flow, run `npm run build` followed by `npm run preview`.

### Robots/AI discovery paths on GitHub Pages project sites

Expand Down
35 changes: 28 additions & 7 deletions docs/public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ Allow: /
User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
User-agent: claude-web
Allow: /

User-agent: Googlebot
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Expand All @@ -37,13 +37,25 @@ Allow: /
User-agent: Bingbot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: xAI-Bot
User-agent: Bytespider
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Meta-ExternalFetcher
Allow: /

User-agent: facebookexternalhit
Allow: /

User-agent: Amazonbot
Expand All @@ -52,13 +64,22 @@ Allow: /
User-agent: AI2Bot
Allow: /

User-agent: YouBot
User-agent: AI2Bot-Dolma
Allow: /

User-agent: CCBot
User-agent: xAI-Bot
Allow: /

User-agent: Applebot-Extended
User-agent: Applebot
Allow: /

User-agent: PetalBot
Allow: /

User-agent: YouBot
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap-index.xml
Sitemap: https://github.github.com/gh-aw/sitemap.xml
34 changes: 29 additions & 5 deletions docs/tests/robots-txt.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ const EXPECTED_ROBOTS_TXT = [
'User-agent: Claude-SearchBot',
'Allow: /',
'',
'User-agent: PerplexityBot',
'User-agent: claude-web',
'Allow: /',
'',
'User-agent: Googlebot',
'User-agent: PerplexityBot',
'Allow: /',
'',
'User-agent: Perplexity-User',
Expand All @@ -40,13 +40,25 @@ const EXPECTED_ROBOTS_TXT = [
'User-agent: Bingbot',
'Allow: /',
'',
'User-agent: Applebot-Extended',
'Allow: /',
'',
'User-agent: cohere-ai',
'Allow: /',
'',
'User-agent: DuckAssistBot',
'Allow: /',
'',
'User-agent: xAI-Bot',
'User-agent: Bytespider',
'Allow: /',
'',
'User-agent: meta-externalagent',
'Allow: /',
'',
'User-agent: Meta-ExternalFetcher',
'Allow: /',
'',
'User-agent: facebookexternalhit',
'Allow: /',
'',
'User-agent: Amazonbot',
Expand All @@ -55,18 +67,30 @@ const EXPECTED_ROBOTS_TXT = [
'User-agent: AI2Bot',
'Allow: /',
'',
'User-agent: AI2Bot-Dolma',
'Allow: /',
'',
'User-agent: xAI-Bot',
'Allow: /',
'',
'User-agent: Applebot',
'Allow: /',
'',
'User-agent: PetalBot',
'Allow: /',
'',
'User-agent: YouBot',
'Allow: /',
'',
'User-agent: CCBot',
'Allow: /',
'',
'Sitemap: https://github.github.com/gh-aw/sitemap-index.xml',
'Sitemap: https://github.github.com/gh-aw/sitemap.xml',
'',
].join('\n');

test.describe('robots.txt', () => {
test('should contain only the expected AI crawler directives and sitemap index', async ({ request }) => {
test('should contain the expected AI crawler directives and sitemap', async ({ request }) => {
const response = await request.get('/gh-aw/robots.txt');
expect(response.ok()).toBeTruthy();

Expand Down