[saddlepaddle]

Why

https://app.superset.sh/robots.txt 307s to /sign-in?redirect=%2Frobots.txt and returns the sign-in HTML. The auth proxy matcher covers .txt, and the public-route list had no entry for it. Link-preview crawlers (Facebook, LinkedIn, Slack) fetch robots.txt before the URL someone pasted, and an HTML login page is at best unparseable and at worst read as a block. This is the most likely reason a public page did not unfurl on Facebook even though the page itself serves full og tags to the Facebook crawler.

What

• app/robots.ts serves User-agent: * / Allow: /, and /robots.txt joins the proxy's public routes. Public pages already carry noindex, and nothing else on this host is reachable without a session, so indexing behaviour is unchanged.
• The public page's og:image carries alt.

Verification

• bun test for proxy-routes and proxy (48 pass, including the new /robots.txt case) and bun run typecheck in apps/web: clean.
• Not verified live: needs the deploy. After it lands, curl -I https://app.superset.sh/robots.txt should be 200 text/plain, and a Facebook Sharing Debugger rescrape of https://app.superset.sh/page/unfurl-test-page-t8mcl5 should show the card.

Summary by CodeRabbit

• New Features
  • Added a public robots.txt file that allows search engine crawlers to access the site.
• Accessibility
  • Added descriptive alt text to page thumbnail images shared through Open Graph and Twitter metadata.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 03af5ee7-d775-4462-8b1e-a72ca87f4bd1

📥 Commits

Reviewing files that changed from the base of the PR and between c5989ca and 92c70b5.

📒 Files selected for processing (4)

• apps/web/src/app/page/[slug]/page.tsx
• apps/web/src/app/robots.ts
• apps/web/src/proxy-routes.test.ts
• apps/web/src/proxy-routes.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 6 remain after this review.

---

📝 Walkthrough

Walkthrough

The web app adds the shared page title as thumbnail image alt text. It also adds a robots route that allows all user agents to crawl / and makes /robots.txt a public route.

Changes

Crawler metadata and robots route

Layer / File(s)	Summary
Shared-page thumbnail metadata apps/web/src/app/page/[slug]/page.tsx	Thumbnail image metadata now uses the shared page title as its alt text.
Public robots route apps/web/src/app/robots.ts, apps/web/src/proxy-routes.ts, apps/web/src/proxy-routes.test.ts	The robots route allows all user agents to crawl /. /robots.txt is included in public routes and in the corresponding route test.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Suggested reviewers: harshithmullapudi

Merge Risk: ⚪ Minimal · up to 92c70

The public robots route can reach crawlers without a sign-in redirect, and the thumbnail metadata adds alt text. No merge-blocking risk is apparent; normal deployment checks can confirm production routing.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title uses conventional commit format and clearly describes serving robots.txt for public link-preview access.
Description check	✅ Passed	The description explains the problem, the implementation, and the verification performed. It omits the template Checklist and uses Why, What, and Verification headings instead of the exact template he…

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🧹 Preview Cleanup Complete

The following preview resources have been cleaned up:

• ✅ Neon database branch

Thank you for your contribution! 🎉