bestdadjokes/site/robots.txt
djg 4c210f2215
All checks were successful
Build and Deploy / deploy (push) Successful in 20s
feat(blog): add /blog/ curated lists section, llms.txt, and AI bot discovery
- Add static /blog/ directory and curated listicle pages (Thanksgiving, Halloween, and Kids)
- Include Article, BreadcrumbList, and FAQPage JSON-LD structured schemas on all listicle pages
- Add quick-copy buttons and instant punchline displays optimized for high-intent searchers
- Add /llms.txt standard file documenting core hubs, seasonal clusters, and open APIs
- Explicitly authorize major AI search crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended) in robots.txt
- Add Blog link to site header navigation and footer
2026-10-01 22:48:12 +00:00

24 lines
357 B
Text

# robots.txt for bestdadjokes.lol
User-agent: *
Allow: /
# Explicit AI Search & Discovery Crawlers
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
Sitemap: https://bestdadjokes.lol/sitemap.xml