Back to all gems

Gem

69-bot allowlist

Live

robots.txt explicit Allow for Tier-1 + Tier-2 + Tier-3 AI crawlers. GPTBot · OAI-SearchBot · ClaudeBot · Claude-SearchBot · PerplexityBot · Googlebot · Bingbot · etc. (Google-Extended and Applebot-Extended are content-use policy tokens, listed as policy · not traffic.)

Status on your site

  • Deployed and verified live
  • Auto-monitored · alerts on any regression
  • Last verified · 2 hours ago

Receipts

Validator output, schema dumps, screenshots, and deploy logs all sit in your vault. Download anytime.