chore: block AI scrapers in robots.txt

Block known AI training bots (GPTBot, ClaudeBot, CCBot, etc.)
from crawling the site.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

robots.txt

@@ -1,5 +1,22 @@
-User-agent: *
+# There is no search benefit to any AI models scraping sites - all they do is steal content for
-Disallow:
+# their own profit, attribution free, which leads to them serving our content without ever sending
+# users to us.
+# Reference: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
+# See: https://github.com/MattWilcox/native-base/blob/45f6e7a837104f5ad83a5c7e280fb9a4eb126219/robots.txt
 
-# Add additional rules as needed
+User-agent: CCBot
-# Example: Disallow: /private/
+User-agent: ChatGPT-User
+User-agent: GPTBot
+User-agent: Google-Extended
+User-agent: Omgilibot
+User-agent: Omgili
+User-agent: FacebookBot
+User-agent: Applebot-Extended
+User-agent: anthropic-ai
+User-agent: ClaudeBot
+User-agent: Diffbot
+User-agent: Bytespider
+User-agent: ImagesiftBot
+User-agent: PerplexityBot
+User-agent: cohere-ai
+Disallow: /