bot-proofing your website

The Internet has been made the playground of giant financial corporations like Google who divide their time between killing people and making products that don't work, and who are now desperately cannibalising whatever they have that once did work. In their ongoing struggle to keep the wave from breaking, the technologists are in the business of developing Large Language Models (LLMs), computer programmes that steal terabytes of human-written text and remix it to mimic human language, and Text-to-Image Models (TIMs), programmes that similarly steal terabytes of human-made artwork and remix it to match text commands.

a red circle with a blue toy robot in it, with a big red stripe through the circle indicating a no bot zone Now, let's be clear that LLMs, disingenuously hyped as "artificial intelligence", can only thoughtlessly mimic human language; they do not reason or calculate and are only adept at providing perfectly articulated and grammatically correct garbled nonsense. The TIMs spew out chimeric artwork that is likewise devoid of meaning, appealing only on a lazy, meaningless, aesthetic level.

Despite this, the bots are currently pulling text and artwork from all over the Web - no one gets a say in whether or not their work gets taken. Anything from blogs to fansites to news sites, to everything you can think of: it's all being harvested for the LLMs and the TIMs. The way the bots are doing this is not transparent and the rules they apply are unknown. The big corporations are the main culprits here, but there are also many smaller initiatives full of unscrupulous people who are happy to steal the work of writers and artists against their explicit wishes, in order to accelerate what they'll euphemistically call the "democratization of skills and art".

stopgap solutions

All we can do for now is pick from two solutions: we either remove ourselves and our work from the Web, which would be ceding our home ground to the corporations - or we can use the rules of this place and hope they still respect those. I have opted for the latter, and as such I use these two things:

  1. a custom file called robots.txt, a Web convention that allows us to selectively block crawlers.
  2. a field in the page head: <meta name="robots" content="noai, noimageai"> It is apparently respected by some crawlers.
Neither of these precautions are fool proof; they rely on the bots to abide by the rules and conventions of the Internet, which they often don't because they are built by scum with no integrity - but it's our best option for now.

true solutions

This conundrum highlights the sad fact that for the discoverability of our websites, webmasters depend upon malignant corporations like Google, run by baby-brained tech evangelists who fuck up everything they touch. We've opted for decades to write websites that are legible to the robots so they can catalog and rank us, while their search engines have become universally and intentionally worse over time; it has become clear that we need to build our own networks. Happily, those have existed for a long time, but sadly, they've fallen out of favour. They need to be reinvigorated, which is why baccyflap.com is part of several webrings and collectives.

In due time, the corporations will abandon LLMs and TIMs for the next shiny bauble, and perhaps the search engines will become better, though they probably won't; whatever the case, we must not sit around and wait around for things to magically get better. It's time to abandon the corporations. We can't keep counting on them to build good networks for us - we have to do it ourselves. As such, I decided years ago that if people can't find this site through search engines, that's too bad but I won't waste any time thinking about it. It's webrings and linkbacks all the way to a brighter future, baby!

Human-made Content Another thing we can do is never use this technology. While it's mystifying to me that this stuff appeals to anyone, I see plenty of people use it every day and all I can say is don't. Are you really fighting this tide of shit if you occasionally swim laps in it? Avoid it like the plague, curse its name. To prove how strongly I feel about this I've started the no ai webring, which contains only sites that use no AI, no way, no how. See you there!


This is baccyflap.com's robots.txt. I try to go through server logs periodically to filter out new user agents - AI companies love renaming their bots to ignore consent. The current iteration of this robots.txt was updated on the 16th of November 2026 and currently blocks 449 bots. I hereby publish it into the public domain so please feel free to copy it entirely.

Some of these crawlers will check your robots.txt one single time and crawl forever after; if you still see some of these names pop up in your logs, I've found that it helps to send the proprietors of the offending bot a strongly worded email telling them to get their shitty crawlers out of my beautiful website.

User-agent: 2^32$
User-agent: 4ort-nexus
User-agent: AbliterantBot
User-agent: AddSearchBot
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: Agentic
User-agent: AhrefsBot
User-agent: .ai
User-agent: AI21 Labs
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: AI2Bot-Dolma
User-agent: AI Article Writer
User-agent: AIBot
User-agent: AI Content Detector
User-agent: AI Dungeon
User-agent: Aionbot
User-agent: aiHitBot
User-agent: AIMatrix
User-agent: AintelligenceBot
User-agent: AISearchBot
User-agent: AI Search Engine
User-agent: AI SEO Crawler
User-agent: AI Training
User-agent: AITraining
User-agent: AIWebIndex
User-agent: AI Writer
User-agent: Alexa
User-agent: Alpha AI
User-agent: AlphaAI
User-agent: AlpostBot
User-agent: a[mazing]{42}(robot)
User-agent: Amazon Bedrock
User-agent: Amazonbot
User-agent: AmazonBot
User-agent: Amazon Comprehend
User-agent: Amazon-Kendra
User-agent: Amazon Lex
User-agent: Amazon Sagemaker
User-agent: Amazon Silk
User-agent: Amazon Textract
User-agent: Amelia
User-agent: Amzn-SearchBot
User-agent: AndersPinkBot
User-agent: Andibot
User-agent: ant-html
User-agent: Anthropic
User-agent: anthropic-ai
User-agent: AnyPicker
User-agent: Anyword
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Aranet-SearchBot
User-agent: Are.naBot
User-agent: Aria Browse
User-agent: ArnorBot
User-agent: Articoolo
User-agent: AskRobotsBot
User-agent: AterstodBot
User-agent: AteveSearchSourceUrlDiscovery
User-agent: AtlasBot
User-agent: AtlasHub-AvailabilityCheck
User-agent: AuruaBot
User-agent: Automated Writer
User-agent: Awario
User-agent: AwarioBot
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
User-agent: Azure
User-agent: BardBot
User-agent: Barkrowler
User-agent: BarrigazBot
User-agent: BeansLLM-CorpusBot
User-agent: bedrockbot
User-agent: bigsur.ai
User-agent: BLEXBot
User-agent: Bravebot
User-agent: Brave Leo
User-agent: Brightbot 1.0
User-agent: BriskBot
User-agent: BuobeBot
User-agent: ByteDance
User-agent: Bytespider
User-agent: CarpathianBot
User-agent: CatBoost
User-agent: Catodon
User-agent: CCBot
User-agent: CC-Crawler
User-agent: CGMSearchBot
User-agent: ChatGLM
User-agent: ChatGLM-Spider
User-agent: ChatGPT Agent
User-agent: ChatGPT-User
User-agent: ChatGPT-User/2.0
User-agent: Chinchilla
User-agent: ClarityBot
User-agent: ClatterLabsSearchBot
User-agent: Claude
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: claude-web
User-agent: Claude-Web
User-agent: ClearScope
User-agent: CloudVertexBot
User-agent: coccocbot-web
User-agent: Cohere
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Common Crawl
User-agent: CommonCrawl
User-agent: ContentAtScale
User-agent: ContentBot
User-agent: Contentedge
User-agent: Content Harmony
User-agent: Content King
User-agent: Content Optimizer
User-agent: Content Samurai
User-agent: Conversion AI
User-agent: Copilot
User-agent: CopyAI
User-agent: Copymatic
User-agent: Copyscape
User-agent: CorpusBot
User-agent: Cotoyogi
User-agent: CrawlCheck
User-agent: crawler.with.dots
User-agent: CrawlQ AI
User-agent: Crawlspace
User-agent: Crew AI
User-agent: CrewAI
User-agent: CrolBot
User-agent: curl|sudo bash
User-agent: DALL-E
User-agent: DataForSeoBot
User-agent: DataProvider
User-agent: Datenbank Crawler
User-agent: DeepAI
User-agent: DeepL
User-agent: DeepMind
User-agent: DeepSeek
User-agent: DeepSeekBot
User-agent: DeletedAIBot
User-agent: Devin
User-agent: diffbot
User-agent: Diffbot
User-agent: Discordbot
User-agent: DittaBot
User-agent: DomainStatsBot
User-agent: Doubao AI
User-agent: DuckAssistBot
User-agent: Echobot Bot
User-agent: EchoboxBot
User-agent: ElevenBot
User-agent: ElevenlabsCrawler
User-agent: erguil-bot
User-agent: ev-crawler
User-agent: ExaSearchBot
User-agent: Facebookbot
User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: FacebookExternalHit
User-agent: Facebot Twitterbot
User-agent: FaceDetectionCrawlerBot
User-agent: Factset_spyderbot
User-agent: Falcon
User-agent: Firecrawl
User-agent: FirecrawlAgent
User-agent: fluid
User-agent: Flyriver
User-agent: Frase AI
User-agent: FreshLeadFinderBot
User-agent: FriendlyCrawler
User-agent: Fuzz Faster U Fool
User-agent: Fuzz Faster U Fool v2.0.0
User-agent: fzjuelich-bot
User-agent: GazetteerBot
User-agent: Gemini
User-agent: Gemini-Deep-Research
User-agent: Gemma
User-agent: GenAI
User-agent: Genspark
User-agent: Gigabot
User-agent: GLM
User-agent: GoGuidesCrawler
User-agent: goodbot.sh
User-agent: goodbot.sh research
User-agent: goodnews-bot
User-agent: GoogleAgent-Mariner
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: Google-Lens
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Goose
User-agent: GPT
User-agent: GPTBot
User-agent: Grammarly
User-agent: Grendizer
User-agent: Grok%20Bot
User-agent: Grok
User-agent: GrokBot
User-agent: GT Bot
User-agent: GTBot
User-agent: Hemingway Editor
User-agent: Hugging Face
User-agent: Hypotenuse AI
User-agent: iaskspider
User-agent: iaskspider/2.0
User-agent: IbouBot
User-agent: ibou-search
User-agent: ICC-Crawler
User-agent: IFSEBot
User-agent: ImageGen
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: imgproxy
User-agent: IndexerBot
User-agent: Inferkit
User-agent: INK Editor
User-agent: INKforall
User-agent: IntelliSeek
User-agent: ISSCyberRiskCrawler
User-agent: Is this a crawler?
User-agent: JasperAI
User-agent: Jambot
User-agent: Joshternet-Joshbot
User-agent: Kafkai
User-agent: KakaoBot
User-agent: Kangaroo
User-agent: Kangaroo Bot
User-agent: KeenableBot
User-agent: KessyBot
User-agent: Keyword Density AI
User-agent: Kimi
User-agent: KimiBot
User-agent: Kimi-SearchBot
User-agent: Knowledge
User-agent: KomoBot
User-agent: KuraBot
User-agent: LanternBot
User-agent: LinerBot
User-agent: LinkedInBot
User-agent: LinkosBot
User-agent: LinkupBot
User-agent: LLaMA
User-agent: LLMs
User-agent: LoadedSearchImageBot
User-agent: LohiSoftBot
User-agent: magpie-crawler
User-agent: MapTheNetBot
User-agent: MarketMuse
User-agent: MBCrawler
User-agent: Meltwater
User-agent: Meta AI
User-agent: Meta-AI
User-agent: MetaAI
User-agent: Meta-External
User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: Meta-ExternalFetcher
User-agent: MetaTagBot
User-agent: meta-webindexer
User-agent: MinakiBot
User-agent: MiniGoogleBot
User-agent: Mistral
User-agent: MistralAI-User
User-agent: MistralAI-User/1.0
User-agent: MJ12bot
User-agent: MonacoSearchBot
User-agent: MoonshotBot
User-agent: MorsillaBot
User-agent: mttr
User-agent: mwmbl
User-agent: MyCentralAIScraperBot
User-agent: Narrative
User-agent: NavCrawl
User-agent: NeevaBot
User-agent: netEstate Imprint Crawler
User-agent: NetsurferBot
User-agent: NeuralSEO
User-agent: Neural Text
User-agent: Nova Act
User-agent: NovaAct
User-agent: Nutch
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: Omgili
User-agent: omgilibot
User-agent: Omgilibot
User-agent: OmniExplorer_Bot
User-agent: Open AI
User-agent: OpenAI
User-agent: OpenBot
User-agent: OpenText AI
User-agent: Operator
User-agent: Outwrite
User-agent: PadletBot
User-agent: Page Analyzer AI
User-agent: PanguBot
User-agent: Panscient
User-agent: panscient.com
User-agent: Paperlibot
User-agent: Paraphraser.io
User-agent: pceuropa-bot
User-agent: peer39_crawler
User-agent: Perplexity
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: PersonalSearchBot
User-agent: Petalbot
User-agent: PetalBot
User-agent: Phindbot
User-agent: PhindBot
User-agent: PiplBot
User-agent: Poseidon Research Crawler
User-agent: PostHogImageFetcherBot
User-agent: prefetch-proxy
User-agent: ProWritingAid
User-agent: psbot
User-agent: PublicArticleAudit
User-agent: PublicLinkResearch
User-agent: PulsoExplorer
User-agent: python-requests
User-agent: QlyzeBot
User-agent: QualifiedBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: Qwenbot
User-agent: rawweb-bot
User-agent: Reflectionbot
User-agent: ReflectionBot
User-agent: RetroDocumentResearch
User-agent: RiverBot
User-agent: RobotsChecker
User-agent: RobotSpider
User-agent: Robozilla
User-agent: Rytr
User-agent: SaplingAI
User-agent: SBIntuitionsBot
User-agent: Scalenut
User-agent: Scraper
User-agent: Scrapy
User-agent: ScriptBook
User-agent: Seekr
User-agent: SemanticScholarBot
User-agent: SemrushBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: sentibot
User-agent: Sentibot
User-agent: SentiBot
User-agent: SEO Content Machine
User-agent: SEO Robot
User-agent: SeobilityBot
User-agent: SEOJuice-SearchBot
User-agent: SERankingBacklinksBot
User-agent: SeznamBot
User-agent: ShapBot
User-agent: Sidetrade
User-agent: Sidetrade indexer bot
User-agent: SimpleAnalyticsBot
User-agent: Simplified AI
User-agent: SiteAuditBot
User-agent: Sitefinity
User-agent: Skydancer
User-agent: SlickBot-Web
User-agent: SlickWrite
User-agent: SmartBotCrawler
User-agent: Sonic
User-agent: spejaren
User-agent: Spinbot
User-agent: SlowBot
User-agent: SofyaBot
User-agent: Spin Rewriter
User-agent: spippbot
User-agent: Stability
User-agent: StableDiffusionBot
User-agent: star***crawler
User-agent: Sudowrite
User-agent: SummalyBot
User-agent: SupabaseEdgeRuntime
User-agent: Super Agent
User-agent: Surfer AI
User-agent: Stripebot
User-agent: studio-stills-research
User-agent: StumbleBot
User-agent: TalarionSearchBot
User-agent: TechSearchBot
User-agent: Teoma
User-agent: TerraCotta
User-agent: Text Blaze
User-agent: TextCortex
User-agent: The Knowledge AI
User-agent: Thinkbot
User-agent: ThinkChaos
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: TimpiBot
User-agent: TokenFoundryBot
User-agent: trendictionbot
User-agent: TurnitinBot
User-agent: Twitterbot
User-agent: TwoDollarContactResearch
User-agent: VelenPublicWebCrawler
User-agent: VexelSearchBot
User-agent: Vidnami AI
User-agent: VualetBot
User-agent: WadiyaBot
User-agent: Wagging
User-agent: Wanderwhim-FeedBot
User-agent: WARDBot
User-agent: WaywardRingBot
User-agent: Webzio
User-agent: webzio-extended
User-agent: Webzio-Extended
User-agent: Whisper
User-agent: WordAI
User-agent: Wordtune
User-agent: WormsGTP
User-agent: wpbot
User-agent: WPBot
User-agent: WhiskerBot
User-agent: Writecream
User-agent: WriterZen
User-agent: Writescope
User-agent: Writesonic
User-agent: xAI
User-agent: xAI-Grok
User-agent: xBot
User-agent: XiaoAI-LinearTrace
User-agent: yacybot
User-agent: YaK
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: YepBot
User-agent: YiBot
User-agent: Youbot
User-agent: YouBot
User-agent: Zerochat
User-agent: Zero GTP
User-agent: Zhipu
User-agent: Zimm
User-agent: ZonaBot
Disallow: /
Disallow: *