ai.robots.txt

This list contains AI-related crawlers of all types, regardless of purpose. We encourage you to contribute to and implement this list on your own site. See information about the listed crawlers and the FAQ.

A number of these crawlers have been sourced from Known Agents and we appreciate the ongoing effort they put in to track these crawlers.

If you'd like to add an AI-related crawler to the list, please see "Contributing" below.

Usage

This repository provides the following files in the top-level directory¹:

• robots.txt
• .htaccess
• nginx-block-ai-bots.conf
• Caddyfile
• haproxy-block-ai-bots.txt
• lighttpd-block-ai-bots.conf

robots.txt implements the Robots Exclusion Protocol (RFC 9309).

.htaccess may be used to configure web servers such as Apache httpd to return an error page when one of the listed AI crawlers sends a request to the web server. Note that, as stated in the httpd documentation, more performant methods than an .htaccess file exist.

nginx-block-ai-bots.conf implements a Nginx configuration snippet that can be included in any virtual host server {} block via the include directive.

Caddyfile includes a Header Regex matcher group you can copy or import into your Caddyfile, the rejection can then be handled as followed abort @aibots

haproxy-block-ai-bots.txt may be used to configure HAProxy to block AI bots. To implement it:

1. Add the file to the config directory of HAProxy

2. Add the following lines in the frontend section:

   acl ai_robot hdr_sub(user-agent) -i -f /etc/haproxy/haproxy-block-ai-bots.txt
   http-request deny if ai_robot
   

   (Note that the path of the haproxy-block-ai-bots.txt may be different in your environment.)

lighttpd-block-ai-bots.conf can be included with include "fragments/lighttpd-block-ai-bots.conf" in your lighttpd configuration either globally or in any conditional section.

Bing uses the data it crawls for AI and training, you may opt out by adding a meta tag to the head of your site.

Related

• Robots.txt Traefik plugin: middleware plugin for Traefik to automatically add rules of robots.txt file on-the-fly.

• Alternatively you can manually configure Traefik to centrally serve a static robots.txt.

• Bot Ledger: free, static directory of verified AI crawlers with a one-click robots.txt and llms.txt generator. No signup required.

• KI-Zugangsindex: open dataset on how widely this kind of blocking is actually deployed in the German (.de) web, measured on a fixed panel of 600 domains so the same sites can be re-checked over time.

• AI Crawler Census: open dataset measuring which of these crawlers the Tranco top 5,000 sites allow or block, with per-domain results published for each run so the same sites can be compared over time. Raw JSON, CC BY 4.0.

• AI Discovery Radar: monthly measurement of how many websites actually publish the files that tell AI systems what they may read or use (robots.txt, llms.txt, ai.txt, tdmrep.json and similar), and whether those files can be fetched at all. Open data, CC BY 4.0.

Contributing

Please note that AI-generated contributions are not permitted.

A note about contributing: updates should be added/made to robots.json. A GitHub action will then generate the updated robots.txt, table-of-bot-metrics.md, .htaccess and nginx-block-ai-bots.conf.

You can run the tests by installing Python 3, installing the dependencies:

pip install -r requirements.txt

and then issuing:

code/tests.py

The .editorconfig file provides standard editor options for this project. See EditorConfig for more information.

Releasing

Admins may ship a new release v1.n (where n increments the minor version of the current release) as follows:

• Navigate to the new release page on GitHub.
• Click Select tag, choose Create new tag, enter v1.n in the pop-up, and click Create.
• Enter a suitable release title (e.g. v1.n: adds user-agent1, user-agent2).
• Click Generate release notes.
• Click Publish release.

A GitHub action will then add the asset robots.txt to the release. That's it.

Subscribe to updates

You can subscribe to list updates via RSS/Atom with the releases feed: https://github.com/ai-robots-txt/ai.robots.txt/releases.atom.

You can subscribe with Feedly, Inoreader, The Old Reader, Feedbin, or any other reader app.

Alternatively, you can also subscribe to new releases with your GitHub account by clicking the ⬇️ on "Watch" button at the top of this page, clicking "Custom" and selecting "Releases".

License content with RSL

It is also possible to license your content to AI companies in robots.txt using the Really Simple Licensing standard, with an option of collective bargaining. A plugin currently implements RSL as well as payment processing for WordPress sites.

Report abusive crawlers

If you use Cloudflare's hard block alongside this list, you can report abusive crawlers that don't respect robots.txt here. But even if you don't use Cloudflare's hard block, their list of verified bots may come in handy.

Additional resources

• Blocking Bots with Nginx by Robb Knight
• Blockin' bots. by Ethan Marcotte
• Blocking Bots With 11ty And Apache by fLaMEd fury
• Blockin' bots on Netlify by Jeremia Kimelman
• Blocking AI web crawlers by Glyn Normington
• Block AI Bots from Crawling Websites Using Robots.txt by Jonathan Gillham, Originality.AI
• AI Access Checker: see which AI crawlers a site's robots.txt allows or blocks by Saar Twito, Greadme

Footnotes

1. Other files with similar names appear in the directory code/test_files. These files are for test purposes only and you must not use them on your own site. ↩