robots.txt and llms.txt generator for AI crawlers, with the trade-offs explained
Blocking a training crawler costs you nothing. Blocking a search crawler costs you readers. Most guidance never explains which is which.
Blocking an AI training crawler costs you no traffic. Blocking an AI search crawler removes you from answers that link back to you. Most guides never explain which bot is which. This lets you choose per crawler, tells you what each choice means, and writes both files ready to upload.
Free, experimental, provided as-is. This tool applies rules of thumb and can be wrong. It is not legal, financial or professional advice, and using it creates no professional relationship with the site owner. You are responsible for any decision you make with the result. Terms.
While you’re here
Other jobs a clerk can take off your desk.
RPM Index
BetaYour ad revenue per thousand views, and what it’s worth to move it. Submit yours, see everyone’s.
Headline Bench
LiveMeasures a title tag instead of generating one: where it gets cut, whether the keyword survives, whether it matches the H1.
Crawler Ledger
LiveDrop in a server log. See which AI crawlers took what, how often, and which pages they went for.