home tags events about login
one honk maybe more

benjojo posted 02 Aug 2025 22:54 +0000

Quite surprised (In a positive way?) that the default when you are adding a new domain to cloudflare is to block/drop AI scrapers, and that the options take up as much room as the most foundational questions "where are the DNS records going to come from?"

Sort of a indicator on the side of the AI companies that all of this is unsustainable if the publishers of actual content hate/discriminate against them this much

A screenshot from the cloudflare menu to add a (censored) domain, half of the page is the options for how DNS will be obtained, "Manually Enter DNS records" is selected, and the rest of the page is "Control how AI scralwers access you site", and "block ai traning bots on all pages" with another check box for "instruct AI bot traffic with robots.txt"