title: "My responses to The Register" source: "https://xeiaso.net/notes/2025/el-reg-responses/?utm_campaign=mi_irl&utm_medium=social&utm_source=bluesky" author: published: 2025-08-22 created: 2025-08-23 description: "A copy of my best quotes they didn't publish" tags:

• "clippings"

Published on , 669 words, 3 minutes to read

A copy of my best quotes they didn't publish

Today my quotes about generative AI scrapers got published in The Register. For transparency's sake, here's a copy of the questions I was asked and my raw, unedited responses. Enjoy!

First, do you see the growth in crawler traffic slowing any time soon?

I can only see one thing causing this to stop: the AI bubble popping. There is simply too much hype to give people worse versions of documents, emails, and websites otherwise. I don't know what this actually gives people, but our industry takes great pride in doing this.

Is it likely to continue growing?

I see no reason why it would not grow. People are using these tools to replace knowledge and gaining skills. There's no reason to assume that this attack against our cultural sense of thrift will not continue. This is the perfect attack against middle-management: unsleeping automatons that never get sick, go on vacation, or need to be paid health insurance that can produce output that superficially resembles the output of human employees. I see no reason that this will continue to grow until and unless the bubble pops. Even then, a lot of those scrapers will probably stick around until their venture capital runs out.

If so, how can that be sustainable?

It's not lol. We are destroying the commons in order to get hypothetical gains. The last big AI breakthrough happened with GPT-4 in 2023. The rest has been incremental improvements. Even with scrapers burning everything in their wake, there is not enough training data to create another exponential breakthrough. All we can do now is make it more efficient to run GPT-4 level models on lesser hardware. I can (and regularly do) run a model just as good as GPT-4 on my MacBook at this point, which is really cool.

Would broader deployment of Anubis and other active countermeasures help?

This is a regulatory issue. The thing that needs to happen is that governments need to step in and give these AI companies that are destroying the digital common good existentially threatening fines and make them pay reparations to the communities they are harming. Ironically enough, most of these AI companies rely on the communities they are destroying. This presents the kind of paradox that I would expect to read in a Neal Stephenson book from the 90's, not CBC's front page.

Anubis helps mitigate a lot of the badness by making attacks more computationally expensive. Anubis (even in configurations that omit proof of work) makes attackers have to retool their scraping to use headless browsers instead of blindly scraping HTML. This increases the infrastructure costs of the AI companies propagating this abusive traffic. The hope is that this makes it fiscally unviable for AI companies to scrape by making them have to dedicate much more hardware to the problem.

In essence: it makes the scrapers have to spend more money to do the same work.

Is regulation required to prevent abuse of the open web?

Yes, but this regulation would have to be global, simultaneous, and permanent to have any chance of this actually having a positive impact. Our society cannot currently regulate against similar existential threats like climate change. I have no hope for such regulation to be made regarding generative AI.

Fastly's claims that 80% of bot traffic is now AI crawlers

In some cases for open source projects, we've seen upwards of 95% of traffic being AI crawlers. For one, deploying Anubis almost instantly caused server load to crater by so much that it made them think they accidentally took their site offline. One of my customers had their power bills drop by a significant fraction after deploying Anubis. It's nuts.

Personally, deploying Anubis to my blog has reduced the amount of ad impressions I've been giving by over 50%. I suspect that there is a lot of unreported click fraud for online advertising.

I hope this helps. Keep up the good fight!

Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Tags: