Fighting bots

With great timing, a post by Nicolas titled “Should I remove this blog from Google Search?” landed in my RSS feed earlier today. I’ve been thinking a lot about the current phase the web is going through, especially after reading all the various news related to Perplexity AI. I don’t think they’re especially bad or wrong in what they’re doing, I’m sure the other companies are equally as bad and they’re also not giving half of a fuck about ingesting whatever they can find if it helps make their products better. They don’t care about book authors, they don’t care about journalists and they for sure don’t care about small personal bloggers.

That brings up the question: what do we do? What even can we do? It’s obvious that robots.txt is no longer an option because most companies don’t even bother checking it. We can try to block the user agents at a server level but they can avoid that by simply sending a generic UA. We could de-list our sites but that would make it very hard for actual users to find our content and I suspect the point of writing for most of us is to share and connect with others. The legal system sure ain’t gonna fix this situation anytime soon. So what’s left? I guess there are only two options left:

1. Accept the fact that some dickheads will do whatever they want because that’s just the world we live in
2. Make everything private and only allow actual human beings access to our content

Both solutions are suboptimal. Reading Nicolas's post made me also think about something else. He wrote:

In the case of Perplexity for example, a company that obviously steals content, lies, doesn’t really credit its sources, and — on top of it all — ignores the robots.txt rules from websites. If I were the TechCrunch, the New York Times, and the Financial Times of the world, I would simply stop reporting on the company. Not a blip on the radar, radio silence, except for their next fuck up. And then good luck finding investors if no one talks about you. They had their chance, they blew it.

What if Google decided to do that? What if Google decided to not return any result related to a company like Perplexity? I know it’s obviously not going to happen but wouldn’t that be funny?