blocking random IPs and shoving everything through mazes of captchas and proof of work and giving poisoned data to anyone who isn’t using a “correct” web browser to try and stop “AI scrapers” it just feels a bit like we’ve created a medicine that’s worse than the disease here

my neighbor keeps using garden hose to fill up their swimming pool. should i dump a bunch of rat poison into my well to stop them

you guys do realize that all these hallmarks of “AI” browsers/scrapers are also the same things that accessibility tools do, right? because both interact with a webpage as a big pile of text, not as a bunch of 2d images you can click on.

but it’s fine. we should be feeding deliberately fake data to any browser that doesn’t act exactly like the latest version of google chrome with all tracking features enabled. we should block entire ASNs permanently because a single connection did something that looked kind of weird in one of your access logs. we should force every device to burn compute hours solving proof of waste puzzles because we need to fight those evil AI companies wasting compute hours. we are the good guys and we are doing the right thing.

clearly, the only solution is to ensure that everyone connecting to your server is actually a person. i propose that every ID card should contain a smart chip which contains a cryptographic key. by verifying the connection has a valid user, you can ensure that there are no more automated scrapers! simply rate limit each individual person’s access. This system will not be used by any other people for any other things, and will be a net good for society and the freedom of information

those who judiciously block random traffic are just as evil to me as big tech companies running poorly behaved crawlers. i hate both of you because you both want to stand over your little fiefdoms of the information you hoard privately as if the capacity of your hard drives is a proxy for the size of your metaphorical penis, and your kink is making people beg to get just a few more nybbles of that sweet sweet data from you.

anyways i need to draw more of my OC who’s an AI agent who’s soooo thirsty for your data. her name is sydney btw real ones get it

@eri I get this, but AI scrapers are literally DDoSing every website they know about, so the options in 2026 seem to be:

• have a website that doesn't work for anyone because it's overloaded 100% of the time
• have a website that rejects Chrome user agents specifically (because all the scrapers are pretending to be Chrome)
• don't have a website

@tully that’s weird because they aren’t doing that to my website

@eri incidentally, there's a reason we don't have a website

@tully @eri My primary approach is to rate limit everyone to a human pace.

That seemed to clear up all my issues!

@alcinnz @tully what shocks me is so many sysadmins going straight to the nuclear option when a basic rate limit is more than enough for most problematic endpoints

@illyBytes @eri yeah… I'm dreading the day the scrapers find our NextCloud instance, or our GTS, or any of the web-facing things we have

@tully @illyBytes if you have an HTTPS cert they already know about your stuff since those are all publicly logged. maybe there’s a reason only some sysadmins are having problems. maybe that reason is poorly programed applications being ran by people who don’t understand how they work

@tully @eri my website has a user agent filter that severs connections to known AI scrapers but doesn't do blanket Chrome bans

@sterophonick glad that still works for you

we're dreading the day we have to set up mitigations. we happily write code, but loathe sysadmin work.

@tully it's definitely taken a considerable load off of my site but it most assurably isn't bulletproof lol

@eri <gestures vaguely at email spam filtering>

We have experience doing that.

@lluad yeah, email spam filters are kind of the endstate of this. you know how you can’t send an email these days unless you spend all your time proving fealty to google and microsoft? that’s what the web will become if we keep doing this

@eri I get why people want to stop the bot traffic (it is INSANELY high), or slow it down.

Feeding poisoned data is just fucking weak shit. Pathetic. People are literally losing the plot.

I am not even a data hoarder but I think that's bullshit.

@eri @tully @illyBytes i don't know if it's poorly coded applications, but i don't know of anyone self hosting a code forge (regardless of software) in the past 2 years who hasn't considered giving up because they kept getting hammered senselessly.

My forge gets 1K requests/s before filtering. Every day. Almost all the time. I have to filter traffic because otherwise the forge is absolutely unusable. It's just a little site where me and a handful of friends put random code up. Yet it receives multiple hundreds of gigabytes of requests a month.

@eri @tully @illyBytes i've theorized that sites like forges or knowledge hoards are extremely appealing to crawlers because they have TONS of unique URLs. If you only serve a static site and don't actively log server request rates it's not obvious you're getting scraped. If it's an app you self host that gets hammered you will see the horrible response time first. I've had to tombstone several sites already just to stop more traffic from coming.

@eri @tully @illyBytes my point is i don't think we should hate the people doing everything they can to keep their little corner of the web up in the face of coordinated abuse from multinational corporations. We should destroy AI companies and any company offering mass web scraping services.

@eri @tully @illyBytes like you're right we risk creating a gated world where only a set of specific browsers are accepted (spoiler: scrapers pretend to be those already, not the obscure ones) and overzealously banning entire IP ranges, or removing access from devices that can't perform a harsh amount of PoW (i've had that happen to me multiple times)

do we just do nothing though?

@SharpLimefox @tully @illyBytes i’m saying maybe you shouldn’t be trying to mimic the entire featureset of github if you’re not a multibillion dollar corporation. you don’t even need a damn website if what you want is git.

https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server

make a single website about the project. put the bare repo up on the webserver and the client can clone just fine.

https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols#_dumb_http

@illyBytes @SharpLimefox @tully i just need to write my blog post about toaster-fridges don’t i

@eri @tully @illyBytes a forge isn't just to show off code it's also where me and friends collaborate on said code

but it's maybe overkill. maybe it'd be better over email.

look i don't know how to explain that having a little forgejo instance running on a small VM so me and friends can make software together isn't "mimicking all of GitHub". i don't think my argument is getting to you, that's fine, let's leave it at that. but i am not going to degrade the experience of software and tools i use just because a big company decided to harass me and the entire web

@SharpLimefox @tully @illyBytes …then just have a private collaboration thing. don’t allow random connections from all over the internet

specifically with forgejo i feel like the problematic endpoints (search + granular commit history afaik?) could probably be made login-only while still allowing any old rando to download a copy of the code and see the readme, maybe even read the issues n stuff

@illyBytes @SharpLimefox @tully see what i mean by shitty software

@eri @SharpLimefox @tully @illyBytes yeah forgejo has a lethal combination of “unoptimized backend” (fucking git) and “an extremely large amount of link targets” which makes these scrapers make a lot of requests in a way that is overwhelming

for git in particular looking at files themselves is also expensive but at least there tends to be fewer of those than number of commits * number of files

@illyBytes @eri @SharpLimefox @tully actually some scrapers don’t just request historical versions of files but entire diffs across random-ass ranges of commits specifically targeted at large repositories such as FFmpeg thus making a single request that uses a whole CPU core to 100% for like half an hour.

the “request a diff” endpoint is actually not even reachable via the UI afaik and yet forgejo allows any unauthenticated user to access it by default.

@eri @SharpLimefox @illyBytes @tully nextcloud should be fine, despite being ungodly slow usually since the only thing it can access is the login page (and maybe a few related pages) and some assets

gotosocial is probably less of a problem than forgejo, seems decently fast to me at least

@charlotte @tully @eri @illyBytes idk why am i not allowed to have slow/mediocre things for me and friends? why does everything have to be streamlined and fast and optimized so that more scrapers can come get my shit? that perspective baffles me. It's like me getting scraped to oblivion is the software's fault for not being able to handle 2k req/s and i'm also at fault if i try and make it usable for my users :c

@eri @SharpLimefox @illyBytes @tully (basically i also don’t think that it’s possible to make git forges fast while continuing to use git on the backend, git is primarily built for the local workspace usage and not for that of any git browsing tool. Like the worst case scenario for reading a file from storage is that you need to read every commit that changed the file)

@SharpLimefox @tully @eri @illyBytes software performance and what WAF you use are like the two main knobs you can turn here, and basically i think that WAF is at most a bandaid that mainly introduces more issues than it really solves (the underlying issue is usually the software having problems)

and a lot of the anti-scraper WAFs also misunderstand both the problem and the solution extremely hard (like iocaine for example which instead of closing the connection just slowly feeds it markov chain generated data)

@charlotte @tully @SharpLimefox @eri @illyBytes one thing i've been looking at is something like https://git.2f30.org/stagit/file/README.html as a post-push hook on the server

"take this git repo that exists locally and pregenerate html files for stuff like "view specific commits" (O(n) and easily generatable once) / "view the log" etc

you wouldn't get "generate a diff from foo..bar" but that's fine. you can either say "just fuckin clone it yourself" (or, the more silly option, have a in-browser JS git client which clones it and then runs ops locally... that wouldn't scale for every op on a large repo though)

and then it's a bundle of static HTML files which is servable at the rate of your bandwidth (and reasonably easy to cache)

@charlotte @tully @eri @illyBytes i guess i'll tombstone my forge then, fuck it