Anubis

Yeah that’s a classic problem, and it's why good tests are such closely guarded secrets: to keep them from becoming training fodder for the next generation of models. Regarding the "model date" vs "benchmark date" - that's an interesting point... I'll definitely look into it! I don't have any captcha systems in place, but I wonder if it might be worth putting up at least a few nominal roadblocks (such as Anubis [1]) to at least slow down the scrapers. A few weeks ago I actually added some new, more challenging tests to the GenAI Text-to-Image section of the site (the “angelic forge” and “overcrowded flat earth”) just to keep pace with the latest SOTA models. In the next few weeks, I’ll be adding some new benchmarks to the Image Editing section as well~~ [1] - https://anubis.techaro.lol

Anubis has had great results blocking LLM agents https://anubis.techaro.lol/

So first, let me prefix this by saying I generally don't accept cookies from websites I don't explicitly first allow, my reasoning being "why am I granting disk read/write access to [mostly] shady actors to allow them to track me?" (I don't think your blog qualifies as shady … but you're not in my allowlist, either.) So if I visit https://anubis.techaro.lol/ (from the "Anubis" link), I get an infinite anime cat girl refresh loop — which honestly isn't the worst thing ever? But if I go to https://xeiaso.net/blog/2025/anubis/ and click "To test Anubis, click here." … that one loads just fine. Neither xeserv.us nor techaro.lol are in my allowlist. Curious that one seems to pass. IDK. The blog post does have that lovely graph … but I suspect I'll loop around the "no cookie" loop in it, so the infinite cat girls are somewhat expected. I was working on an extension that would store cookies very ephemerally for the more malicious instances of this, but I think its design would work here too. (In-RAM cookie jar, burns them after, say, 30s. Persisted long enough to load the page.)

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

For example: https://anubis.techaro.lol/ Wikipedia has a good explanation: https://en.wikipedia.org/wiki/Anubis_(software)#Design

Perfect to check if Anubis [0] is correctly installed. [0]: https://anubis.techaro.lol/

Is the solution to everything simply more catgirls [1]? Proof-of-work was, after all, about countering email spam and PR spam is but the latest in that long tradition. 1- https://anubis.techaro.lol

If it's purely bot traffic, then Anubis could help You could have seen it on some websites already https://anubis.techaro.lol/

Perfectly: They're not; that's not really possible. Adequately: Proof of work. https://anubis.techaro.lol/

Do you not run Anubis or have strict fail2ban rules? I just straight up ban IPs forever if they lookup files that will never exist on my servers. That plus Anubis with the strictest settings. https://anubis.techaro.lol/

https://anubis.techaro.lol/ or https://github.com/dgl/haphash/ .

> I get constant "Making sure you're not a bot!" anime girls That's Anubis: https://anubis.techaro.lol/ There is an unbranded version available, allowing image customisation under a paid tier: https://anubis.techaro.lol/docs/admin/botstopper The only other one I've seen deployed out there in any wide capacity is Cloudflare's. Makes sense that Codeberg favoured an open and non-centralised solution.

Anubis and/or Bunny are good alternatives/combination depending on your exact needs - https://anubis.techaro.lol/ - https://bunny.net/

Some HNers already mentioned that the internet has not been a safe haven for a long time. All these vulnerability scanners and parsers were pinging my localhost servers even in mid 2k. It has just become worse, and even OSS and usually captcha-free places are installing things like Anubis [1]. All of this reminds me of some of Gibson's short stories I read recently and his description of Cyberspace: small corporate islands of protected networks in a hostile sea of sapient AIs ready to burn your brain. Luckily, LLMs are not there yet, except you can still get your brain burnt from AI slop or polarizing short videos. [1] - https://anubis.techaro.lol/

If you see this, something isn't working with your main site: https://anubis.techaro.lol/

In the past 6 months, I've seen tons of news about the Anubis PoW. Also on HN, like https://news.ycombinator.com/item?id=43427679 So I decided to take a deeper look into it, and it turned out that not only does Anubis have a fragilely designed PoW (Native accel + Concurrent solver can solve it within 100ms), it also tends not to use PoW by default. As a PoC, I've created a solver in Python, which can be used like: - import requests from anubis_solver import solve cookie = solve("https://anubis.techaro.lol/") print(cookie) content = requests.get("https://anubis.techaro.lol/", headers={"Cookie": cookie}, verify=False).text - I don't really like to think this way, but it made me doubt whether it's really possible to make something, especially FOSS ones, that can stop crawlers or similar stuff. Anubis, at least, doesn't seem "right" to me.

PoW might not work for long, but Anubis is very nice: https://anubis.techaro.lol/ That said ... putting part of your soul into machine format so you can put it on on the big shared machine using your personal machine and expecting that only other really truly quintessentially proper personal machines receive it and those soulless other machines don't ... is strange. ... If people want a walled garden (and yeah, sure, I sometimes want one too) then let's do that! Since it must allow authors to set certain conditions, and require users to pay into the maintenance costs (to understand that they are not the product) it should be called OpenFreeBook just to match the current post-truth vibe.

I make this: https://anubis.techaro.lol. I have yet to add the SQL injection or IP list layers, but I can add that to the roadmap.

If you don't want payment, there is: https://anubis.techaro.lol/ Used by https://gcc.gnu.org/bugzilla/ for example. It is less annoying than CAPTCHA/Turnstile/whatever because the proof of work runs automatically.

time to get rid of the freeloaders with Anubis? https://anubis.techaro.lol/

Anubis (https://anubis.techaro.lol) requires Javascript and is required to view some otherwise static websites now because AI scrapers are ruining the internet for small websites.

Nice - I like that most of the ai scraper bot blocking was done using Nginx configuration. Still, once fail2ban was added to the mix (meaning: additional service and configuration), I wonder if considering something like Anubis (https://anubis.techaro.lol/) would have been more automatic. I’ve seen Anubis verification pages pop up more frequently around the web!

> There is another level of parallelism that could be exploited within a web site/app. And yes we would have to make our planner rate limit aware for that. People are already deploying tools like Anubis[1] or go-away[2] to cope with the insane load that bots put on their server infrastructure. This is an arms race. In the end, the users lose. [1]: https://anubis.techaro.lol [2]: https://git.gammaspectra.live/git/go-away

Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/

That’s Anubis. [0] [0]: https://anubis.techaro.lol

Anubis bot protection: https://anubis.techaro.lol

> Work functions don't make sense as a token tax; there's actually the opposite of the antispam asymmetry there. Every bot request to a web page yields tokens to the AI company. Legitimate users, who far outnumber the bots, are actually paying more of a cost. Agreed, residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs. So I'm not really sure why Anubis works—my best guess is that the bots have some sort of time limit for each page, and they haven't bothered to increase it for pages that use Anubis. > with a content protection system built in Javascript that was deliberately expensive to reverse engineer and which could surreptitiously probe the precise browser configuration a request to create a new Youtube account was using. > The next thing Anubis builds should be that, and when they do that, they should chuck the proof of work thing. They did [0], but it doesn't work [1]. Of course, the Anubis implementation is much simpler than YouTube's, but (1) Anubis doesn't have dozens of employees who can test hundreds of browser/OS/version combinations to make sure that it doesn't inadvertently block human users, and (2) it's much trickier to design an open-source program that resists reverse-engineering than a closed-source program, and I wouldn't want to use Anubis if it went closed-source. [0]: https://anubis.techaro.lol/docs/admin/configuration/challenges/preact [1]:…

If the rest of my post is accurate, that's not the actual concern, right? Since I'm not sure if the check itself is meaningful. From what is described in the documentation [1], I think the practical effect of this system is to block users running old mobile browsers or running browsers like Opera Mini in third world countries where data usage is still prohibitively expensive. Again, the off-the-shelf scraping tools [2] will be unaffected by any of this, since they're all built on top of Puppeteer, and additionally are designed to deal with the modern SPA web which is (depressingly) more or less isomorphic to a "proof-of-work". If you are open to jumping on a call in the next week or two I'd love to discuss directly. Without going into a ton of detail, I originally started looking into this because the group I'm working with is exploring potentially funding a free CDN service for open source projects. Then this AI scraper stuff started popping up, and all of a sudden it looked like if these reports were true it might make such a project no longer economically realistic. So we started trying to collect data and concretely nail down what we'd be dealing with and what this "post-AI" traffic looks like. As such, I think we're 100% aligned on our goals. I'm just trying to understand what's going on here since none of the second-order effects you'd expect from this sort of phenomenon seem to be present, and none of the places where we actually have direct data seem to show this…

> yes, it's a cartoon character; get over it I think it's cute. It wouldn't be a problem if it wasn't cute. It's inoffensive and doesn't get in the way of the software serving its function, the only critique I could do if I wanted to is people unfamiliar with the project would get a bit confused if they only saw the character for a split second: https://anubis.techaro.lol/ I think even Cloudflare gives you a second or two to see what's going on before forwarding you to the page behind it, that would be a nice UX improvement. Otherwise, agreed about self-hosting and the ability to choose whatever platform aligns with your goals the closest to be a good thing! > And now everyone has to learn GitHub Actions, Gitlab pipelines, Jenkins pipelines, and more. A part of me doesn't like the churn of new CI solutions coming in all the time. On the other hand, after learning GitLab CI, I'd generally prefer it to something like Jenkins, it just feels more pleasant. But with something like Drone CI or Woodpecker CI my non-work needs are also covered wonderfully, especially with a lot of the software I build being packaged in Docker containers, which further simplifies and somewhat standardizes things! A lot of the time I can also encapsulate most of the logic in shell scripts that are easy to run regardless of the environment (CI server or locally), which makes it even more portable.

So, Anubis? https://anubis.techaro.lol/