Post by @fd0
Running a Mastodon server is becoming increasingly problematic due to GenAI.
There’s multiple different things happening, including but not limited to:
- very aggressive scraping of user content and block evasion, with the resource costs that come with it
- AI agents aggressively trying to register accounts and evade restrictions
- rising cost of RAM - Mastodon is memory hungry, RAM is really expensive now
It’s odd to experience first hand, it’s like being consumed by.. a void.
@fasnix@fe.disroot.org @GossiTheDog@cyberplace.social yes, this is something I can confirm. I had to close the registrations at BSD Cafe (first time in three years) because of this.
@stefano @fasnix@fe.disroot.org @GossiTheDog@cyberplace.social
Same thing for polyglot.city. we had to revert to manually approving requests 😳
@francois @stefano @fasnix@fe.disroot.org @GossiTheDog@cyberplace.social
I have a little script running on FreeBSD that retrieves a list of parasites from https://scienceispoetry.net/files/parasites.txt
retrieve it from there, compare last-modified header. if not changed, then stop. otherwise replace a pf table with new contents
saves a lot of scraping
I added a list of User-Agents to my HAproxy to block them
AhrefsBot
Amazonbot
anthropic-ai
Applebot
Bytespider
CCBot
censys
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
dataforseo
Diffbot
FacebookBot
facebookexternalua
Google-Extended
GPTBot
ImagesiftBot
Meltwater
mj12
OAI-SearchBot
Omgili
Omgilibot
PerplexityBot
Seekr
webmeup
YouBot
zoominfo
This works for me
@fd0 @stefano @fasnix@fe.disroot.org @GossiTheDog@cyberplace.social
Thanks for the tip. Will look at the lists in place and diff it with yours.