Post by @francois
@francois @stefano @fasnix@fe.disroot.org @GossiTheDog@cyberplace.social
I have a little script running on FreeBSD that retrieves a list of parasites from https://scienceispoetry.net/files/parasites.txt
retrieve it from there, compare last-modified header. if not changed, then stop. otherwise replace a pf table with new contents
saves a lot of scraping
I added a list of User-Agents to my HAproxy to block them
AhrefsBot
Amazonbot
anthropic-ai
Applebot
Bytespider
CCBot
censys
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
dataforseo
Diffbot
FacebookBot
facebookexternalua
Google-Extended
GPTBot
ImagesiftBot
Meltwater
mj12
OAI-SearchBot
Omgili
Omgilibot
PerplexityBot
Seekr
webmeup
YouBot
zoominfo
This works for me
@fd0 @stefano @fasnix@fe.disroot.org @GossiTheDog@cyberplace.social
Thanks for the tip. Will look at the lists in place and diff it with yours.