Post by @fd0

Replying to a post that is no longer available

@francois @stefano @fasnix@fe.disroot.org @GossiTheDog@cyberplace.social

I have a little script running on FreeBSD that retrieves a list of parasites from https://scienceispoetry.net/files/parasites.txt
retrieve it from there, compare last-modified header. if not changed, then stop. otherwise replace a pf table with new contents

saves a lot of scraping

I added a list of User-Agents to my HAproxy to block them

 AhrefsBot
 Amazonbot
 anthropic-ai
 Applebot
 Bytespider
 CCBot
 censys
 ChatGPT-User
 Claude-Web
 ClaudeBot
 cohere-ai
 dataforseo
 Diffbot
 FacebookBot
 facebookexternalua
 Google-Extended
 GPTBot
 ImagesiftBot
 Meltwater
 mj12
 OAI-SearchBot
 Omgili
 Omgilibot
 PerplexityBot
 Seekr
 webmeup
 YouBot
 zoominfo

This works for me

Replying to @fd0
francois
@francois

@fd0 @stefano @fasnix@fe.disroot.org @GossiTheDog@cyberplace.social

Thanks for the tip. Will look at the lists in place and diff it with yours.