clearly, the only solution is to ensure that everyone connecting to your server is actually a person. i propose that every ID card should contain a smart chip which contains a cryptographic key. by verifying the connection has a valid user, you can ensure that there are no more automated scrapers! simply rate limit each individual person's access. This system will not be used by any other people for any other things, and will be a net good for society and the freedom of information

those who judiciously block random traffic are just as evil to me as big tech companies running poorly behaved crawlers. i hate both of you because you both want to stand over your little fiefdoms of the information you hoard privately as if the capacity of your hard drives is a proxy for the size of your metaphorical penis, and your kink is making people beg to get just a few more nybbles of that sweet sweet data from you.