Had to spend another 30 minutes to block whole subnets of scraper bots that claimed to be googlebot or such that overloaded my server. This is all such a pain in the ass.