Exactly! As far as I’m concerned, robots.txt should be enough: I tell your bot to stay the hell away, or not, and your bot obeys. What it scrapes for doesn’t matter, IMO.
We don’t need more standards and rules for assholes to ignore, we need assholes to adhere to the rules.
Exactly! As far as I’m concerned, robots.txt should be enough: I tell your bot to stay the hell away, or not, and your bot obeys. What it scrapes for doesn’t matter, IMO.
We don’t need more standards and rules for assholes to ignore, we need assholes to adhere to the rules.