@inthehands if they decide that people doing this hurts their business model they will simply stop respecting things like robots.txt. their gamble is that people rely on Google more than they do on other websites and if they have to kill the rest of the web to monopolize access to information they will.