Hacker News new | past | comments | ask | show | jobs | submit login

That’s a weird statement to be absolutist about. The majority of individuals and companies who want to be successful do not do so by scrapping websites, thus have no reason to disobey robots.txt. Most people in the world, ambitious or not, wouldn’t even understand what your sentence refers to.



OP is obviously talking about people whose area of research/development/product would involve web scraping... This feels like being purposefully obtuse




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: