What's interesting about web crawling is that it has always been controversial.
Many website operators back then did not like crawlers. And that sort of sentiment continues to this day. The web has never fully consented to being indexed.
This has always struck me as odd. Perhaps there should have been more thought about a bifurcation of "private" web and "public" web. Of course in retrospect the value of the public one seems far greater; Google's corporate wealth points to that conclusion.
It was probably a very neat hack (I didn't use it; my first search engine of choice was Webcrawler in '95. Then Altavista and Alltheweb (Norway, NTNU! First known as ftpsearch.ntnu.no.). Then omfg, this Google thing really works, '98 - "why won't they let me buy shares in their company"?)
But I think Jonathon would agree that it was the obvious next step. There was already the (overly) academical WAIS etc.
What's interesting about web crawling is that it has always been controversial.
Many website operators back then did not like crawlers. And that sort of sentiment continues to this day. The web has never fully consented to being indexed.
This has always struck me as odd. Perhaps there should have been more thought about a bifurcation of "private" web and "public" web. Of course in retrospect the value of the public one seems far greater; Google's corporate wealth points to that conclusion.