World Wide Web Wanderer

The World Wide Web Wanderer, also simply called The Wanderer, was a Perl-based web crawler that was first deployed in June 1993 to measure the size of the World Wide Web. The Wanderer was developed at the Massachusetts Institute of Technology by Matthew Gray, who also created back in 1993 one of the 100 first web servers in history, www.mit.edu.[1] As of 2022, he has spent more than 15 years as a software engineer at Google. The crawler was used to generate an index called the Wandex later in 1993. While the Wanderer was probably the first web robot, and, with its index, clearly had the potential to become a general-purpose WWW search engine, the author does not make this claim[2] and elsewhere[3] it is stated that this was not its purpose. The Wanderer charted the growth of the web until late 1995.