MetaCrawler

MetaCrawler was originally developed in 1994 at the University of Washington by graduate student Erik Selberg and Professor Oren Etzioni as Erik Selberg's Ph.D. qualifying project.[1] Originally, it was created in order to provide a reliable abstraction layer to web search engine programs in order to study semantic structure on the World Wide Web. However, it was a useful service in its own right, and had a number of research challenges. MetaCrawler was not, however, the first metasearch engine on the World Wide Web. That feat belongs to SavvySearch, developed at the Colorado State University, albeit launched just four months prior to MetaCrawler.[2]

MetaCrawler was originally operating on four Digital Equipment Corporation AlphaStations[3] and processing several hundred thousand queries per day. This was starting to create significant bandwidth load at UW. It became clear that MetaCrawler needed to have some method of paying for the queries it was forwarding to the primary search engines. Some time after the search engine launched, NetBot, Inc., which was cofounded by Etzioni,[4] was initiated to commercialize MetaCrawler[5] and three other UW programs: Ahoy! The HomePage Finder, Occam, and ShopBot. Ahoy! and Occam were never actually commercialized. NetBot then combined the core of MetaCrawler with ShopBot to create a meta-shopping website, Jango.[6]

MetaCrawler launched on July 7, 1995.[7]

MetaCrawler site in 1996

As of late 1995, MetaCrawler logged over 7,000 search queries per week, and accessed six services: Galaxy, InfoSeek, Lycos, Open Text, WebCrawler and Yahoo.[8] By late 1996, there were over 150,000 queries per day.[9]

MetaCrawler's owners were unable to determine a reasonable business model, so in January 1997 they sold it to another Internet startup company, Go2Net,[10] in which Microsoft co-founder Paul Allen later invested a 54 percent stake.[11] Go2Net went public in April that year, registering on Nasdaq.[12] MetaCrawler had about 30,000 daily visitors at the start of 1997, but by mid 1998 jumped to 275,000.[13]

Old MetaCrawler logo, used c. 1997 to 2003

NetBot would eventually be purchased by Excite in October 1997 for $35 million, where Jango became part of the Excite Network Shopping Channel.[14] Both Selberg and Etzioni resumed working for UW until 1999, when they joined Go2Net for a year, quitting just prior to Go2Net's acquisition by InfoSpace, Inc. in July 2000 for $4.2 billion.[15] By that time, Go2Net had purchased another metasearch engine, Dogpile.[16]

In 2014, MetaCrawler was merged into another one of InfoSpace's search engines, Zoo.com,[17] which was originally launched in 2006.[18] The MetaCrawler domain at first redirected to Zoo.com,[19][20] but was afterwards changed to redirect to msxml.excite.com, the search page for Excite, also operated by InfoSpace.[21][22]

In July 2016, InfoSpace was sold by parent company Blucora to OpenMail for $45 million, putting MetaCrawler under the ownership of OpenMail.[23] OpenMail was later renamed System1.[24]

In 2017, MetaCrawler relaunched as its own search engine.[25]

• List of search engines
• WebCrawler
• InfoSpace
• Dogpile

• Official website