Knowledge Engine (search engine)

In 2015, WMF applied for a $250,000 grant from the Knight Foundation to support development of the Knowledge Engine. Its grant proposal noted: "Commercial search engines dominate search-engine use of the Internet, and they're employing proprietary technologies to consolidate channels of access to the Internet's knowledge and information."[5] The project was designed in four stages, each scheduled to take about 18 months.[6]

The project planned to draw information from Wikipedia-related projects and eventually to search other sources of public information such as the U.S. Census Bureau.[5] Leaked internal WMF documents stated the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation."[2] The new search engine was not expected to immediately replace a general purpose search engine because at first it would only draw on information from Wikipedia and its other free knowledge projects,[5] though it might in time also have included academic and open access sources in its search results.[7] Matt Southern in Search Engine Journal attributed media confusion about the Knowledge Engine's scope to the fact that later WMF statements clarifying the organization's intentions were "quite a contrast to the original grant application documents".[8]

The project was not discussed publicly with the Wikipedia community while developing the concept,[9] nor part of the existing annual plan.[10] This secrecy was mirrored by a degree of confusion within the organization, and seen as at odds with the goal of transparency.[2] An initial blogpost by WMF Executive Director Lila Tretikov about the project did not address why the original proposal was so much broader than an internal search engine.[11] Some staff and WMF board members felt the WMF was still not being straightforward with the Wikipedia community.[12] This led to a crisis for the organization,[1] leading to Tretikov's resignation in February 2016.[11][13]

The goal of the Knowledge Engine was to let readers and editors be less reliant on proprietary search engines when looking for new information.[3] The project proposal asked, "Would users go to Wikipedia if it were an open channel beyond an encyclopedia?"[1]

Example of federated data sources potentially used by the Knowledge Engine.

The Knowledge Engine was designed to be open and transparent about how a piece of information originates and allow access to metadata.[4] It would have no advertisements, protect user privacy, and emphasize community building and sharing of information.[4] It would draw information from Wikipedia-related projects and eventually perhaps search other sources of public information such as the U.S. Census Bureau,[5] OpenStreetMap,[14] the Digital Public Library of America,[15] and external sources like Fox News.[1] Jimmy Wales and the WMF stated that the project would focus on improving search on Wikipedia and related Wikimedia projects.[2] The grant application stated that it would "create a model for surfacing high quality, public information on the internet."[2] It also advised that "commercial search engines dominate search-engine use of the internet" and stated that "Google, Yahoo, or another big commercial search engine could suddenly devote resources to a similar project, which could reduce the success of the project."[2]

Since mid-2012,[17] Google Search has included fast facts from Wikipedia articles on its search results pages[2] via the Google Knowledge Graph.[17]

A central source of confusion for the project was the extent to which it would directly compete with traditional search engines as a place to search the Web. According to Vice, "the Wikimedia Foundation, the nonprofit that finances and founded Wikipedia, is interested in creating a search engine that appears squarely aimed at competing with Google."[2] According to The Guardian, "there was considerable doubt over what the tool was actually intended to be: a search engine aimed at halting a decline in Wikipedia traffic sent by Google, or simply a service for searching within Wikipedia?"[11]

Since 2012, Google Search and other search engines had started highlighting brief informational summaries from Wikipedia in knowledge panels alongside search results, reducing traffic to Wikipedia from those search engines.[2] According to Search Engine Watch, this led to a battle for attention,[1] and this project could have recouped some of that traffic.

Leaked internal documents from early concepts framed the plan more boldly than the final public description.[18] They said the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation."[2]

The apparent contradiction between different descriptions of the purpose led to confusion in the media and in the community. In response to speculation, the WMF published a response clarifying its intentions: "We're not building a global crawler search engine ... Despite headlines, we are not trying to compete with other platforms, including Google. As a non-profit we are noncommercial and support open knowledge. Our focus is on the knowledge contributed on the Wikimedia projects. ... We intend to research how Wikimedia users seek, find, and engage with content. This essential information will allow us to make critical improvements to discovery on the Wikimedia projects."[8] Director of Discovery Tomasz Finc added "we are building an internal search engine, and we are not building a broad one.[1] Jimmy Wales stated that suggestions that the WMF is creating a rival to Google are "trolling", "completely and utterly false", and "a total lie",[2][19] while allowing that the Knowledge Engine might in time include academic and open access sources in its search results.[7]

Matt Southern in Search Engine Journal attributed media confusion about the KE's scope to the fact that this was "quite a contrast to the original grant application documents",[8] an assessment echoed by James Vincent in The Verge,[9] Matt McGee in Search Engine Land,[20] and Jason Koebler in Vice.[21]

Large-scale WMF projects are almost always discussed publicly with the Wikipedia community, but this did not happen with the Knowledge Engine development.[9] Wikipedians were unaware of the existence of the project as a concept,[2][22] and the KE project was not mentioned in the WMF's annual plan.[10] According to the English Wikipedia's community newsletter, The Signpost,[23] some community members expressed outrage at the perceived secrecy around it and their lack of ability to give input, and this raised questions about WMF's commitment to transparency with the Wikipedia community.[9]

James Heilman, a member of the WMF's Board of Trustees, noted in The Signpost that while on the Board, he had insisted multiple times that the grant documentation be made public, without success.[19] He was dismissed from the Board in December 2015, and it was suggested that his push for transparency concerning the grant had been a factor in his dismissal—a suggestion rejected by Jimmy Wales.[2] The Wikipedia community re-elected Heilman to the Board in 2017.[24]

Ruth McCambridge said in Nonprofit Quarterly, "Wikipedia editors have been requesting from December for the grant proposal and grant letter for a project that many surmise is a bid to remain technologically cutting-edge by the Wikimedia Foundation, but which may divert resources and attention from other pressing needs of the community."[23]

Commenting on the reluctance to share the grant documents with the community, referencing privacy concerns, McCambridge saw "a major difference in culture and values assumptions" compared to previous Wikimedia practice.[23] McCambridge said that "the power of important strategic decisions" here seemed to rest "between funders and the top of the organizational hierarchy" and was "not shared with volunteer editors."[23]

The WMF initially published only portions of the grant documentation,[25] later making the full grant agreement available in February.[19] Further internal documents were leaked shortly after.[2][9] The full agreement clarified the initial concept for the first stage of the project.[23] Tretikov said she regretted being so late in informing the Wikipedia editing community about the grant.[15]

Longtime Wikipedia editor and journalist William Beutler told Vice Magazine's Jason Koebler, "Leaving aside whether a search engine is a good idea, let alone feasible, the core issue here is about transparency. The irony is that the Wikimedia Foundation failed to observe one of the movement's own core values ...."[21] UK Wikipedia editor Ashley van Haeften told Ars Technica via e-mail that "Lila, Jimmy, and the rest chose to keep the project and the Knight Foundation application and grant a secret until the projects were underway for six months, and even then this only came to light because it was leaked."[18]

Tretikov's initial public post about the Knowledge Engine project did not explain why the original grant proposal had such a grander vision than the later public plan to develop an internal search engine.[11] Staff who had been uncomfortable about the project's development felt the WMF was not being sufficiently straightforward with the community.[12] According to statements posted of an internal meeting on the WMF's website,[12] a member of the Discovery team member said to Tretikov, "My concern is that we still aren't communicating it clearly enough. This morning's blog post is the truth, but not all of the truth. Namely that we had big plans in the past. It would have been much easier to say that we did have big plans, but they were ditched ... we still haven't acknowledged it. We can't deny it."

Former deputy director of the WMF Erik Möller, up to April 2015, portrayed the events as "very much out of control" and "a crisis."[1] Disagreements about the project, and the response to the resulting controversy, led to many WMF staff members departing,[26][27] culminating in Tretikov resigning on February 25, 2016.[11][13]

• Discovery homepage on MediaWiki.org