Crowdsourced psychological science

Crowdsourcing is a collaborative sourcing model in which a large and diverse number of people or organizations can contribute to a common goal or project.

First examples of crowdsourcing science can be found during the 19th century. A Yale University professor launched a call for open participation of citizens to maximize the number and diversity of observations he could get on the Leonids of 1833 meteor storm phenomenon.[1]

Crowdsourced science has been set aside for a long time and has only recently gained popularity in science. Actually, it helps overcoming several limits of the classic model on which science has been conducted up till now. For example, in psychology, scientific research has often been limited by small sample sizes and a lack of diversity in studied populations.[2] These limits can be tackled with a more collaborative approach of scientific research (i.e., crowdsourced science).[3]

Crowdsourcing initiatives can be described along two different continuous axes.[4] The first dimension represents the degree of communication between project members, ranging from very few collaborators and little communication, to a large crowd of collaborators with a rich amount of communication. The second dimension corresponds to the degree of inclusiveness, varying from only selecting groups of people with a high expertise in the field of interest, to open to anyone—with or without expertise—interested in collaborating. This twofold distinction draws four types of crowdsourced science projects. However, these two dimensions lie on a continuum: A single project is neither entirely inclusive (vs. selective) nor entirely high (vs. low) in communication.[4]

In psychology, the traditional way of doing research follows a vertical model.[4] In other words, most of the time, research is carried out by small teams from a specific lab or university, and organized around one or two main researchers (often referred to as Principal Investigator, or PI). This small team contributes to all different stages of the research project: conception of the research question, design of the study, data collection, data analysis, discussion of the results, and publication of the manuscript.[4]

However, vertical science has its own limitations that can impede the progress of science. Conducting science within small independent teams makes it difficult to conduct large-scale projects (with large samples, large databases and high statistical power), which may limit the scope of research. In traditional science, researchers have access to less resources, data, and methodologies.[4] Small, independent team projects are also limited in the feedback they can get from other perspectives.[4]

Nowadays, research teams do not necessarily have to make compromises anymore (e.g., small sample size, same stimuli, same contexts and no replication),[4] because much of these limits could be tackled by a more crowdsourced research.

In the scientific method, replicability is a fundamental criterion to qualify a theory as scientific. The replication crisis (or credibility crisis) is a methodological crisis in science that researchers began to acknowledge around the 2010s. The controversy revolves around the lack of reproducibility of many scientific findings, including those in psychology (e.g., among 100 studies, less than 50% of the findings were replicated).[5][6]

Some of its main underlying causes are referred to as “questionable research practices”.[7] These include p-hacking (i.e., exploiting researcher degrees of freedom until a significant result is obtained), HARKing (i.e., hypothesizing after results are known),[8] publication bias (i.e., the tendency among scholarly journals to only publish significant results),[9] statistical reporting errors,[10][11] and low statistical power often due to small sample sizes.[12][6]

Among these various issues, small sample sizes and the lack of diversity within samples can be addressed through crowdsourced science—increasing the generalizability of findings and therefore their replicability as well. Indeed, samples in psychology often rely on college students[13] and Western, Educated, Industrialized, Rich and Democratic (WEIRD) populations.[14][15][16]

For these reasons, the replication crisis has contributed to the rise of crowdsourced, large-scale projects, especially replication projects held at an international scale like the Many Labs project,[17][18][19][20][21] and the Psychological Science Accelerator (PSA).[3] These crowdsourced projects aim at solving some issues raised by the replication crisis, more specifically by assessing the replicability of studies and generalisation of the results to other populations and contexts.

In contrast to the vertical model of doing research, the horizontal model—an inherent principle of crowdsourced science—mainly relies on variations in terms of inclusiveness and communication. Its core principle is about the non-authority of one or two researchers in terms of resources, ownership and expertise.[22] Following this principle, the different tasks within a project are distributed among many researchers. The whole project is then supported by a team, which ensures the coordination of the contributors.[4]

A perfect horizontal model does not really exist because vertical and horizontal models are more conceptualized as extremes of a same continuum.[4] This distributed model of science work is gaining popularity in the scientific community. In about 40 years and across different scientific disciplines, research teams have roughly doubled in size (from 2 to 4 people on average).[23]

By encouraging larger crowds to contribute to research projects in psychology, the horizontal model aims at reducing some limits of the vertical one. It has three distinct ambitions:[4]

• Carry out wide-ranging works
• Encourage a democratized psychological science
• Establish robust findings

The first ambition of the horizontal model is to enable researchers to conduct more large-scale research projects (i.e., ambitious projects that cannot be conducted by small teams).[4] By aggregating various skills and resources, it is possible to move from a model where research is defined by available means, to a model where research itself defines the necessary means to answer the research question.[4]

The second ambition of the horizontal model is to compensate for inequalities (e.g., in terms of recognition, status, and success in a researcher's career). Psychology and more generally social sciences show a strong bias of what is called the Matthew effect[24] (i.e., academic advantages going to those who are already the most renowned).

Early career researchers from less renowned institutions, less economically developed countries or underrepresented demographic groups are generally less likely to have access to high-profile projects.[25] Crowdsourcing enables such researchers to contribute to impactful projects and gain recognition for their work.[4]

The third ambition of the horizontal model is to improve the robustness, generalizability, and reliability of findings in order to increase the credibility of psychological science.[4][26] Horizontal collaboration between research teams facilitates the replication of studies[27][17][28] and makes it easier to detect biased effect sizes, p-hacking, and publication bias—different problems raised by the replication crisis[29] (see also #The replication crisis).

This section aims at detailing how crowdsourcing can contribute to the different stages of the research process, from the generation of research ideas to the publication of the outcomes.

Ideation is the first step of any research project. In psychology, it refers to the process of defining the general idea behind a project—purpose, research question, and hypotheses. This step can be done in collaboration between several researchers to scan a broader spectrum of ideas and select those of broadest interest and impact.[4] Faced with a research question, the different collaborators can bring their expertise in the construction of hypotheses.

Crowds can also help to generate new ideas to solve complex-problems, such as illustrated by the Polymath project.[30][31][32][33]

When assembling resources, crowdsourcing can be useful, especially for labs with less resources at their disposal.[4] Online platforms such as Science Exchange and Studyswap allow researchers to establish new communication lines and share resources between labs. Matching resources from labs across the globe minimizes waste and facilitates research teams’ ability to meet their goals.[4]

Sharing tasks across labs can improve the efficiency of a research project, especially highly time-consuming ones. In biology, for example, studying the entire genome takes a lot of time. By distributing its investigation and combining resources across multiple labs, it is possible to accelerate the research process.[34]

There can be many ways to design a study. Research teams across the world neither have the same theoretical background, nor are they all equipped with the same materials.[4]

Crowdsourcing can be useful in the case of conceptual replications (i.e., testing a same research question through different operationalizations).[35][36] When testing a same research question, variations in study designs can lead to strong variations in effect size estimations.[28] Diversifying the methods used to test the same hypothesis across different populations—through collaborative projects—enables better estimations of the true consistency of a scientific claim.[4]

In psychology, data collection often relies on samples drawn from Western, Educated, Industrialized, Rich and Democratic (WEIRD) populations, which impedes the overall generalizability of findings.[15][16] Crowdsourcing the data collection process by relying on multi-lab data collection and online crowdsourcing platforms (e.g., Amazon Mechanical Turk, Prolific[37]) makes it easier to reach a wider audience of participants from different cultural backgrounds and non-WEIRD populations.[4] When the research question makes it possible to rely on internet samples, it is also an easy way to recruit larger samples of participants with minimal financial input and within short amounts of time.[38][39][40]

Most of the time, members of the general public are recruited to undergo studies as research participants but they can also be recruited to collect data and observations.[4]

In research, data analysis refers to the process of cleaning, transforming, and modeling data using statistical tools, often with the purpose of answering a research question. Within a research project, this is typically done by a single analyst (or team) and results in a single analysis of a dataset.

Analytic strategies can differ greatly from one team to another.[4][41] For example, a study found that among 241 published articles on fMRI, 223 different analytic strategies were used.[42][43] Moreover, there can be many ways to test a single hypothesis from the same dataset. Although defensible, decisions in data analysis remain subjective, which can greatly affect research results.[41] A way to counter this subjectivity is transparency. When data, analytic plans, and analytic decisions are made transparent and open access to the rest of the community, it facilitates criticism and gives the opportunity to explore alternative ways to analyse data.[41]

“Crowdsourcing the analysis of the data reveals the extent to which research conclusions are contingent on the defensible yet subjective decision made by different analysts.”—Uhlmann et al., 2019[4]

A research report is a document reporting the findings of a research project. The overall quality of a research report can benefit from crowdsourcing practices, especially during the writing process.[44] Aggregating a large number of contributors increases the range of expertise and perspectives,[44] which contributes to build more solid arguments.[4] It also makes proofreading easier (e.g., catching grammatical errors, weird phrasing, typos, biases, factual errors, claim-checking).[4][44]

To facilitate collaborative writing, some researchers have suggested guidelines for writing manuscripts with many authors.[44] There should always be a leading author (or a few, but with individual responsibilities) who takes care of managing the writing process and takes explicit responsibility for any mistake—avoiding diffusion of responsibility in case of errors.[44] It is also recommended that the leading author(s) follow the four general principles mentioned below:[44]

• Care in crediting the coauthors team
• Clear and frequent mass communication
• Make sure materials associated to the manuscript are well-organized
• Early and deliberate decision-making

Before getting published in an academic journal, submitted papers undergo peer review (i.e., the process of having an author's academic work reviewed by experts from the same field). Typically, it is performed by a limited number of selected reviewers. Crowdsourcing the peer review process increases chances of getting reviews from a larger number of experts in the relevant domain.[4] This is also a way to significantly increase opportunities for better criticism and faster fact-checking before an article gets published.[45]

Crowdsourced peer-review can be accomplished alongside open access peer-review[46] (e.g., through centralized platforms dedicated to the discussion and criticism of research reports).[47]

The replication crisis served as a prelude for the emergence of many large-scale collaborative projects. Some of the most important ones include the ManyLabs project,[17][18][19][20][21] the ManyBabies project,[48][49] the #EEGManyLabs project,[50] the Reproducibility Project,[6] the Collaborative Replication and Education Project (CREP),[51][52] and the Psychological Science Accelerator (PSA).[3]

In response to the COVID-19 pandemic, several initiatives to improve collaboration between scientists from a variety of fields have been launched.[53][54]

Studies exploring the impact of the COVID-19 pandemic on behavior and mental health are conducted[55] and widely shared across social media such as Twitter.[56][57][58][59][60][61]

In this context, the PSA made a call for studies surrounding the COVID-19 disease[62] and retrieved 66 study proposals from different experts within one week.[63] At the date of May 2020, three studies have been selected and are being conducted worldwide.[63] Two of these studies aim at reaching a better adoption of health behaviors to avoid the spread of COVID-19, and one aims at helping people regulate their negative emotions during the crisis.[63] This project aims to get results that could help all countries to face the pandemic with means adapted to their population.

Although a horizontal model of conducting science seems promising to overcome some limits of the vertical model (see also #Limits of the traditional vertical model and #Ambitions of the horizontal model), it is difficult to empirically assess its benefits.[4] It remains unclear whether two sets of research that study the same question—either through a vertical or horizontal way of doing science—would lead to different outcomes or not.[4]

Currently, to sustain a collaborative project, researchers either have to use money from their own grants or apply for fundings.[3] Such projects are thus limited in the studies they can conduct without financial independence.[3]

Coordinating hundreds of labs to conduct a study requires consequent administrative work.[3][4] Structures like the Psychological Science Accelerator (PSA) have to ensure each participating lab retrieves ethics approval to conduct a given study (see also Institutional review board). Within the PSA, this background work is currently completed voluntarily by dedicated teams or researchers[64] beside their main occupation.[3] By reaching a financial independence (as it is the case for the CERN), these projects could optimize their functioning through the opening of jobs dedicated to these missions.

A researcher's career path in the academic world (i.e., job opportunities, grant attributions, etc.) depends on major contributions to research projects, which is often assessed through the amount of publications where one appears as the lead author.[65] In psychology, it is common practice to list authors in order of contribution, with involvement decreasing as we move down the list.[65] Truth is, it is not rare that multiple authors on the same paper contributed equally to the project, but in different ways.[66] In that sense, contributors on a project do not always get the credit they deserve—as the order the authors are listed in does not capture well the contribution of each author. This is especially true for large-scale collaborative projects with many contributors (e.g., the PSA001 project has over 200 contributors).[67]

An alternative to the current authorship system is the CRediT taxonomy, a taxonomy describing 14 distinctive categories (e.g., conceptualization of the project, administration of the project, funding acquisition, investigation) that represent the roles typically played by contributors in a scientific project.[65][68] Papers relying on this taxonomy allow for a more representative description of the involvement of each contributor on a project.

The enrolment of students through different collaborative projects could foster open science and crowdsourced practices early in a researcher's career.[4] For instance, in the Collaborative Replication and Education Project (CREP),[51] students are taught the roots and importance of such practices toward the replication of recent major findings in psychology.

Editorial policies of scientific journals also play a role in the adoption of open science and crowdsourced practices, especially by defining new publication criteria.[4] For instance, more than 200 journals now adopt an “in principle acceptance” format of peer-reviewing papers.[69] In this publishing format, articles are accepted for publication prior to data collection, on the basis of the provided theoretical framework, methodology, and analysis plan.[69]

Flexibility in data analyses and other biases that collaborative projects should address by aggregating experts are not always overcome.[70]

It has also been shown that crowdsourced projects involving low-trained and low-involved actors (e.g., students) can lead to the falsification of data.[71] Linking up a wide array of contributors can thus imply structural problems that may impact research outcomes.

Both issues highlight the importance of educational practices on open science and crowdsourced practices (see also #Developing open-science and crowdsourced practices).