Data Colada
Data Colada is a blog dedicated to investigative analysis and replication of academic research, focusing in particular on the validity of findings in the social sciences.[1]
It is known for its advocacy against problematic research practices such as p-hacking, and for publishing evidence of data manipulation and research misconduct in several prominent cases, including celebrity professors Dan Ariely and Francesca Gino. Data Colada was established in 2013 by three behavioral science researchers: Uri Simonsohn, a professor at ESADE Business School, Barcelona/Spain (as of 2023), Leif Nelson, a professor at University of California, Berkeley, and Joe Simmons, a professor at University of Pennsylvania.[1]
History
Around 2011, Simmons, Nelson and Simonsohn "bonded over the false, ridiculous, and flashy findings that the field [of behavioral sciences] was capable of producing", such as a paper by Cornell psychologist Daryl Bem that had supposedly found evidence for clairvoyance.[2] They reacted by publishing an influential[1] 2011 paper about false positive results in psychology, illustrating the problem with a parody research finding that supposedly showed that listening to the Beatles song "When I’m Sixty-Four" made experiments subjects one and a half years younger.[2]
The "Data Colada" blog was launched two years later, in 2013, carrying the tagline "Thinking about evidence, and vice versa", and "became a hub for nerdy discussions of statistical methods — and, before long, various research crimes and misdemeanors" (New York Times).[1]
In particular, the three researchers objected to what the then widespread practice of cherry-picking data and attempts to make insignificant results appear statistically credible, especially an approach for which they coined the term p-hacking in a 2014 paper.[2][3][4]
Notable findings
Apart from calling out faulty but presumably well-intended research practices, Data Colada has also published evidence of data manipulations and research misconduct, such as in the studies about the concept of moral high ground by psychologist Lawrence Sanna, and in research by Flemish psychologist Dirk Smeesters.[2] Following the findings of Data Colada, "[t]he two men’s careers came to an unceremonious end" (according to The New Yorker).[2]
An August 2021 post by Data Colada found that data from a field study in a 2012 PNAS paper[5] by Lisa L. Shu, Nina Mazar, Francesca Gino, Dan Ariely, and Max H. Bazerman had been fabricated.[6][7] All of the 2012 study's authors agreed with this assessment and the paper was retracted.[7] The study's authors also agreed that Dan Ariely was the only author to have had access to the data prior to transmitting it in its fraudulent form to Nina Mazar, the analyst.[6] Dan Ariely denied manipulating the data prior to forwarding it on to Mazar[8] but Excel metadata showed that he created the spreadsheet and was the last to edit it. He also admitted to having mislabeled all of the values in an entire column of the data in e-mail communication with Mazar that took place shortly after he initially sent her the data.[6][9] Dan Ariely has stated that someone at the insurance agency that provided the data must have fabricated it.[10][11][2]
Reception
The work of Data Colada has been credited with contributing to the start of the replication crisis in social sciences,[2] and with helping to establish better research practices such as the sharing of replication data.[3]
The Nobel-prize winning psychologist Daniel Kahneman described Data Colada in 2023 as "heroes of mine" and expressed his regret about having previously endorsed some research findings that the blog later showed to be faulty.[2] Brian Nosek of Center for Open Science applauded Data Colada for "hav[ing] done an amazing job of developing new methodologies to interrogate the credibility of research."[3]
On the other hand, as summarized by The New Yorker, "Data Colada’s harshest critics saw the young men as jealous upstarts who didn’t understand the soft artistry of the social sciences", such as psychologist Norbert Schwarz, who accused them and other reformers of engaging in a "witch hunt," or psychologist Daniel Gilbert who denounced the "replication police" as "shameless little bullies".[2]
Francesca Gino lawsuit
In June 2023, Harvard Business School placed Francesca Gino on unpaid administrative leave after an internal investigation determined she had falsified data in her research.[12][13][14] Gino subsequently filed a defamation suit against Harvard, Harvard Business School Dean Srikant Datar, and the three members of Data Colada for $25 million, alleging that they had conspired to damage her reputation with false accusations, and that the penalties against her amounted to gender-based discrimination under Title IX.[14] The suit does not contest or refute the defendants' scientific findings. It asserts that since the researchers used inference and probability to argue that the anomalies between the original dataset and the data used in Gino's analyses could not have been due to random chance or benign error, and therefore occurred from fraudulent manipulation, the defendants cannot prove she in particular committed the scientific misconduct.[15]
References
- Scheiber, Noam (2023-09-30). "The Harvard Professor and the Bloggers". The New York Times. Retrieved 2023-10-01.
- Lewis-Kraus, Gideon (2023-09-30). "They Studied Dishonesty. Was Their Work a Lie?". The New Yorker. ISSN 0028-792X. Retrieved 2023-10-01.
- Subbaraman, Nidhi (2023-09-24). "The Band of Debunkers Busting Bad Scientists". Wall Street Journal. Archived from the original on 2023-09-24. Retrieved 2023-10-08.
- "APA PsycNet". psycnet.apa.org. Retrieved 2023-10-08.
- "A study on dishonesty was based on fraudulent data". The Economist. August 20, 2021. ISSN 0013-0613. Retrieved August 23, 2021.
- "[98] Evidence of Fraud in an Influential Field Experiment About Dishonesty". Data Colada. August 17, 2021. Retrieved August 18, 2021.
- Lee, Stephanie M. (August 20, 2021). "A Famous Honesty Researcher Is Retracting A Study Over Fake Data". BuzzFeed News. Retrieved August 23, 2021.
- Ariely, Dan (August 16, 2021). "Dan Blog Comment" (PDF). datacolada.org. Retrieved 29 January 2023.
- Charlton, Aaron (2021-08-17). "Conflicts between Dan Ariely's statement and Footnote #14 (DataColada #98)". OpenMKT.org. Retrieved 2023-01-30.
- Charlton, Aaron (2022-08-21). "Dan Ariely claims authorship order shields him from blame; speculates that a low-level envelope stuffer committed the fraud". OpenMKT.org. Retrieved 2023-01-30.
- "דן אריאלי: "אנשים צועקים עליי ברחוב, קוראים לי רוצח ופסיכופת"". הארץ (in Hebrew). Retrieved 2023-01-30.
- "Francesca Gino - Faculty & Research - Harvard Business School". www.hbs.edu. Retrieved 2023-08-07.
- Quinn, Ryan. "Harvard Dishonesty Researcher Now on Administrative Leave". Inside Higher Ed. Retrieved 2023-07-24.
- Hamid, Rahem D.; Yuan, Claire (2023-08-03). "Embattled by Data Fraud Allegations, Business School Professor Francesca Gino Files Defamation Suit Against Harvard". The Harvard Crimson. Retrieved 2023-08-03.
- Svrluga, Susan (2023-08-03). "Professor accused of faking data in studies on dishonesty sues Harvard". Washington Post. ISSN 0190-8286. Retrieved 2023-08-04.