{"id":160778,"date":"2020-05-20T11:26:57","date_gmt":"2020-05-20T15:26:57","guid":{"rendered":"https:\/\/today.uconn.edu\/?p=160778"},"modified":"2020-05-21T15:22:38","modified_gmt":"2020-05-21T19:22:38","slug":"reproducibility-crisis-might-software","status":"publish","type":"post","link":"https:\/\/today.uconn.edu\/2020\/05\/reproducibility-crisis-might-software\/","title":{"rendered":"The Reproducibility Crisis Might Be All in Your … Software"},"content":{"rendered":"

Science is supposed to be repeatable. But more and more often, scientists are finding it hard to re-do published experiments and get the same results. Now, a large group of neuroscientists suggest software may be part of the trouble.<\/p>\n

Science relies on repeated observation to understand the world. Anyone who does the same experiment, observing the same phenomenon, should be able to draw the same conclusions. But lately, fields as diverse as behavioral psychology and cancer biology have had trouble reproducing results from supposedly identical experiments.<\/p>\n

Now, a large collaboration of neuroscientists report in the May 20 issue of Nature<\/a> that they have managed to reproduce the reproducibility crisis in a single experiment. The premise was simple: many different teams of neuroscientists would all analyze the same brain imaging data to answer the same set of questions. In theory, they should all come to the same set of conclusions. But that\u2019s not what happened.<\/p>\n

Brain imaging data is an appealing target. Popular descriptions of the technique make it sound as if brain imaging delivers obvious results: the subject thinks about or does something, and a specific part of their brain lights up. But in practice, it requires quite a bit of data analysis.<\/p>\n

First of all, the raw brain imaging data isn\u2019t assessed by eye. Researchers must use software analysis to process it into pictures, and there are many choices involved. As Roeland Hancock, Associate Director of the UConn Brain Imaging Research Center and assistant research professor in the Department of Psychological Sciences, explains, first the researchers have to pick the software package you want to use to process the data. Then they choose the specific algorithm within that program. Researchers also must choose which confounding factors to account for, and which probably don\u2019t matter and won\u2019t be accounted for. And then there\u2019s the smoothness, or how intentionally blurred the image will be to account for tiny spatial inconsistencies in the data.<\/p>\n

\"Two
Both of these images represent the same set of brain scan data. But the yellow highlighting shows that one team\u2019s analysis (top) discerned brain activity in far broader regions than the other team\u2019s analysis (bottom) did. (Courtesy of Roeland Hancock)<\/figcaption><\/figure>\n

 <\/p>\n

\"\"<\/p>\n

\u201cThis shows how variable the results might be even between what we might consider standard methods of analysis,\u201d says Hancock.<\/p>\n

The organizers of the study gave 70 teams, including two teams of UConn graduate students funded by a joint program of the Connecticut Institute for the Brain and Cognitive Sciences and the Brain Imaging Research Center (Charles Davis, Monica Li, Yanina Prystauka, Emily Yearling, and Xu Zhang), the same set of data: fMRI brain scans from 108 people engaged in a money game. Along with the data came a list of true-or-false hypotheses about that data: were, or were not, various specific brain regions engaged in that task?<\/p>\n

Out of nine hypotheses, four received very consistent answers on which the majority of teams agreed. But the other five hypotheses received very inconsistent answers.<\/p>\n

After analyzing the results from all the teams, the study authors have some recommendations for anyone who wants to do reproducible science: process your data in multiple different ways, then aggregate the results. And ideally, ask someone uninvolved in the research to process your data too.<\/p>\n","protected":false},"excerpt":{"rendered":"

UConn researchers say that software may be partly the cause for many science experiments’ inability to be repeated. <\/p>\n","protected":false},"author":79,"featured_media":161202,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_crdt_document":"","wds_primary_category":0,"wds_primary_series":0,"wds_primary_attribution":0,"footnotes":""},"categories":[2226,2076,2225],"tags":[],"magazine-issues":[],"coauthors":[1899],"class_list":["post-160778","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-clas","category-research","category-uconn-storrs"],"pp_statuses_selecting_workflow":false,"pp_workflow_action":"current","pp_status_selection":"publish","acf":[],"publishpress_future_action":{"enabled":false,"date":"2026-04-13 01:33:41","action":"change-status","newStatus":"draft","terms":[],"taxonomy":"category","extraData":[]},"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/posts\/160778","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/users\/79"}],"replies":[{"embeddable":true,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/comments?post=160778"}],"version-history":[{"count":7,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/posts\/160778\/revisions"}],"predecessor-version":[{"id":161279,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/posts\/160778\/revisions\/161279"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/media\/161202"}],"wp:attachment":[{"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/media?parent=160778"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/categories?post=160778"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/tags?post=160778"},{"taxonomy":"magazine-issue","embeddable":true,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/magazine-issues?post=160778"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/today.uconn.edu\/wp-rest\/wp\/v2\/coauthors?post=160778"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}