An annoying uniqueness

Science magazine published a study , the results of which may seem unpleasant to many scientists. The authors repeated 100 psychological experiments taken from the publications of recent years, and found that it was possible to get the same results as the authors of the initial study only in 39% of cases.

The project to study the reproduction of psychological experiments was conducted by scientists from Open Science CollBoration, this is a team of 270 experts from different countries, which was headed by the professor of psychology Brian Nosek from the University of Virgin. Experiments for playing attempts were selected from two areas: cognitive and social psychology. All selected experiments were taken from publications no earlier than 2008 in three authoritative scientific journals: Psychological Science, Journal of Personality and Social Psychology and Journal of Experimental Psychology: Learning, Memory, Memory, Memory, Memory And Cognition. The authors of original studies were invited to take part in the preparation of an experiment to reproduce the results they published.

For the sake of example, we will describe one and experiments on cognitive psychology, which psychologists under the leadership of Brian Noseck undertook to reproduce. In 2008, a test was held, the participants of which had to be called objects shown in the pictures. In the same pictures, words were written that were not related to images. According to the authors of the study, the participants experienced more difficulties when the words denoted objects from the same conceptual sphere as the drawn objects. For example, when the inscription “Caretra” was under the image of a steam locomotive, under the image of a gun - the inscription “Pistol”, under the plane - “ferry”, under the elephant - “mollusk”. However, with a repeated experiment, such an effect was not reproduced.

There were discouragingly many such bad effects and unconfirmed patterns. As already mentioned, only 39% of the experiments described were confirmed. Moreover, if you take only experiments on social psychology, then the number of reproduced results falls up to 25%. And in cases where the effects established by the authors of initial studies were confirmed, their quantitative expression was often significantly lower, on average, as indicated in the publications.

“The key warning that the reader must endure from our study,” says Bayan Noseck, is that no publication is the last word. Science is a process of reducing uncertainty, and no study is almost never a final result in itself. ”

The results obtained seemed disappointing to many, but many scientists reacted more or less calmly to them. Refutation of previous results is a natural part of progress. To know the structure of the world, scientists formulate questions and choose the ways to search for answers at their risk. Mistakes here are inevitable. As Alan Kraut from the Association of Psychological Sciences noted: “The only discovery that will be reproduced in 100% of cases will be banal, boring and probably already known: yes, dead people can never learn to read.” Another thing is that it would be nice to reduce the share of unreliable results at the stage of publication, and most importantly, to avoid the situation when the published study is perceived as truth, which is pointless to work or refut.

In this regard, the organizers of the results for the reproduction of the results consider such work and the publication of their results importantly. Until recently, magazines were very reluctant to print articles on the reproduction of previous experiments. This situation began to be corrected only in recent years. Already now, the editors of some psychological magazines have for publication in Science. Lynn Cooper, a psychologist from the University of Missouri, Colombia, who became one of the editors of the Journal of Personality and Social Psychology in January of this year, says that the article is sobering up and setting a clear task. Already, according to Lynn Cooper, changes are taking place in the journal’s policy, he will publish more articles on reproducibility of results.

It is interesting that most authors of those 100 publications, the results of which were checked, reacted positively to this and considered the work of Brian Noseck and his colleagues with a useful continuation of their research. Although most of those whose results could not be repeated are not ready to abandon their ideas, they plan to conduct further research on more representative material and eliminating the shortcomings of the methodology. Such a complacency attitude of scientists to the rechecking of their work is not always there. In May 2014, the magazine Social Psychology devoted a separate issue of testing research. In online comments, this activity was often evaluated critically. The accusations were expressed that she "resembles witch hunt", "is completely in the hands of a pile of smug, self -proclaimed sheriffs." Psychologist Pete Etshels, commenting on all this in his blog on the Gardian website, tried once again to explain that the inscrutability of the result does not mean accusing the authors of the initial study of dishonesty or, moreover, of conscious fraud, but only that the result is not repeated - the reasons for this should be revealed in further research. There can be many such reasons, so we must not swear, but try to find them. But it is unlikely that Etshels managed to convince all his opponents.

The fact that even when confirming the result of the experiment, its quantitative expression is on average half as much than in the initial study, the nose and his colleagues associate both with the actions of the authors who try to weed out of unsuitable results and with the policy of magazines, which take mainly those work, the results of which demonstrate quantitatively large effects. The last factor can be corrected if there is goodwill for the editorial offices.

As one of the methods to increase the reproduction of the results of psychological research, Brian Noseck proposes to change the existing procedure for the researcher. Usually, the psychologist formulates the hypothesis and comes up with an experiment, conducts this experiment, then, if the result is indicative, the psychologist writes an article, refers it to a scientific journal, where it goes through the review procedure and, finally, published. According to the proposed algorithm, the researcher should at the beginning describe the “Experiment design” - a detailed plan for its implementation, including goals, conditions, method, evaluated parameters, selected methods of statistical processing of the results. This description of the experiment passes through reviewing, during which possible omissions of the author are revealed. And only then does the researcher conduct his experiment, and publishing the results obtained in any case, regardless of whether his initial hypothesis was confirmed.