Repeat, please

In the series “Dr. House” there is an episode where House spite other doctors leads a mentally unhealthy guy named Steve, who believes that he has superpowers, to the aerotheme: look, they say, friend, you really know how to fly, you really are a superhero. After which the guy immediately joyfully jumps from a multi -storey parking lot.

From the point of view of science, Steve demonstrates a reasonable approach to business. The scientific is considered to be what can be reproduced. If you fly once, be ready to do it again. The hero makes, however, one mistake does not observe, as scientists say, the protocol of the initial study. But the very intention to reproduce the experiment of a commendable.

Science magazine on Thursday finally published the first results of a large -scale international project to verify scientific articles on psychology . For three years, the distributed “team” of the authors of 270 people and another 86 volunteers scrupulously repeated 100 experiments and statistical layouts from articles published in three leading specialized journals.

Confidently reproduced the result only for 39 - that is, less than half.

The authors carefully emphasize: psychology was not chosen for the fact that there are some particularly bad studies. It's just their specialization. Psychologists are also quite polite in response. Joint John Bokhannon, in his note for Science, quotes the president of the American Association of Psychological Sciences Charles Gallistel : “This very well performed study demonstrates that when it comes to reproducibility of results, psychology has nothing to be proud of.”

Both those and others appear on a complex information background. Just a few months ago, Science was forced to withdraw - in other words, to declare incorrect - a very loud article that attitude to same -sex marriages or abortions can allegedly change with the help of short personal conversations with people who have personal experience in this matter. (The study was the theme of one of the outputs of the super -popular American radio show This American Life , which is listened to more than two million people, so the review of the article from the scientific journal attracted a lot of attention in the United States.)

Confidently reproducing the result was only possible for 39 experiments out of 100.

Then the questions to the quality of the source data, which eventually led to the recall of the article, were the first to ask two third -party researchers who decided to continue their bright work. Anomalies in the initial data due to their falsification put an end to the star quarries of Dirku's mixture and dideric of the slippe (about the recall of the 55th in a account of the scientific article of the slipway, it became known literally on Wednesday, the day before the release of a fresh critical publication in Science ).

Psychology as experimental science has one important advantage. To interview people, give them assignments or observe their reaction, as a rule, easier than years to deal with Drisophila, resulting or mice. Nevertheless, a similar project to reproduce research results has recently begun for cancer biology. Oncologists, probably, still cannot forget how, several years ago, AMGEN and Bayer tried in their laboratories to reproduce several dozen key preclinical (that is, early) research.

Then the pharmaceutical companies - that is, people who should take the latest achievements of medical science and bring them to patients - received indicators of 11 percent and 25 percent of reproducibility. Of the 53 key ones for the field of work selected by AMGEN , only six were successfully repeated.

Another authors of the article in Science are persistently asking not to perceive their conclusions as evidence that modern science is “broken”, everything is terrible and it would be necessary to the basis, and then. Moreover, this does not mean that the authors of any of the unconfirmed articles - charlatans and crooks and there is no more trust in science.

It is not so simple to decide how exactly them should be perceived, especially people far from laboratories.

Imagine that you have put a huge multi -storey cake covered with glaze in front of you, and offered to bake exactly the same. Everyone around them says that the repetition of other people's cakes is a very important, necessary and useful thing, and that cakes and confectioners need to be judged not by beauty or reputation, but by whether, following the recipes, to get the same result.

At the same time, the recipe, a list of ingredients (including the most exotic ones), you have to find out the subtleties of cooking yourself. The kitchen, household appliances, assistant assistant and money for going to the store are also to find it yourself. Maybe there are people who will give them to you, but there are no guarantees.

You, most likely, will not be able to sell this cake, it will not add fame to you as a cook (except that other cooks will treat you wary, because you can question their skill in this way), and the author of the original confectionery can also be very offended or even refuse to cooperate. And yes, for the sake of this cake you are invited to abandon your own cakes.

Now mentally add a cherry to this. If you managed to repeat the cake, then this only says that the recipe was correct, and the ingredients are fresh - but not about the fact that a cake cannot be poisoned or that it is healthy. If it did not work, then this (suddenly) also does not mean that the recipe was wrong: perhaps you can’t read, unknowingly, by accident, they accidentally replaced sour cream with mayonnaise, bought another flour or opened the furnace door too early.

And then people who are not related to cooking will come-okay, frankly, journalists-who will ask: can anyone repeat your experiment to repeat the cake? And by the way, they will be right. (This question was, I'm not joking, the first at the telephone briefing on the occasion of the release of the article.)

If you have not yet escaped from this hypothetical cuisine-hmm, have you ever thought about an academic career?

So, if rudely, the problem of testing reproduction for scientific results looks like. This is a terribly ungrateful business that does not fit well with almost everything that characterizes modern Western science as a industry. The approach “Publish or or Perish) is still flourishing around, when the number of articles and their citation is used as the main, if not the only one at all. Under his oppression, on the one hand, there are few incentives to repeat other people's works (they are still not very willing to be very willingly, and where is the novelty?), On the other hand, there are a lot of temptations to “varnish” their own results and adjust them so that they are nevertheless accepted for publication.

Imagine that you have put a huge multi -storey cake covered with glaze in front of you, and offered to bake exactly the same.

Psychologist Anthony Grinuold once quite convincingly, in the scheme, showed how the existing system of scientific publications expectedly forms a skew not in favor of “negative” results (when, for example, ties or dependencies between the two phenomena could not be found). Such results are unfairly considered poor, unusual - unprintable, in general.

Greenoold painted his scheme in 1975; After more than three decades out of 100 selected articles, the “positive” result was found in 97. If real scientific studies really had such a Success Rate , the frequency of detection of new and significant effects - well, I don’t know, apple trees would have bloomed on Mars, probably.

Finally, the leading author of the study in Science about the rechecking of 100 psychological experiments - the psychologist Brian Nosek from the University of Virginia at a briefing assessed the number of scientific articles published annually at one and a half million - and immediately added that, most likely, this figure had already managed to become outdated. Unfortunately, we have no one more, “parallel” or “shadow”, science for rechecking all these results.

So, reproduction of the result against everything else. For the authors of the article, the choice seems to be unequivocal, because its first sentence sounds like this: "reproducibility is the fundamental principle of scientific progress." Disputes about whether it is possible to draw certain conclusions from the results obtained and why, they are meaningless if no one could get the results more, scientists write.

Does this mean that, say, it is necessary to legally oblige all graduate students the first n years to strictly repeat other people's work? Perhaps (there are such wishes). But you can still not only drive everyone on the subbotniks, but also to litter less - that is, to fight for the quality of the original articles.

The solution to the problem begins with its recognition, and here the case moves: from the purely internal affair of scientists, this problem is becoming more confident in public.

In 2011, in 2011, the Nature magazine celebrated a tenfold increase in the number of cases of calling articles in the last decade - according to their estimates at that time, there were about 11 percent of cases for the irreplaceability of the results, while plagiarism, self -plagiarism and data falsification - another 44 percent. Even then, it was clear that the matter is not only and not so much in the sharply fallen as scientific work, but that the shortcomings, mistakes and deception began to be detected more often.

A small blog Retraction Watch , which was only a year then, had just begun to accustom scientists and publishers to the fact that reporters and society as a whole have a matter of withdrawn publications. In the same note by Nature , the blog authors, Aivan Orange and Adam Marcus, recall that once the editor of one of the magazines in response to the question of a too foggy message about the recall said to them: “This is not yours, damn it, business.” Now, so they are unlikely to decide to answer.

After in 2000, the clinical tests of drugs in the United States were obliged to register in advance and publish the article at any outcome, the share of positive results fell from 57 percent to 8.

Not always of its own free will, not always fast, but science is rebuilt. Magazines agree to publish more articles on research of research, scientists - to make the initial data more affordable and send a protocol of the planned analysis to the journal (so that later it is impossible to “adjust” it for the desired results; the magazine, in turn, agreeing with the protocol, is then to publish any, not only beautiful, result).

On the example of medicine, it is clear how this threatens. After in 2000, the clinical tests of drugs in the United States were obliged to register in advance (and published at any outcome), the share of positive results fell from 57 percent to 8 .

There is some temptation to call what is happening “cleansing”, but, before succumbing to it, it is useful to remember that scientists are not aliens, science does not exist in any parallel dimension, and nothing that else, too, does not happen there. Let me know when a quote about the news and the person who bit the dog will cease to be relevant.

It is also useful to never forget that the vast majority of us are looking at science exclusively from the outside - no, in general, even looks not at science, but at its reflection in the media and popular culture. It is useful to remember that science also looks crooked (I can write it as a journalist) mirror, and sometimes we also make it look around there. Which of us has never joked about British scientists in whom coffee is useful today and is harmful tomorrow, he can take a pie from the shelf.

Senior SCIENCE editor Gilbert Chin unexpectedly began the same press-baking with the definition of the concept of “science” from Wikipedia. It seems that it would be worth adding a quote from a recent chic essay about the state of science on the statistical project Fivethirtyeight : “Science is difficult. Very, [unprintable], difficult. "

So no, science is not broken - you can believe people who themselves found 61 percent of non -processed results. Although, if you want, you can wait until their result also reproduces their result.