How to evaluate and preacting scientists?

How to evaluate the effectiveness of the scientist? How to encourage the best? Is there a foreign experience that can be used to use? The article by Pavel Chebotarev , a doct is devoted to these and other issues. physical. Sciences, G.N.S. Institute of Management Problems named after V.A. Trapeznikova RAS, member of the Society of Scientists.

About an attempt to fulfill this order not for us

One of the instructions [1] following the results of the meeting of the Council on Science and Education, held on December 20, 2013 [2], was to “submit proposals for the establishment of increased wages to certain categories of scientists who have achieved high results in scientific activity. The term is May 1, 2014 . Responsible: Medvedev D.A., Fortov V.E. "

Back in January, several members of the OHR, including the author of these lines, tried to offer something on this subject. The first thought was to take as a starting point the proposal of the Council of Onr Summer 2012 [3]. It came down to the following: at the first stage, supporting scholarships of 30 thousand rubles. In a month of employees of state scientific institutions that have published over the past 3 years a no less thorough number of articles in magazines and conferences indexed by Web of Science.

The calculation then showed that 10 thousand scholarships will turn out if you set the threshold at about 6 articles and take 3.6 billion rubles from the budget. per year. It was also proposed after a year to conduct a second stage based on a more detailed analysis of applications (applications were expected from scientists who have published at least three articles indexed by WOS for three years). Here it would be possible to take into account the scatter of publication activity in the regions, impact factors of magazines, etc. And to allocate another 10 thousand standard scholarships (30 thousand rubles per month), as well as 2 thousand increased scholarships (60 thousand rubles per month) for scientists who have a significant number of publications in the most authoritative highly impressive journals. Financing the second stage (taking into account the first) would require about 8 billion rubles. per year.

When discussing this idea in January 2014, a fear was expressed that the second stage could fall as a victim of bossy arbitrariness or general instability, so differentiation in the regions is already desirable at the first stage - otherwise there will be almost no humanities among its winners, and with the “mats”, biologists, physicists and chemists will leave mainly. How not to offend representatives of humanitarian and social sciences? It was proposed to take into account for them also publications in magazines indexed by SCOPUS, and in a number of RInts magazines.

But how to choose the magazines of the RSCT-according to the impact factor? There were also examples of reputable humanitarian magazines with a very low impact factor of the RINC, and magazines of far from solid, having a rather high wound impact of the Rins (in the same areas). It became clear that the magazines of the RSCT must be taken expert. However, conducting a correct examination is a difficult matter. After all, many potential experts are included in the editorial board of various magazines, and this is fraught with a conflict of interests. Another problem is that in a number of areas it is impossible to judge only by articles, since monographs and other results of scientific activity are important. And to reject monographs even more difficult than magazines.

In general, an attempt to build a balanced universal proposal based on a bibliometric approach came across serious difficulties.

On the sketch of an integrated approach

We could not forget that not only humanities are always rebelled against the bibliometric measure of success (among which philosophers are especially decisive [4]), but also mathematics. Moreover, they do not sit back, but already form expert structures. So, in December 2013, at the separation of the mathematical sciences of the Russian Academy of Sciences, an expert council was created to assess the effectiveness of scientific organizations.

The goals of the Council include the creation of expert commissions to solve specific problems, among which are the assessment of the success of individual scientists. During the work of such mathematics commissions, they propose to use the best features of the certification system of scientists Research Excellence Framework (Ref) - 2014 - in this system, which we will discuss in the last section of the article, taking into account bibliometric indicators is strictly limited, and for many sciences, is completely excluded.

It is quite obvious that to impose a bibliometry in those areas where scientists not only in one vote declare its unacceptability, but they themselves create expert structures themselves are wrong. Therefore, the proposal may be approximately as follows.

1. To divide the general premium fund between the regions in proportion to the number of scientific personnel.

2. To develop recommendations on the number of scholarship levels (level 1, 2 or 3), the magnitude of the scholarship of each level and the percentage ratio of the amount of scholarships of each level. There is a reason (although not iron) in the uniformity of these values ​​in areas. And the easiest way if the level is one.

3. Each profile department of the Russian Academy of Sciences (remember that the responsible for the assignment is V.E. Fortov), ​​together with representatives of institutions, forms a competitive council. In those areas where there is a corps of experts [5], its representatives should be at least 2/3 of the advice. It makes sense not to include directors of institutes and their deputies, as well as other functionaries in the council, so that it does not become a lobbying tool.

Each competitive council makes one of two: either develops a premetric criterion for awarding a scholarship of each level, or forms a commission for expert awarding of scholarships and develops a detailed regulation of the work of these commissions. The prototypes of some elements of the regulation can be found in the document “Panel Criteria and Working Methods” [6] of the English certification system Ref-2014. But it must be borne in mind that this is a system of assessing units, not individual researchers. Like the system [7], recently proposed by the public control commission and the results of reforms in the field of science; It makes sense to use it to no less.

4. It is fundamentally that competitive councils and expert commissions enjoy the confidence of the scientific community. Therefore, the best way to form them is elections guaranteeing representation of both RAS and researchers who do not have academic titles. The procedures for nomination of candidates, elections, as well as the regulations of expert commissions should be approved by the professional conference of researchers. This approach will provide decisions with legitimacy.

Is there a passage between Scylla and Haribda?

The approach described in the previous section, I then offered colleagues, but did not receive their support. Because, with all the vices of the bibliometry, the involvement of decisions in such [ex-] of the “feudal” structure as RAS, it seemed even more vicious to them. Even if you demand the approval of all fundamental decisions by the scientific community. Indeed, in art, take a good idea (for example, the idea of ​​democracy), leave an empty form from it and fill it with the opposite (say, anti -democratic) content of domestic craftsmen. The only thing this number may not pass with is a simple formal criterion.

It should be noted, however, that colleagues are mainly naturalists, i.e. Representatives of regions where the correlation of bibliometric indicators and real success has a good level of significance.

The worst, however, is not even that there are other areas, but that formal criteria destroy scientific motivation. Bonuses for the number of articles provokes Salami Slicing [8]: the author produces Smallest Publisable Units, scatters them through referred magazines and easily overtakes a stronger colleague for whom such tricks are unacceptable. Once, a sick Salami Slicing'om is unlikely to once take up difficult tasks, fraught with the loss of growth in the number of publications. But it will very succeed in minimizing their specific quality, i.e. In the production of articles making the minimum possible contribution to science.

To preserve healthy motivation, it is necessary that the next scenario is possible. Based on the results of certification, experts say that the scientist X, who published 1 of the article in 2 years, did something in it, having all the chances of entering the textbooks. And it should be put forward to the award, as well as think about how to help him develop a promising idea. And the scientist Y, who published 8 "Slimes", perhaps, corresponds to the position held, but nothing more. And he would not have worsened the impression of his work if he published all this in 2-3 articles.

But for such a scenario to be possible, at least an examination is needed.

A little about the new certification system in the UK

Great Britain is, of course, one of the flagships of the Western world regarding the seriousness and thoughtfulness of scientific policy. All the more interesting to us is how scientific certification occurs there. In this brief article, we can touch only a few of its features.

The outgoing English assessment system, RAE (Research Assessment Exercise) [9, 10], had, no, not the number of publications, but the quality of the study; She, like the new one, was expert. The Research Excellence Framework (Ref) [6] implemented system is also very interested in quality, but 20% of the final assessment in it is IMPACT (not to be confused with the magazine impact factor), i.e. The influence, impact of research on the world and, in particular, its significance for the development of science. Impact questions are annoying, but taxpayers want to know what the money is going for.

All scientific disciplines in Ref can be divided into those where bibliometry is not used in evaluating in general, and those where it is used with great restrictions. The first includes mathematical, engineering, humanitarian, social sciences, management, etc. To the second - the science of life, about Earth, chemistry, physics, computer science and economics.

“Used with restrictions” means the following: bibliometric indicators are considered as one of the means of evaluating quality, but not the main (primary) means. It is noted that the number of citation is often an unreliable indicator, for example, in cases of applied or fresh studies.

Quoting for most disciplines of the second type are calculated by Scopus; Specialized databases can also be used, and for computer science - also Google Scholar; WOS is never mentioned.

In general, we are talking about the expert assessment system with detailed and carefully worked out regulations that differ for different fields of knowledge. And what else is interesting: neither the impact factors of magazines, nor any ratings of magazines or publishers should be taken into account when evaluating. Of course, in an implicit way, all this is taken into account, but in general, perhaps, ReF somewhat limits the use of biblometry compared to RAE.

In contrast, the assessment systems, for which the more publications, the better, in the English system, both the former and the new, the scientist presents for a five -year reporting period no more than four publications! But - with the formulation of the contribution to the science introduced by each of the work, compared to what was “what was” [6,10]. Note that at the same time, the cutting of a piece of salami into 8 “Slies” will be more likely to harm: half of them simply will not be taken into account. The usefulness of limiting the number of articles under consideration has been noted more than once in some offers of OHP.

Finally, it makes a strong impression as follows: the Ref system has been developed since 2007. In 2008-2009, a pilot assessment was carried out on the basis of several universities, the result of which was the rejection of the planned wide use of bibliometry. In 2010, the planned introduction of the system was postponed for a year. The current draft regulation has been published in January 2012. Then it was discussed for almost two years; a large number of standardized questionnaires with reviews were collected; The typical volume of the filled questionnaire is 10 pages. Assessment by the new system begins in 2014; Its first results will be announced at the end of the year. Thus, more than seven years have passed from the start of development to the first results. Moreover, this is for the system, not fundamentally new, but developing the existing ... We should learn this smoothness and thoughtfulness of changes!

I think that it would be useful to translate the document [6] into Russian and familiarize all interested persons with it in the Ministry of Education and Science, Fano, RAS, the Council of Science and Education, and institutions.

Of course, Russia is not England. But if there is an unreliable impact factor via WOS, then how should we look at the Impact Factor on the RINC? And the main argument, because of which the British built and improve the expert system, depends little on local conditions. It consists in the fact that science is one of the most substantial entities in the world, and the reduction of the achievements of scientists to Tsifiri introduces them into a strong temptation of profanity.

Finally, one remark must be made. It is very strange to consider the order to “establish increased wages to the certain categories of scientists who have achieved high results” in isolation from the future structures of the states and the salaries of scientific institutions, and there is no clarity with this yet.