Once again about the RSCI

TrV-Nauka has repeatedly addressed the topic of the Russian Science Citation Index, RSCI (“What to do with the citation index”, No. 45, 01/19/2010 [1]; “Persistent assault on methodological heights”, No. 57, 07/06/2010 [2]; interview with project leaders in No. 58, 07/20/2010 [3]). Since RSCI data became a mandatory element of applications for competitions held by the Ministry of Education and Science, and became part of the indicators for assessing the performance of scientific institutions of the Russian Academy of Sciences, interest in this database has only increased. Numerous discussions on the Scientific.ru forum showed examples of obvious inconsistencies, but so far the conversation has been mainly about the incompleteness of the database, and, as a rule, developers quickly corrected those errors that became the subject of discussion. Moreover, in their July interview they said that the problem of incompleteness will be solved in September 2010 with the incorporation of data from Scopus into the RSCI. Published letter from the director of the Library for Natural Sciences of the Russian Academy of Sciences, Professor N.E. Kalenov, which he sent at the end of last year to the Minister of Education and Science A.A. Fursenko shows that the matter is much more serious - it seems that there are fundamental errors in the very algorithmic design of the database. Examples of such errors are given in the appendices to the letter, which are published on the TrV-Nauka website:

To the Minister of Education and Science of the Russian Federation

Mr. Fursenko A.A.

Dear Andrey Alexandrovich!

Since 2005, your Ministry has been funding the work to create the Russian Science Citation Index (RSCI), which is being carried out by Scientific Electronic Library LLC (SEL).

According to the “Standard Methodology for Assessing the Performance of Scientific Organizations Performing Research, Development and Technological Work for Civil Purposes,” approved by the Ministry of Education and Science by Order No. 406 of October 14, 2009, the RSCI should form the basis for assessing the publication activity of scientific teams and individual scientists .

RSCI presented on the NEB website ( www.elibrary.ru ), in 2009-2010. was actively discussed at scientific conferences and on Internet forums (see, for example, [4]). At the same time, almost all the presentations were critical in nature - representatives of various scientific organizations gave examples of RSCI errors related to the reflection of the publications of their employees.

Considering the importance of the RSCI as a tool specified in the “Standard Methodology...”, in July 2010 we conducted a fairly serious test of the RSCI on several randomly selected journals, organizations and scientists.

Testing of the RSCI for completeness and reliability of data showed the following.

1. There is no control and assurance of completeness of data entry for journals processed in the RSCI, which leads to loss of information.

2. The number of articles entered into the RSCI for a specific year for a specific journal, indicated when displaying information about the journal, does not correspond to the number of articles indicated in the distribution by year. At the same time, for some journals the total amounts are the same, for others they are not, which indicates an algorithmic error in the system.

3. Algorithmic errors of another kind within the system lead to the fact that articles by employees of the organization do not fall into the lists of articles of the organization, thereby distorting its rating.

4. There are losses of articles when comparing the journal and the author. For example, for 6 out of 14 authors of articles from the journal “Applied Mathematics and Mechanics” (2010, vol. 74. issue 1), there were no articles from this journal in their author profiles.

5. The system erroneously assigns articles to authors (for example, out of 22 works by one of the authors we tested, 7 were assigned to him incorrectly, while when logging into the system from another point, the correct authorship is given, which indicates a fundamental error in the system).

6. The sums of works and citations of individual authors of the organization do not coincide with the total number of publications and citations of the organization’s works both as a whole and in individual journals (for example, out of 15 works of one of the employees of the Institute of Mechanics of Moscow State University, only 8 were attributed to the latter, out of 16 works of another - 3).

7. There are repeated inclusions of articles in the database, statistical errors, etc.

The test results were brought to the attention of the developers, reported in August 2010 at a scientific conference in Yekaterinburg and published in a collection of scientific papers [5] with copies of the original WEB pages issued by the RSCI (the text of the publication is given in Appendix 1 ).

The specific errors we cited in the report have been corrected by the developers. However, the fundamental sources of these errors were not eliminated. In October 2010, we tested the RSCI using the example of the Information Resources of Russia magazine. At the same time, all the errors listed above appeared. The results of this testing, also with the original WEB pages of the RSCI, were published in No. 6 of the magazine “Information Resources of Russia” for 2010 [6] ( Appendix 2 ).

Our contacts with developers showed that they do not want (or cannot) correct obvious algorithmic errors.

In this regard, first of all, the question arises: who and how accepted into operation the RSCI system, implemented with public money and containing obvious errors? It seems that testing of the system was not carried out at all or that the admissions committee (if one existed) did not have qualified specialists in the field of computer science.

We were not able to find a single publication in the professional scientific press related to the principles of RSCI construction or the data processing algorithms underlying it. It is surprising how a research work for which not a single article has been published can be considered completed (especially in light of the “Standard Methodology...” approved by the Ministry that accepted such work)!

Despite numerous criticisms regarding the RSCI database, in August 2010, the Ministry headed by you announced an open competition for the right to conclude government contracts to carry out research for state needs within the framework of the Federal Targeted Program “Research and Development in Priority Areas of Development of the Scientific and Technological Complex of Russia for 2007 -2012" (event 5.1 - stage II), lot No. 1 of which is formulated as "Development of an information and analytical system for statistical analysis of bibliometric indicators of science based on data from the Russian Science Citation Index."

The very name of the lot and the tender documentation for it, approved by your deputy, Ms. I.P. Bilenkina, suggest that the basis of the created system should be the RSCI in the form in which it exists.

The employees who prepared and approved the competition documentation were either not familiar with the criticism of the RSCI, either in terms of the completeness of the reflection of materials, or in terms of the numerous errors contained in the RSCI database, or they deliberately ignored known facts.

I will quote some provisions of the competition documentation:

4.5. Specifications

4.5.1. The developed IS should provide the ability to conduct statistical analysis and ranking of research organizations and individual scientists of the Russian Federation on the basis of the following main information arrays:

4.5.1.1. Information about all publications from the 1500 most cited Russian journals (according to the Russian Science Citation Index (hereinafter referred to as RSCI) for 2005-2010) for the period from 2003 to 2012;

4.5.1.2. Information on publications of Russian authors in foreign and Russian translated journals for 1996-2012 (at least 400 thousand publications).

4.5.2. Formation of information arrays specified in paragraphs. 4.5.1.1. and 4.5.1.2., is provided by the research performer independently.

The wording of the last paragraph clearly determines that only the RSCI developer can be the winner of an “open” competition, since only he has the information array specified in clause 4.5.1.1.

The question arises: why should the contractor (if it is not NEB LLC) create a resource created at public expense independently?

Subsection 5.2 of Section 5 of the Tender Documentation (Technical and Economic Indicators) clearly demonstrates the incompetence of the Customer (in this case, your Ministry):

5.2. In the process of carrying out research work, the following values ​​of program indicators and indicators must be achieved (the minimum established values ​​are given, which the procurement participant has the right to increase when submitting an application for participation in the competition):

It is obvious to any specialist, even superficially familiar with the topic of the competition, that these indicators have nothing to do with the development of a “statistical system of bibliometric indicators of science.” Moreover, they do not make sense at all and cannot be verified, since the concept of “leading world information resources” is not defined and is extremely vague.

It is surprising how such a “passage” could appear in the documentation approved by your deputy!

Naturally, for such a “competition” there was not and could not be submitted a single application by anyone other than the developer of the RSCI (NEB LLC), who, having “won” the competition, will create some kind of analytical system based on erroneous data using public funds.

Immediately before sending this letter, I became convinced that fundamental errors in the RSCI had not been corrected. An example of the reflection in the RSCI of the journal “Space Research” and publications in it by an employee of the Institute of Problems of Mathematics of the Russian Academy of Sciences named after. M.V. Keldysh Corresponding Member RAS V.V. Beletsky is given in Appendix 3 .

When analyzing the results of the development of the RSCI, it becomes obvious that its developers completely lack a mathematical culture; they do not know how to check the correctness of the calculation of such trivial characteristics as the sum and average.

It is obvious that the continuation of work on the creation of an “information and analytical system for statistical analysis of bibliometric indicators of science based on data from the Russian Science Citation Index” without correcting the errors contained in the RSCI will lead to the emergence of an inoperable product, the use of which in practice will cause significant harm to domestic science.

As a specialist in the field of information technology, I believe that in the current situation it is necessary:

— suspend work on the creation of an “information and analytical system for statistical analysis of bibliometric indicators of science based on data from the Russian Science Citation Index”, canceling the results of the competition;

— create an expert group of specialists in the field of computer science and bibliometrics for testing and identifying RSCI errors;

— oblige the developers of the RSCI (Scientific Electronic Library LLC) to correct the algorithmic errors inherent in the system, identified by the expert group;

— publish and discuss in the professional press the requirements for an information and analytical system for statistical analysis of bibliometric indicators of science;

— announce a competition for the creation of a system for statistical analysis of bibliometric indicators of science that meets published requirements supported by specialists.

I am absolutely sure that my position is supported by all representatives of the scientific and educational spheres who, to one degree or another, have encountered the current version of the RSCI.

Nikolay Kalenov,
professor, doctor. tech. Sciences, Director of BEN RAS

1. http/trv-science.ru/2010/01/19/chto-delat-s-indeksom-citirovaniya

2. http://trv-science.ru/2010/07/06/upornyj-shturm-metodicheskix-vysot

3. http://trv-science.ru/2010/07/20/provedite-poisk-v-rinc-samostoyatelno

4. Indexes of discord: materials of the “Open Seminar” polit.ru .
www.polit.ru/author/2008/11/07/seminar.html .

5. N.E.Kalenov, O.V.Selyutskaya. On the Russian citation index // New technologies in information and library support for scientific research: collection of scientific papers. - Ekaterinburg, 2010. - P. 183-198 ( Appendix 2 ).

6. N.E.Kalenov, O.V.Selyutskaya. Some assessments of the quality of the Russian Science Citation Index using the example of the journal “Information Resources of Russia” // Information Resources of Russia, 2010, No. 6. - P. 2-13 ( Appendix 2 ).

7. Glushanovsky A.V., Kalenov N.E., Leksikova E.E. Database “SCIENCE CITATION INDEX” on CD-ROM. - M.: Bioinformservis, 1993. - 38 p.

8. Mokhnacheva Yu.V., Kharybina T.N. Application of bibliometric research methods in information support for scientists of the Scientific Research Institute of the Russian Academy of Sciences // Library of Natural Sciences of the Russian Academy of Sciences. Results and prospects. Digest of articles. M., 2008. - P. 255-270.

9. Beskaravainaya E.V., Mitroshin I.A., Slashcheva N.A., Mokhnacheva Yu.V., Bespalova L.A. Bibliometric analysis of publications of organizations of the Moscow region in the field of nanotechnology // Information support of science: new technologies: Coll. scientific tr. - M.: Scientific world, 2009. - P. 263-271

10. Mokhnacheva Yu.V. Russian-Belarusian scientific cooperation: bibliometric analysis of the current state and development prospects // Information resources of Russia, 2010. - No. 5. - pp. 11-15.