"The Rooter", take two

Another journal from the “VAK list,” “Bulletin of Tomsk State University (philosophy, sociology, political science),” published a computer-generated article. However, unlike the case with the “Journal of Scientific Publications of Graduate Students and Doctoral Students,” which was excluded from the Higher Attestation Commission list, this did not have any administrative consequences for the journal.

In September 2008, the editors of the Trinity Option conducted an experiment. A computer -generated hoax article “Rooter: an algorithm for typical unification of access points and redundancy” was sent to the “Journal of Scientific Publications of Graduate and Doctoral Students . ” We translated the text generated (in English) by the SCIgen program (which was created by a group of students at the Massachusetts Institute of Technology) using another program, ETAP-3 (developed at the Laboratory of Computational Linguistics, IPPI RAS). The goal of TrV was to draw the community’s attention to the existence of journals that pretend to be scientific and are even included in the Higher Attestation Commission list, but in fact should not be considered scientific. The details of this story were described in TrV No. 13. The experiment turned out to be more successful than could have been expected. In addition to the fact that the word “Rooter” itself became a household word in the scientific community, the scandal led to quite tangible administrative measures: the journal was immediately excluded from the Higher Attestation Commission list (for a discussion of the results and consequences, see TrV No. 15, p. 3 and TrV No. 16, p. 7).

It has now become clear that there were other consequences. Rosobrnadzor, together with the RAS Commission for Combating Pseudoscience, created a special working group, which was tasked with conducting similar checks (by distributing computer-generated quasi-scientific texts “for publication”) of all scientific journals on the Higher Attestation Commission list that are academic or university publications. At the same time, to generate texts, a new original program RODOS was used, developed by a group of graduate students and students of the Moscow Institute of Physics and Technology and the Institute of Information Problems of the Russian Academy of Sciences.

We can state with some relief that the situation was not as catastrophic as it seemed: of the first fifty audited journals on the VAK list, only one succumbed to this insidious trick: “Bulletin of Tomsk State University (philosophy, sociology, political science).” He published an article “Darwinism” [1], containing a stream of (computer) consciousness with the following content:

"1. Satan. It was necessary to start the conversation about Darwin directly with Satan. The road is straighter” (p.89);

“I’ll tell you a secret: teeth, their first appearance and transformation into a jaw is for me personally the most impenetrable moment, a refutation of Darwinism. And the gums themselves. They are made from a different substance than the meat that fills our body. This substance is hard, almost like bone, and when the tooth is pulled out, it is loose and blood flows. How did all this, along the evolutionary path, fly into the mouths of creatures and take hold there? Evolutionary. It wasn’t, it wasn’t, and then gradually it became. Nonsense, nonsense. Baby talk” (p. 104);

“Some kind of invertebrate worm crawled and crawled, and in the billionth generation it has a mutation: inside the cartilage has ossified, the future, I’ll say right away, the spine. I’m not talking about the impossible, how he will pass on this crutch inside himself to his son or daughter by inheritance - it’s absolutely impossible. I'm talking about him himself - a freak. After all, with this prosthesis inside him, he will no longer be able to crawl, and I’m afraid that he won’t be able to mate either” (p. 105);

“Some “scientists” (pickled cucumbers)...What can we say about these pickled futurists? Darwinists! (p. 107);

“And I’m afraid to even say the word “brain.” Two kilograms of unsympathetic substance. Looks like a pile of crap. But it doesn’t smell, he thinks!” (p. 108);

“Why am I so mad? Why am I so excited about Darwinism? I also found something good for me. Well, it’s not great, it’s bad, so why be sad about that?” (p. 111);

"3. Darwinism and the Bible. I wrote this chapter, and there were about two pages in it, and I erased it” (p.95).

And so on.

The fate of the "Bulletin of Tomsk University" and its editorial board headed by prof. S.S. Avanesov was hanging by a thread. The magazine was saved by legal subtleties. The fact is that the SCIgen program used for “Rooter” can generate texts only in the field of computer science. Therefore, the working group used another program to create texts in various fields, RODOS, which generates nonsense by independently combining small fragments of other people's source texts.

In particular, for texts on biology and evolution, the computer shuffled finely chopped content from creationist sites and forums on the Russian-language Internet (antidarvin.ru, etc.). As a result, a number of members of the working group had concerns: had the source texts been chopped up enough for the final product to be considered patent nonsense? The working group was unable to find legally precise wording.

This would not have become a serious obstacle to the adoption of administrative measures, if not for the recent appeal of the editor-in-chief of the “Journal of Scientific Publications of Graduate Students and Doctoral Students” V.V. Ivanov to the arbitration court with a claim against the Ministry of Education and Science (see TRV No. 4/23, p. 12 ). V.V. Ivanov contested the exclusion of his journal from the Higher Attestation Commission list, arguing that this would limit free competition and violate antimonopoly laws. Although the court ruled in favor of the defendant, the Ministry of Education and Science, the Higher Attestation Commission decided not to get involved with the Tomsk journal, especially since a general revision of the entire list is planned (see TRV No. 17, p. 5).

It is possible and necessary to discuss the question of how correct this method of checking logs is by administrative authorities. This really looks like a provocation, and what is allowed to private individuals no longer looks so good in the hands of governing bodies. But in any case, I am glad that our students were able to create a program that is not inferior in efficiency to the famous SCIgen.

1. V.B.Rhodes. Darwinism. Bulletin of Tomsk State University (Philosophy. Sociology. Political science). No. 1(2), pp. 89-119 (2008) http://sun.tsu.ru/mminfo/000063105/phil/02/image/02-089.pdf

Kirill Bocharov

Interview with a member of the working group for checking the quality of journals, developer of the RODOS program, candidate of technical sciences Mikhail Kovalev

— Are you satisfied with the results of the experiment?

- Yes and no. As a Russian scientist, I am glad that the level of most of our scientific journals turned out to be not as terrible as everyone thought after the story with the “Rooter”. And as the author of the program, I, of course, would like the texts it generates to be more similar to real ones. I must say that based on the results of the work, some changes will be made to the next version of the program.

- For what? Does anyone plan to continue testing magazines for strength?

— Yes, as far as I know, this work continues. There is also a purely scientific interest - no one has canceled the Turing test. And then, in the world there are not only journals - there are hundred-page reports that scientific institutes are forced to write in large numbers, there are patents, conference abstracts, in the end, there are student abstracts...

— It seems that students get by by simply downloading ready-made essays from the Internet.

— Yes, but such abstracts are easy to catch using Google or various “anti-plagiarism” programs. And RHODES makes the text unrecognizable.

— RHODES — what does this name stand for?

- No way. This is not an abbreviation, but the name of a Greek island. When the story with “The Uprooter” happened, we said that it was not very difficult to write a program that generates humanoid texts, and our colleagues remembered Aesop’s fable “The Braggart.” It tells how a man who boasted of extraordinary jumps on this island was told: “Here is Rhodes, jump here!” (Hic Rhodus, hic salta).

— So, you wrote this program “weakly”?

- Not certainly in that way. Our group has been working in the field of coherent text generation for a long time. But the reason for writing this program was, indeed, “The Rooter.”

— What is the difference between your program and the one who wrote “The Rooter”?

— The SCIgen program uses a context-free grammar. This is a well-known algorithmic technique. In principle, since the source code is available, it would be possible to retrain the program on new material. But this would require too much effort. So we went a different route. We used two well-known algorithms that were developed for other purposes, but, as it turned out, are suitable in our case and, most importantly, do not need to be retrained for each new area.

One algorithm is used in the famous bibliographic biomedical database PubMed , which has the concept of “related articles”. The algorithm analyzes abstracts of articles and groups them according to similar content. In our program, this algorithm forms the basis for the article - a sequence of fragments that talk about the same thing.

The second algorithm is called “Markov Morphological Analyzer”. This is a linguistic technique, which in our case edits successive fragments so that there are no contradictions in grammar - sentences must be consistent in time, number, etc. Simply put, the text should be “smooth”.

-Did you manage to achieve this?

- I think so. See for yourself - the text of the article is available on the Internet ( http://sun.tsu.ru/mminfo/000063105/phil/02/image/02-089.pdf ).

— What texts would you take as source texts?

— Various pseudoscientific Internet sites and forums were mainly used. In addition, “to revive the style”, A.P. Chekhov’s “Letter to a Learned Neighbor” and some stories by Mikhail Zoshchenko were added.

— Have you seen reviews of articles written by your program?

— Yes, some journals not only rejected the articles, but also sent reviews. I must say they were quite similar. As a rule, these were rather short texts, in which a lack of novelty was noted, sometimes a poor article plan, and general incoherence. Many reviewers pointed out factual errors, which is not surprising - materials of rather dubious origin were used as sources of text for the articles.

— Why was your development used?

— Partly by coincidence. When the director of our institute (IIP RAS) G.L. Skuratov-Belsky was included in the academic working group to study the quality of journals, he asked us to speed up the necessary work on fine-tuning RHODES. On the other hand, I am not aware of other projects of this kind.

— Is your program available via the Internet?

- Not yet. In fact, we are not sure that this would be correct. We need to weigh the pros and cons.