Translated material
"Far reading" of "Gorky"
What conclusions can be drawn? There are quite a lot of references to Stalin and Hitler, there are other modern politicians, but a much more important figure is L. Tolstoy (important for I. Pernikov, L. Oborin, S. Naranovich, K. Milchin, B. Kupriyanov, I. Kozlov). G. Yuzefovich is important for L. Oborin and V. Vladimirsky. Pushkin - for M. Nesterenko, I. Napreenko, I. Martov. Dostoevsky - for I. Pernikov, M. Nesterenko, B. Kupriyanov. As can be seen from the word clouds, among the top ones, authors with a background other than philological (and/or writing on other topics) stand out: N. Protsenko, S. Naranovich, I. Napreenko, D. Borisov. In general, top authors are distinguished by literary and social criticism of leftist orientation.
Topic Modeling
The authors of Gorky write about a wide variety of books: not only fiction, but also scientific ones, about books in different languages and from different centuries. Gorky contains interviews with various scientists, writers, publishers and public figures. Moreover, beyond the top 15, the specialization of the authors of Gorky is much wider. It would be interesting to know into which thematic collections the Gorky texts can be divided without reading more than 4,000 texts. The pragmatics of this for an electronic publication is that each text will receive a thematic hashtag, which the reader can use if he wants to view publications similar to the text he just read. One solution is to display texts by the same author or with the same keywords. (This method does not always work: the same author can write on completely different topics; the same thing can be spoken about in completely different words or from the perspective of different disciplines.) It is better to show texts of similar topics or content.
It is also important to note that the text is rarely devoted to only one single topic: as a rule, any material is a mix of several cross-cutting topics present in it in different proportions. Thus, one text can and should have several tags or topics.
This is why topic modeling of a text corpus is needed. There are three most common statistical models, or formulas, used for this: Latent Semantic Analysis (LSA), Probabilistic LSA (PLSA), Latent Dirichlet Allocation (LDA). I won’t go into detail about the materiel; there are excellent lectures about it by K. Vorontsov. For the Gorky topic modeling, I used the LDA model, which, like the others, is implemented in the Genism library for python. Using this model also requires text preprocessing. Inside LDA is a bag of words. Texts are transmitted in the form of a list of lemmas, corresponding texts, and are processed not entirely, but in parts, so that there is enough computing power. Processing is also iterative: the model repeats calculations several times to increase accuracy.
The catch is that it is necessary to transfer to the model the number of topics that will be identified in the text corpus depending on the keywords specific to these topics. The quality of the model is measured by the average coherence of the identified topics. Each topic has its own coherence, which is determined by the share of keywords of this topic in the texts assigned to it compared to the shares of keywords from other topics. (There are two coherence metrics: u_mass and с_v.) The optimal number of topics can be found by sequentially training LDA models with an increasing number of topics and monitoring the dynamics of average coherence. After a certain number of topics, their average coherence, as a rule, stops growing. The more diverse the text corpus, the more topics it should be divided into.
However, in addition to the mathematical metric, there is also human interpretability of the topics identified by the model. In other words, can a person, looking at the keywords highlighted by the model for each of the topics, easily understand what kind of topic it is, that is, name it, title it. The problem of interpretability is aggravated by the fact that sets of keywords related to different topics may overlap. Therefore, the Jaccard similarity measure is closer to human interpretability . The selection of an optimal topic should maximize the coherence of topics and minimize the similarity between them.
The optimal number of topics for the Gorky LDA model is 11.
For each of the 11 selected topics, I will build a graph of 10 keywords, which shows how often a keyword appears in texts related to the topic, and what its importance is in highlighting the topic.
Let's name the topics by their keywords (there are 10 of them in the graphs above, more can be displayed):
Topic_0: Russian classics, domestic prose
Topic_1: Lyrics, criticism, translated literature, music
Topic_2: Soul, poetry, physicality, philosophy
Topic_3: Comparative history (history of state, religion; military history)
Topic_4: Cultural projects, events, literary awards
Topic_5: Science fiction, awards
Topic_6: Publishing, market
Topic_7: Literature and drama, feminism
Topic_8: Revolution of 1917, USSR, memory
Topic_9: Social sciences (sociology, political science, economics, philosophy)
Topic_10: Natural sciences (biology, physics, astronomy), Japan
As you can see, Topic_1 and Topic_2 intersect in “poetry” (“verse” and “poem” are contextual synonyms), Topic_3 and Topic_9 intersect in “state”, Topic_8 and Topic_9 - in “revolution”, Topic_4 and Topic_5 - in “premium” " Topic_1: Lyrics, criticism, translated literature, music and Topic_2: Soul, poetry, physicality, philosophy, as if they could be combined into one. They can probably be combined with Topic_7: Literature and drama, feminism. Perhaps due to the large number of texts written by individual authors from the top 15 on generally the same (or similar in meaning) topics, the model records their unique language as different topics. (So, Topic_1 is mostly texts by L. Oborin.) In this case, a good indicator of the quality of the model is how successfully it unites different authors writing on the same topic, and also separates the texts of one author on different topics, contrary to linguistic and stylistic similarity.
As a result, the LDA model represents each text as a mix of topics in certain proportions depending on the representation of keywords characteristic of a particular topic. For example, LDA presents the text by B. Kolonitsky “Whoever goes first behind the coffin is in charge” dated 12/14/22 with the subtitle “On the politics of memory and new products of the EUSP publishing house” as follows:
(topic number, its share)
[(0, 0)
(1, 0.039592195)
(2, 0)
(3, 0.5091702)
(4, 0)
(5, 0)
(6, 0)
(7, 0)
(8, 0.15597652)
(9, 0.29443944)
(10, 0)]
This means that, according to the model, it is 50% Topic_3: Comparative history (history of religion, state; military history) and 30% Topic_9: Social sciences (sociology, political science, economics, philosophy), 2% Topic_8: Revolution of 1917 years, USSR, memory. In principle, this is a good and quite meaningful result. B. Kolonitsky is one of the leading Russian cultural historians of the 1917 revolution and the interwar period. Cultural history involves the reconstruction of how (with the help of what words) contemporaries of historical events understood and constructed the meaning of what was happening and an explanation of why they did it this way, i.e. the anthropological, philosophical-hermeneutic, cultural and sociological components of cultural history are really huge. The text is a transcription of the presentation of new books by the European University from the series “The Age of Wars and Revolutions”, the general theme of which is the study of the experience of contemporaries of the First World War and the Civil War in Russia. That is, the share of Topic_8: Revolution of 1917, USSR, memory could be noticeably higher. However, the text also contains B. Kolonitsky’s answers to listeners’ questions about the methodology of memory studies; moreover, in comparison with other Gorky texts classified by the model as Topic_8, B. Kolonitsky uses a different language, or the proportion of key words for Topic_8 is small (book after all, about the Civil War, and not about the revolution of 1917, which, as follows from the text, the whites preferred to call the “Red Troubles”).
Let's find 20 texts that, in terms of the mix of topics (according to the LDA model), most resemble the text by B. Kolonitsky above. To do this, we multiply the matrix, where the rows are the texts of the Gorky corpus, and the columns are the shares of topics in the text by the vector of B. Kolonitsky’s text, i.e., only the weights of the topics corresponding to it, then we sort the result and take the indices of texts for which it is minimal , i.e. the mix of themes is the same as that of B. Kolonitsky’s text or close to it. The model considers the following texts to be close:
A. Ranchin (06.27.2022): about one of the fundamental Russian chronicles
F. Nikitin (05/16/2023): seven books about Baptistism in Russia
A. Ranchin (03/09/2022): about the earliest surviving works of ancient Russian literature
N. Rhodessky (01/19/2022): about Jan Assmann’s book “Political Theology between Egypt and Israel”
D. Stakhov (03.10.2018): review of new books on Russian history
A. Ranchin (08/23/2022): about the first ancient Russian life of a warrior prince
N. Protsenko (05/23/2019): review of a book about the formation of the Ukrainian nation
F. Nikitin (12/25/2023): seven books about Mennonites and their faith
A. Ranchin (08/11/2022): about the lives of two saints executed for political reasons
K. Galeev (06/11/2019): review of the book by Michael Khodarkovsky “Steppe Frontiers of Russia”
I. Pernikov (08/23/2023): about Andrei Bannikov’s book “Being a Legionnaire”
A. Ranchin (05.10.2022): about the life of the abbot of the Russian land
I. Yurchenko (09.23.2020): about Amiran Urushadze’s book “Free Water”
D. Stakhov (06/27/2017): from the Golden Horde to the religious schism
As can be understood from the mix of topics, in similar 20 texts there were selected materials in which, like B. Kolonitsky, approx. 50% Comparative history (history of religion, state; military history) and approx. 30% Social sciences (sociology, political science, economics, philosophy), and there is also an admixture of Russian history. As a result, as is obvious from the subheadings, among them there are many texts about books on the history of religion in Rus' and comparative history, about the interwar period - a couple of books, only one text about a book about the Civil War. The semantics are a bit lacking, although the disciplinary selection is correct. I. Pernikov’s review of A. Bannikov’s book “Being a Legionnaire” can help you understand something not only about the legionnaires of the Roman Empire, but also about the soldiers of the empires who fought in the First World War.
The LDA model also makes it possible to represent the entire corpus of Gorky texts in the form of points on a two-dimensional plane (scatter diagram), where the color of the points is the dominant theme of the text. The graph is below, but we won’t dwell on it.
Transformer embeddings and latent semantic analysis
The problem of latent semantic analysis based on the vector representation of words using their TF-ID metrics is that such frequency models (based on the occurrence of words in a separate document and in the corpus as a whole) do not take into account word order and context, and therefore for them “treble clef” and “spanner” are the same “wrench.” Fortunately, this problem can be solved using distributive semantics, or graph representations of words. In this case, the meaning of the word is determined precisely by the context in which it is used.
We won’t go deep into mathematics, just imagine that the neural network saw a lot of texts and “remembered” in what contexts certain words are usually used. She takes this information about the context of the word, presented in the form of a one-dimensional vector, as its semantics. For example, the embeddings/vectors of the words “cat” and “dog” are similar because they are used in the same context (conventionally, the context “pets”), the embedding of “cat” will be closer to the embedding of “dog” than to the embedding of “king” , and, in turn, the embedding of the “king” is closer to the embedding of the “state” (conditionally, the context is “politics/government” - see, for example, here ).
To obtain embeddings of not only words, but also entire texts, neural network models are used, known as transformers , which, when encoding, take into account the order of words in the encoded text. (There are encoder transformers, i.e. encoders and decoders, as well as transformers that combine both - the latter include models like GPT, which take meaningful text as input and produce meaningful text as output.) To obtain As a rule, pre-trained encoder transformers such as BERT are sufficient for embeddings. The problem is that the context of transformers (i.e., the number of words accepted as input before the neural network “forgets” what was at the beginning) is limited.
Obtaining embeddings is a time-consuming process that requires computing power. As a rule, transformers are used on video cards. Luckily, Google Colab allows you to use a certain number of T4 graphics cards per day. In addition, embeddings of all paired Gorky texts can be obtained in parts. One way or another, the problem of the limited context of transformer models will always be relevant due to their very neural network architecture; To make both attention and memory large, the language model must grow to enormous sizes (in terms of the number of parameters), just like modern models like GPT, the training of which requires a huge amount of information.
Before receiving embeddings, each text was divided into fragments of 1000 whitespace words with the intersection of the divided fragments. These fragments were then passed into the model. I tried embeddings of two models SBERT (Sentence-BERT, i.e. BERT for sentences) and YandexGPT Light. The latter predictably performed better. The larger the transformer model (it has seen more texts and has more parameters), the better the embeddings (including the longer the fractional part of the numbers in them). You can get YandexGPT embeddings by accessing the model via the API using the Langchain library for python, which, among other things, is designed to solve the problem of context length.
Another bottleneck, as it turned out after the fact, was the length of the texts fed into the model to obtain embeddings. As can be seen from the histograms at the beginning, in Gorky there are texts of different lengths - the longest are three or four times longer than the shortest. Technically, adding zeros to the embeddings of short texts up to the length of the longest texts (this procedure is called “paddig”) is not a problem. However, it turned out that when clustering, embeddings of short texts are always combined into one group (see below), despite the difference in the meanings of these texts. (So zeros went from being unimportant to being very important.) Compressing vectors to be smaller than the length (in whitespace) of the smallest text would mean losing a significant portion of the meaning of long texts. As a result, I left the short texts unchanged, and cut the long ones until the embeddings were approx. 4000 whitespace words (tokens). As a result, the embeddings turned out to be 1024 long.
I note that for an unknown reason (unknown error) YandexGPT refused to issue embeddings of the following nine texts (I will give the author’s full name and date of publication in Gorky in format): Chubarov I. 2021-07-19; Incognito 2020-10-13; Grand A. 2020-08-16; Incognito 2018-07-18; Martov I. 2017-09-25; Incognito 2016-11-30; Grand A. 2023-04-26; Lukoyanov E. 2019-02-27; Pishchikova E. 2018-07-18. I did not find anything in common between these texts that distinguishes them from the rest. YandexGPT has taboo topics that it refuses to talk about, but, as far as I know, there are no taboo topics for getting embeddings. There may have been errors during parsing that cannot be detected automatically (format checks and cleaning of unreadable characters) or detected visually. My apologies to the authors. One way or another, this is a drop in the ocean of the remaining 4243 texts.
After receiving the embeddings, I clustered them to combine the texts by meaning. Unlike the LDA model, which grouped by distributions of keywords, in this case we group by the meanings of texts, represented as numerical vectors with which mathematical operations can be performed.
There are many types of cluster analysis: cluster analysis using the k-means method, hierarchical clustering (agglomerative or divisional), density-based clustering (DBSCAN), etc. Different methods are applicable and perform better for different point distributions and give different results . I used k-means and agglomerative clustering (with cosine distance).
The k-means model, like LDA, requires you to indicate how many clusters the texts should be divided into. The optimal number of clusters must be selected iteratively, training models with an increasing number of clusters and focusing on quality metrics. (For K-Means there are three methods, metrics of obtaining the optimal number of clusters: the elbow method (minimization of inertia), silhouette method (maximization of the Silhouette_Score metric) and method based on GEP statistics.) These metrics often indicate in favor of different number of clusters as an optimal . So it turned out this time.