"Non -toxic algorithm." Interview with the participants of the Khakaton "Flight 2021" on how to cure sexism of translators and chatbots

The word “science” is closer to the word “man” than to the word “woman”, and by the way “woman” is close to the word “helplessness”. This is not a linguistic fact, but the conclusion that algorithms make, studying on historically accumulated language data. Then gender stereotypes are reproduced in automatic translators, chatbots and, for example, in the analysis of your resume.

It is generally accepted that the best protection against toxicity for the algorithm is training in neutral data sets in which there are no stereotypes. But finding such data is almost impossible. Participants in the Hakaton of the Teplitsa and Novaya Gazeta “Flight 2021” Mikhail Kuznetsov, Artyom Stay and Timofei Korzhebin proposed another, simpler and more universal approach to “treatment”, or rather, “upbringing” of algorithms. Olga Tarakanova asked Mikhail Kuznetsova, the leader of the team, how their “non -toxic algorithm” works, what the project has prospects and restrictions.

- Michael, did you manage to create a non -toxic algorithm on a hackathon?

-We did not believe that you can create a gender-neutral language model, teaching it on Leo Tolstoy’s novels. We thought that a gender-neutral set of data was needed. We were looking for medical encyclopedias, base of scientific articles, consulted with one of the best specialists in the processing of the natural language in the NLP in English telegram community. But, studying on neutral data, the model still issued stereotypes. It was an insight for us: bias exists, even when we do not notice them.

Mikhail Kuznetsov (right), leader of the Non -toxic algorithm team. Photo: From the archive of Mikhail.

By the end of the first day of Hakaton, we decided to improve the regulatory function. They spent the whole night on writing code, did not sleep from Saturday to Sunday, but by the morning they got the first technical result. Why on Tolstoy? This is a classic not only of Russian and world literature, but also machine learning - a thick book in which the text is enough to train a full -fledged model. And if our function coped with such stereotypical text, then it should work on others.

- Tell us how and why algorithms absorb stereotypes existing in society?

- Algorithms are studied on datasets, which are considered grammatically correct and from which you can draw out semantic connections between words. There are several approaches to processing a natural language using machine learning. I'll tell you about Word2VEC that we used.

Word2VEC model actually represents a dictionary in which each word is a vector. Vector is a mathematical concept, a n-dimensional set of numbers. For example, two -dimensional. Imagine that we have a coordinate plane, like at school. The vector is two numbers, coordinate along the X axis and coordinate along the Y axis. The vector of the word is the same set of numbers, only in it not two numbers, but, say, 200, as in our algorithm.

Between words in vector form, you can find a distance. A good example, which is always given at introductory lectures on NLP (Natural Language Processing, processing a natural language): take the word Paris, subtract France from it, add Britain and get London. That is, we deduced the coloring of the country, received the general concept of "capital", added another country and received a vector of another capital.

By installing the distances between words, you can ask for a model to analyze or finish the sentences. The words that stand nearby in the sentence are usually close to each other and in the spaces of the dictionary.

Sexism of algorithms in the real world-issuing Google on request CEO (general director, but in English it is a grammatically gender-neutral word). The screenshot of the author.

The problem is that sometimes such an analysis makes the algorithm draw inadequate conclusions. A bright case-an HR robot from Amazon, which hired mainly men [and also automatically rejected the resume in which the word "female" appeared. - approx. FROM.]. A more everyday example is an automatic translation in which the word Physicist from English into Russian will most likely be translated as “physicist” and “he”. In historical sets of data on which translators studied, words such as “physicist”, “engineer”, “science” are closer to the word “man”.

- What are the approaches to cleaning algorithms from stereotypes? What is the difference between the one that you suggested?

-In order to train the model to do something useful, it is necessary to clean and bring the data into a normal form. I will give an interesting case with a Microsoft bot, which was released to study raw data from Twitter (Microsoft computing power allows). A few hours later, this model turned into someone like a postpubert fascist teenager who ran into everyone in a row; Microsoft had to apologize. Or a hypothetical example: if we taught automatic translators on transfers in which non -normative vocabulary is used, they themselves would have given out the texts in much more, let's say, free form. That is, the training of algorithms on Wild Data, on raw data, can lead to the reproduction of not only stereotypes, but also other language habits.

To explain how to prevent stereotypes in algorithms, I would like to draw an analogy, technically not in all aspects of the correct, but working one. The machine learning model can be compared in something with the child. When we are talking about “cleaning”, it looks like this idea: in order to educate a child well, we will turn a blind eye to all the bad and say that this does not exist.

Our approach is that we show: yes, something bad really exists, but it is better not to do it. Maybe Leo Tolstoy has a more fragile creature, but in our time we understand that this is a stereotype. Technically, what we have developed is called a penalty function, or regulatory function.

Slide from the presentation of the “Non -toxic Algorithm” project demonstrates the proximity between different words (left) to the words “man” and “woman” before and after the work of the regulatory function.

- How did you determine, for the proximity between which words you need to fine the model, and what words should remain nearby?

- We do not consider ourselves specialists in ethics and therefore we cannot say which words should really be considered genderly painted and which are not. I can give an example with medical terms that describe, say, secondary sexual characteristics. On the one hand, it is logical to say that they are genderly painted, on the other, it is just scientific terminology.

As part of the experiment on Khakaton, we considered the conceptual words genderly painted, that is, those that are difficult to determine without addressing the gender attribute: father, mother, daughter, son, actor, actress. In the future, we plan to consult with specialists from areas of ethics, sociology, linguistics. In large companies that work in the field of machine learning, there are certainly special ethics departments.

- That is, you took advantage of the logic from the contrary: if the word is not conceptual, then it should not be close to the word “man” or the word “woman”.

- Yes.

- There are contexts in which gender -colored connections between words are appropriate. For example, a female letter is literary texts written by women and comprehending specifically female experience. In them, perhaps, there will even be the connection “woman - kitchen” or “woman - object”, but this connection will be the subject of criticism. And there will also be much less obvious and therefore valuable and interesting associations. How much is your project open to the involvement of specific data sets and experts - that is, what is possible at the technical level? And what are your own plans for the development of a “non -toxic algorithm”?

- This is a really complex comprehensive question - how to formalize concepts like “experience”. But I believe that this is possible if you complicate the penalty function and the logic of its work. As specifically, it’s hard to say so far - consultations with specialists are needed both from the areas of ethics and sociology, and in machine learning.

Now we plan to write a scientific article in the field of machine learning about the function that we proposed on Hakaton. We also want to check whether a penalty function will work if you use other approaches to machine learning, not only Word2VEC. Word2VEC is good to check the hypothesis. It does not require large computing power and dataset.

During Hakaton, we ourselves were not sure that our function would work. We assumed that an attempt to adjust the space of meanings during training would cause harm to the entire model and that it still needs to be trained at gender-neutral dataset. But in the end, it was possible to think about the function in such a way that the model showed non -stereotypical results after training at Anna Karenina and War and Peace, which, to put it mildly, you can’t call the texts unbiased by gender.

Artyom cut (left), Timofey Korzhebin (right), members of the Non -toxic Algorithm team. Photo: Victor Gokhovich.

Nevertheless, so far we do not know how our function behaves with other technical implementations of machine learning, for example, with Transformer architecture. The ability to apply a function with different approaches is important, because in the field of machine learning everything develops so swift pace that the methods that were used two years ago are already considered outdated.

-Since the problem of the gender balance is in the IT industry not only at the algorithm level, but also at the level of people, I’ll ask: do your team have women?

- There were no women in our team. We formed this team for a long time: the “spotlight” was our first adult Hakaton (we are all 18 years old), but we participated in many school, Gikov hacatons. In this sense, we may be closed and conservative. We rallied and have been walking since then.

But there is an interesting moment. I have a friend, Anastasia Gisina , who is a member of the Russian Hack Team team, until 18 years old she participated in dozens of hackathons and most of them won. When I saw Nastya in a “spotlight” chat and thought that she would participate on a par with us, he immediately decided: “We could not win.” And also to our group - I am studying in MEFI in the direction of "Software Engineering" - a girl, Polina Belova, who makes projects with neururalates, recently, and we began to actively communicate. I want to involve her in our work, call Hakatons. It is always nice to communicate with ideological and interesting people.