Python. Library Natasha

We extract first and last names from texts in Russian using programming

Recently, Important Stories, together with Meduza, published an investigation into who helps police officers falsify drug cases. The so-called regular witnesses help them with this. People who, as a rule, know the police or are dependent on them, because they themselves have already been convicted or were detained the day before, and the threat of a new prison sentence hangs over them. Instead of honest testimony, such witnesses may simply confirm everything that the police officers say. And more than once.

How to find such witnesses? They usually speak in court or their testimony is read out there. Therefore, you can look in the court verdicts. We downloaded all the documents on all drug charges from the Moscow City Court website. Next, we needed to extract all the names from their texts and find those that appear not only in one case, but in two or more.

And today I’ll tell you exactly about that part of the work where we extracted names from the text. To do this, we used an excellent library called Natasha , which was worked on by Alexander Kukushkin’s Data Analysis Laboratory.

Video: Gleb Limansky

The library solves all the basic problems of natural Russian language processing: segmentation into tokens and sentences, morphological and syntactic analysis, lemmatization, extraction of named entities.

Let's use one sentence as an example to see how this works. The text of the verdict can be downloaded here .

First, let's export the library and write all the necessary lines for further text processing.

And upload our text of the verdict.

In this verdict there are quite a few different names on which you can check how the library is coping. First, let's transfer the text to our library. Let's call this object the variable doc. And we will perform all possible actions with this variable that the library is capable of.

And then we can check how the library copes with segmentation. To do this, let's check how the text beats into tokens. And we'll ask you to print out the first five.

And let's see how the library breaks the text into sentences. Let's check the first five.

This library can also do morphological parsing of words. This can be useful if, for some task, you need to isolate and work only, for example, with adjectives or nouns. Let's look again at the analysis of the first five words.

As you can see, the library gives a complete analysis. For each word we see the part of speech, and also, for example, in what number, gender, etc. this word is used.

This library can also normalize words - bring them to the correct form. And not only words, but also entire phrases, which we usually perceive as a whole entity. Let's look at a small sample.

As you can see, the library perceives, for example, “Russian Federation” not as two separate words “Russian” and “Federation”, but as a whole entity. And also knows how to bring this to the correct form - “Russian Federation”. And the same thing happens with names. The library can recognize their genus and reduce it to the initial form.

But if you need to reduce each word to its initial form, that is, lemmatize it, then you can use the lemmatize command.

The Natasha library can also extract dates.

This is how, for example, you get dates from a verdict. But you can put this into a more convenient form using f-strings.

What names! The library can extract names in different spellings from texts. In sentences, it is often the surname and initials.

As you can see, each part of the name is recognized separately: there is a surname, a first name and a patronymic. Or you can take the recognized entity as a whole. In our case, this was necessary. Therefore, using the keys() command, we can get all the keys of the dictionary. And then make a list from it.

Yes, unnecessary elements are also recognized along the way. But all this can be easily cleared out at the data cleaning stage. The main thing is that nothing is lost in this way.

At this point, the stage of extracting names from the texts of sentences was completed. Then all that remained was to check whether these names were also found in other cases.

You can find the workbook for this lesson here . And if you have any questions, ask them here .

Support those
who is telling the truth! You can become our donor by making a one-time or monthly donation. We want to depend only on you - those who want to know the truth, those who are not afraid to be free. Help us tell you the Stories That Matter.Support Important Stories