University of Rochester (New York) researcher Ehsan Hoque, who studies the role of emotions in human-computer interaction, has developed a computerized communication assistant. The system monitors hand movements, gaze directions, nods, facial expressions and pauses in conversation, responding to signs of friendliness, intonation or excessive use of certain words. “Live interactive social skills assistance” (Lissa) appears to the interlocutor in the form of an avatar with a human face, its reactions are based on the reactions of real people assessing each other’s communication style.

The nonverbal component of our communication can be very informative. An ironclad masculine handshake signals a desire to assert oneself, eye contact that lasts a little longer than necessary signals a willingness to flirt, and an awkward silence after presenting your brilliant idea shows that the idea is not so good. Edward Sapir, the famous linguist and anthropologist who died in 1939, wrote: “We respond to gestures ... in accordance with an intricate and secret code, which is not written down anywhere, unknown to anyone, but understood by everyone.” Hawk is trying to write the most universal code for nonverbal communication possible.

Hock is one of MIT Technology Review's global rankings of the world's top young innovators under 35, a list that includes many notable innovators before him, such as Jonathan Ive, Apple's lead designer. According to Hawk, computers can help us become better versions of ourselves. Lissa supports aspiring speakers, shy people hesitant to get a date, or college graduates trying to impress employers. But this was not Hawk’s main goal when choosing the direction of development. The researcher's younger brother is a teenager with Down syndrome and limited speech skills. According to Hawk, it is precisely for such people, as well as for autistic people, that Lissa should be useful in the first place. Preliminary research has shown that people with communication difficulties feel more comfortable receiving ratings from a machine than from a human. The US military hopes that the interactive system will also help identify cases of inconsistency between human gestures and speech. Security forces are funding research into automatic detection of deception.

Involuntary gestures truly speak about emotions, even if we hide these emotions in words. If the program is able to interpret our gestures, then we can be compared to an open book that is not difficult to read. This book was written in the course of many thousands of years of evolution, one of the results of which was a universal set of non-verbal actions such as frowning eyebrows or flaring nostrils with negative emotions or an involuntary smile with positive ones. The classic expression of helplessness or indifference - the shrug - is found and clearly understood in almost all cultures.

Translated by Alexander Silonov

A program has been developed that can recognize non-verbal signs during communication