Crimean survey and unsatisfactory data quality

We publish a full version of the commentary of Alexei Kupriyanov , associate professor of the Department of Sociology of the St. Petersburg branch of the Higher School of Economics, dedicated to the context and analysis of the VTsIOM survey (12/31/15/01.16). In the paper and PDF version of the newspaper, an abbreviated version of this article is published.

The least of all I would like my comment to be perceived as digging in technical details. My common opinion consisted and consists in the fact that not a single survey company should agree to conduct this survey for a number of reasons.

Firstly, because, in fact, we are not dealing with a survey of public opinion (no matter how controversial the concept of public opinion is), but with a demonstrative distribution of responsibility for the political adventures of the Russian government on the population of the Crimean Peninsula. Secondly, because the formulations of questions were clearly manipulative and provocative (including criminal liability for one of the answer options).

Thirdly, because in conditions when professional autonomy is suppressed to such an extent that it is impossible to adjust the wording, nor expanding the list of questions in reasonably (for example, the most obvious questions “did you have encountered electricity shutdown?”, “Do you have power supply right now?” You haven’t been asked), it’s not clear at all, or how to conduct it.

In addition, however, there are also technical issues regarding the procedures for collecting and analyzing data. They must also be discussed. To a large extent, because, judging by the discussions that have recently been on the open and closed areas of the Internet, it seems to many that everything is safe here (yes, questions are not formulated in the best way, yes, the political order is obvious, but at least the “craft” component is honestly worked out).

My note is that these expectations, apparently, are not justified. In order to understand why, you need to carefully look at the results of the survey.

Turning to the analysis of the results, first of all, I would like to thank the VTsIOM employees and personally Valery Fedorov for the fact that the data array ( DatAST ) was posted on the overall view on the VTsIOM website and for its willingness to supplement it with a number of variables at the requests of sociologists (we hope that the updated expanded Dataset will answer several more questions).

The data array includes 3025 records containing 14 variables (basic: settlement / type of settlement, gender, age up to a year, level of education, answers to the first and second questions, age up to the age cohort, beginning, end and duration of the interview with an accuracy of a second, region, date of the response coefficient).

I'll start with the obvious. In fact, we are dealing with a selective research design (and, therefore, an attempt to evaluate the parameters of the entire population of the peninsula according to the sample parameters). Everyone who is familiar with this type of research design knows that it would be desirable for the sample to successfully simulate the population (correctly represented it) according to the studied parameters.

The problem is that the distribution of the studied parameters in the population is usually unknown. Theoretically, it can be expected that the desired can be achieved using a randomized (random) sample. In what sense should it be accidental? The selection criterion should not at least give rise to a systematic error regarding the studied parameter. Working blindly, they use various randomization procedures (“disobedience”) sample, which is written in sufficient detail in each textbook on research methods, no matter what subject areas they concern.

The rather strict application of these procedures gives hope that we did not make a systematic error at the stage of data collection and we will not get a significant displacement of assessments of the parameters of the population. However, there can never be complete confidence. That is why thorough descriptions of the methods that give hope for identifying sources of systematic error in further research are unusually important.

Our position is somewhat simpler when we know some parameters of the population. In such cases, we can check the success of randomization by how well the sample models the population according to these known parameters (this is especially important if we suspect that these parameters can be somehow related to our “unknown”).

Of the VTsIOM survey parameters at our disposal, we can focus on socio-demographic (gender, age, education) and geographical (individualized large settlement, for example, Sevastopol or Simferopol, or type of settlement). Fortunately, the socio-demographic parameters and the distribution of residents by settlements for the population of Crimea are more or less known, thanks to the census of the population last autumn 2014.

Express testing showed that the VTsIOM sample differs significantly from the census data in a number of parameters. In short, these differences come down to the following: in the VTsIOM sample, older age and women are excessively represented and youth / average age and men are not represented (see Table 1–2 and Fig. 1–2). This in itself can only be evidence of the systematic error of the telephone survey itself as a method of achieving respondents (which can hardly please, but at least not catastrophically). However, in addition, a number of oddities are noteworthy.

When analyzing, in addition to the “large” displacements described in favor of women and older ages and small peaks at a multiple of five values ​​of age in years, abnormal peaks at the age of 24 and 57–59 years attract attention. You can express various assumptions about what they are determined (for example, the interview technique, when the interviewer arbitrarily attributes some age to the respondent with an evasive answer), but it is impossible to check them without additional data.

Table 1 . A comparison of the structure of the population according to the 2014 census and the VTsIOM survey structure: the floor ratio (X-Squared = 91,248; DF = 1; P-Value <2.2*10-16).

Table. 2 . A comparison of the population structure according to the 2014 census and the VTsIOM survey structure: the ratio of age cohorts used by VTsIOM analysts when summing up the results (X-Squared = 318.82; DF = 4; P-Value <2.2*10-16).

One can say one thing. An attempt to localize these anomalies quickly led to an unexpected and not very pleasant result. These peaks arise only in the part of the sample that was collected on January 1, 2016, and they are noticeable in all types of settlements.

The sexual age structure of the part of the sample collected on December 31, 2015 is more homogeneous. Being half as much as possible, this part, although it does not correspond to the data on the sexual age structure, is at least devoid of unnatural extensive failures and peaks characteristic of the age structure of the sample on January 1 (see Fig. 3).

This “discovery” is important in the context of the details of the survey that already disclosed by the VTsIOM employees. The fact is that, according to VTsIOM representatives, the second part of the sample was formed after it became clear that the first thousand respondents were scored on the 31st. Moreover, if we compare the first and second day of the survey on the age cohorts used by the VTsIOM to present aggregated data, it turns out that the submission of these two days at a high level of significance differs from each other.

On the first day, the second and fifth cohorts (25–34 years and 45–59 years) were excessively represented, the first and fourth (18-24 years and 60 or more years or more) were not represented, and only with the third (35–44 years) there are no problems (see Fig. 4 and Table 3).

Table. 3 . Comparison of samples for December 31, 2015 and January 1, 2016 by age cohorts used by VTsIOM analysts when summing up the results (X-Squared = 59.56; DF = 4; P-Value = 3.589*10-12).

In fact, this suggests that we are unlikely to consider the submission of December 31, 2015 and on January 1, 2016 as part of one sample . Prior to the emergence of additional information about the course of the survey, it is difficult to say what this effect can be specifically related to, however, the “torn” distribution by age with twice the number of respondents on January 1 definitely indicates a violation of randomization and/or interviewing procedures.

This, in turn, doubts the possibility of using data in the analysis of data for this day as a whole or at least a narrowing of the possibilities for analysis in view of the need to exclude a number of variables.

Speaking without abysses, this means that if the data for December 31 can somehow be analyzed, then the data on January 1 is collected so poorly that the use of them in the analysis practically does not make sense (this largely depreciates the subtle observations of Kirill Kalinin in the title article, although, in paradoxically, it is compatible with some of its general conclusions, since it revealed a change in the results of the survey from the first on the second day).

A separate discussion deserves the problem of a different kind - what remained hidden from us as a result of the fact that the sample has such serious displacements? VTsIOM analysts suggested that it is enough to adjust the weight of each response using the coefficient taking into account the gender, age and settlement. However, this is true only if we also assume that these strata are quite homogeneous and successfully represented in the sample (in addition, the factor of the survey date was not taken into account in the coefficients, which, as we already know, had a significant impact on a number of parameters).

This aspect is all the more difficult to analyze until the data on the underpass have been received (it was not possible to get through to the subscriber) and about the refusals of the interview. From fragmentary information, it can already be assumed that information about what is due to the inaccessible (the line is busy, the subscriber is not responded, the mobile subscriber is disabled or is outside the network of network).

Due to the fact that a significant part of modern stationary phones requires additional power from the network, and mobile devices require recharging, the sample could be shifted in favor of people outside the electricity shutdown zones. Apparently, no information was recorded by the interviews that refused to interviews (I would like to hope that this problem can be solved at least for stationary phones).

The inability to control at least one of the parameters (geography, gender) of the “refuseniks” does not allow in any way to characterize part of the population, which explicitly refused to participate in the survey (and evaluate the contribution of “failures” to the selection of the sample). The problem can only be solved by the disclosure of additional data on the Crimean survey and similar data on other surveys, but at the moment we do not have it.

During the discussion, various hypotheses were expressed, including that the survey was not conducted at all, and its results were completely falsified. The most radical critics must disappoint. The survey, apparently, was conducted, although a number of implaced moments remain in his organization. At the same time, judging by the state of the collected data, the quality of the survey leaves much to be desired. The systematic differences in the parameters of the samples on December 31 and January 1, by their nature, cannot be explained by the causes on the side of the respondents and talk about serious problems with the design of the study.

During the discussion, it was repeatedly expressed that the Crimean survey did not bring us anything new (as a result of the survey, a high level of support for the actions of the Government of the Russian Federation was confirmed), since according to the results of previous studies there is no doubt that the level of declared support should be high. However, I absolutely do not understand how, with such a negligence to the design of the research that brings through the massif of the VTsome, you can be sure that we can generally say that something can or cannot surprise us.

Perhaps this negligence may be due to the difference in the tasks of workers in questionnaires and sociologists. The first, especially in such obviously “custom -made” studies, are only interested in the approximate correspondence to the desired result, for the sake of which you can neglect flaws in the design of the study. The second should be interested in the “actual state of affairs”, which can be clarified only as a result of an impeccable following the research procedure and constant critical rechecking of the hypotheses, including the quality of the collected data.

So, the discussions indicated to me that questionnaires deliberately avoid the formation of randomized samples, since it is a laborious process, therefore, various kinds of non-accident and post hoc are used calibration coefficients to approximately assessing the situation. However, this is good when we have independent assessment methods that we can calibrate the method.

At the same time, the story with election predictions, which, being repeatedly “calibrated” based on the materials of the massively focused elections, turned into a tool to adjust uncomfortable data under obviously false result (which was not given the “counting honestly” team without massive falsifications, and the survey industry was still fresh. I have to recognize her defeat by “missing” dozens of interest points). This deliberate loss of connection with reality seems to me quite dangerous, since in the current crisis situation you can result in a social explosion of serious power.

The main thing is that as a result of a more carefully conducted study, the anti-government position could gain or lose several percentage points, but that the question is forced to use the agenda: is it possible to use the data collected in this way for any reasonable conclusions, except for the unsatisfactory quality of the data.

Note 1. While I wrote this text, additional details were found out. According to VTsIOM employees, after several hours of the survey, they changed the range of response options offered to respondents, removing the option “I find it difficult to answer from it.” The result of this was a phenomenon quite interesting from the point of view of social psychology, the joy of the discovery of which is lubricated by the fact that the data is collected so casually that it will be difficult to correctly describe it.

As soon as the respondents stopped reading the option “I find it difficult to answer”, and the share of people who choose the “government-notable” answer option fell. This may mean that not only the category of citizens who previously hesitated in choosing an answer, but also some part of the former “opposition”, joined the stan of “conformers”. If this is true, then this gives a fairly clear example of the influence of specific formulations and ways to “close” on the results of the survey. The unpleasant side of this effect is that again, as a result, we will learn more about the tool than that we hoped to measure it with it.

Note 2. For those who wish, a script (*.ZIP) is attached to the article , written for work in the environment of statistical programming and analysis of data R, with the help of which all the calculations and illustrations necessary for writing this text were performed. On its basis, any person who owns R can reproduce and continue the analysis independently. Для работы с ним необходимо также скачать с сайта ВЦИОМ архив с данными (см. http://wciom.ru/index.php?id=237&uid=115540 ), распаковать его и переименовать извлеченный из архива файл в crimea.sav