Short Answer: Probably more than 10% of users will cheat when filling out the questionnaire, but it will not affect the final results of the survey.

Online questionnaires are the most commonly used research method by user researchers. Many questionnaire platforms also provide various services to help people reach users and collect questionnaires more quickly. I am a fan of the cross-fill questionnaire function in Questionnaire Star, and I also rely on this function to collect and complete multiple surveys. However, instead of completing the collection of questionnaires, what we really care about is of course the reliability and validity of the survey results--whether the results displayed by these questionnaires really reflect what the users think and do.

In this article, I wish to discuss the relationship between online survey respondents and data quality.

How many people cheat when answering the questionnaire:

In a previous article, I briefly introduced several methods to prevent cheating in the questionnaire. Jeff Sauro summarized these methods and explored the data of 5 questionnaire surveys through ‘choice 3 of this question’ and found that: A total of 11.98% of users cheated in the survey. In the conference paper by Julie Down et al., they tested whether users carefully read the email content related to the question to determine whether users filled in the questionnaire carefully. The results found that: 38.94% of users were not serious when filling out the questionnaire.

Not long ago, I investigated the SUPR-Q scores of several questionnaire platforms. In the questionnaire, there is a question: ‘Which questionnaire platforms have you used:’. Since all the answer sheets are collected through Questionnaire Star's cross-fill questionnaire, I will treat the answer sheets that did not select 'Questionnaire Star' in this question as invalid questionnaires (the cross-fill questionnaire is a function that can only be participated in after Questionnaire Star has released the questionnaire) . The questionnaire collection was divided into three times. In total, about 15% of the respondents were not serious:

The impact of professional respondents on data quality:

In the above We know that although most professional respondents still answer the questionnaire carefully, there are always different proportions of 'cheaters' in each questionnaire survey. So, are the results of our questionnaire still showing the real situation? In the literature I have seen, exploration of this issue dates back to 1962. But here, I will briefly introduce two articles.

D. Sunshine Hillygus and others from Duke University reviewed many previous studies in "Online Panel Research: A Data Quality Perspective" and pointed out that there is no unified opinion on this issue in the academic community. : Some studies have found that professional fillers have faster completion times, and are more likely to choose randomly/no choice/choose 'don't know' and answer straight lines; at the same time, other studies have found that professional fillers' answers are more accurate than questionnaire results. does not constitute a significant impact. Through their own research, Hillygus et al. also found that professional respondents spent more time and effort filling out questionnaires, but at the same time they also skipped non-required questions more often, used the 'don't know' option, or used open-ended questions. Useless information is provided in the question.

Suzette Matthijsse and others from the Netherlands surveyed 9,461 users of 19 survey platforms in an attempt to explore the impact of professional fillers on questionnaire results. Its research found that professional test-takers are somewhat different from non-professional test-takers in social and psychological factors: they are more independent workers and thinkers rather than team workers and actors. But these characteristics do not mean that the responses of professional survey takers will affect the overall survey results.

Everyone’s situation is not exactly the same, and we cannot blindly trust other people’s research results. Fortunately, I was able to draw my own conclusions through my own data: in my previous research on music players, a small half of my data came from the spread of friends, and the other half came from mutual filling of questionnaires. For the former, I think their motivation for answering the questionnaire lies in interest, friend requests, etc., and define them as 'volunteers'. There are 184 people. For the latter, due to the incentive mechanism of filling out the questionnaire for each other, I think their answer The motivation mainly comes from interests, so they are defined as 'fillers', and there are 285 of them. Next, let’s compare the differences in data results between the two.

1. User attribute distribution

As shown in the figure, both among the respondents and volunteers, women account for a larger proportion, and their ages are concentrated between 18 and 25 years old. . The consistency in attributes between the two groups also facilitates further comparison of other data.

2. Brand awareness

This question asks about the music player brands that users know. As shown in the figure, although there are certain differences in values, there is no obvious difference between the respondents and the volunteers in terms of the overall trend.

3. Commonly used music players on PC and mobile terminals

This question asks users about the player brands they use most frequently on PC/mobile terminals. As shown in the figure, both among the survey respondents and volunteers, the most frequently used players are: Kugou, QQ and NetEase Cloud Music. Only on the mobile side, there are inconsistencies in the commonly used players among test takers and volunteer groups.

4. Net Promoter Score of PC and Mobile Music Players

As shown in the figure, there is a certain difference between the respondents and the volunteer group in terms of NPS scores. . Despite this, in terms of the overall trend, there is no obvious difference between the two groups: on the PC side, NetEase Cloud's NPS value is ahead of Kugou and QQ Music, with the latter two being relatively flat; on the mobile side, NetEase Cloud's NPS The net promoter score is far ahead of Kugou and QQ, but Kugou’s NPS value is still higher than QQ Music to a certain extent.

Summary: From the above data, it can be seen that the group differences between respondents and volunteers will not significantly affect the final results of the questionnaire. Some of the differences presented above are more likely due to the number of questionnaires returned rather than differences between the two groups. That is, although there are some cheaters among professional test takers, this does not mean that the test takers will change your survey results and draw wrong conclusions.

PS: Because I have completed many surveys using the cross-fill questionnaire function of Questionnaire Star, this comparison can be regarded as a reassurance for myself. But similarly, I still hope that everyone can be more cautious when using mutually filled questionnaires. If your questionnaire requires a specific user group to complete, please give up the mutually filled questionnaires.

Another PS: The fit between my circle of friends and the users of Questionnaire Star is so high~~~

Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor , L. F. (2010). Are your participants gaming the system? Screening Mechanical Turk Workers. International Conference on Human Factors in Computing Systems, CHI 2010, Atlanta, Georgia, Usa, April (p.2399-2402).

Matthijsse, S. M., De Leeuw, E. D., & Hox, J. J. (2015). Internet panels, professional visitors, and data quality . Methodology, 11(3), 81-88.

Hillygus, D. S., Jackson, N., & Young, M. K. (2014). Professional respondents in nonprobability online panels. Pediatria Polska(11-12) , 1351-1354.