{"id":31527,"date":"2024-03-05T17:36:51","date_gmt":"2024-03-05T22:36:51","guid":{"rendered":"https:\/\/ianchadwick.com\/blog\/?p=31527"},"modified":"2024-03-06T08:41:18","modified_gmt":"2024-03-06T13:41:18","slug":"opt-in-polls-bad-data-bad-science-and-bad-actors","status":"publish","type":"post","link":"https:\/\/ianchadwick.com\/blog\/opt-in-polls-bad-data-bad-science-and-bad-actors\/","title":{"rendered":"Opt-in Polls: Bad Data, Bad Science, and Bad Actors"},"content":{"rendered":"
<\/div>\n

<\/g><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

\"Voodoo<\/a>I<\/span> was perusing the usual collage of semi-news, trivia, and fluff on CollingwoodToday<\/a> recently and noticed a poll at the bottom asking “How old are you<\/a>?” It piqued my interest enough to add my vote. I’m always interested in demographics and statistics, and am curious about changing media engagement, especially now that print media has been scuttled locally. Who is reading it online? And how might that reflect both our community’s demographics and political leanings?<\/p>\n

But wait… is voting really<\/em> that simple? Point and click? No screening questions<\/a>? Nothing to qualify me, prevent me from voting more than once, determine where I live, anything to verify my status? Ask how often I visited the site? Nothing more to make it data-rich for further analysis? Nope. Just a single question anyone can answer. Okay: click. Now look at the results…<\/p>\n

This type of survey is known as an “opt-in poll” or “open-access poll<\/a>” about which Pew Research noted<\/a>, “On average, error on opt-in samples was about twice that of probability-based panels… much of the error on the online opt-in samples is due to the presence of \u201cbogus respondents,\u201d who make little or no effort to answer questions truthfully.” Pew found the percentage of bogus respondents<\/a> was fairly low in their study: “about 4% to 7%, depending on the source” and while it might not be terribly relevant to this particular poll, it could have a significant impact on others, such as political leanings polls.*<\/strong><\/span><\/p>\n

\"Irrelevant<\/a>The website has a disclaimer: “The CollingwoodToday.ca poll is a sampling of public opinion intended solely to allow our readers to express themselves on issues of the day. Its findings may not be representative of the general population of Collingwood or other areas.” The latter sentence is quite the understatement. Without appropriate qualifiers, a poll cannot<\/em> represent any geographic area. But can it actually represent the age groups themselves? Again no, since there is no way to verify the information or the respondents. (And asking someone’s age or employment status is not a “sampling of public opinion” because no opinion is being asked. But that’s just local media, eh? Imagine what they could be if they had an editor!)<\/p>\n

The results page has the date the poll was added to the site, but not<\/em> the date of the latest results. That’s a serious omission because a reader has no way of knowing from the results page if the poll is still open, if and when it closed, or the results are those from that one specified date (from what I can tell, many, possibly most, of these polls are still open, but responding depends on having it randomly appear on whatever page you happen on). Nor is there any indication of the frequency of responses during the open period (did most respond when it was first posted, spread over the entire time, at the end, or bunched in discrete segments that might be paired with particular dates or news items?). Again, imagine what local media could accomplish with the data if they had an editor!<\/p>\n

The Canadian Press Stylebook (18th edition) has five pages on how media should treat polls. I realize it’s quite a stretch to expect local media to read let alone heed any<\/em> style guide, but they make a point on page 131: “Public opinion polls are like snapshots: it’s not always obvious what they’re showing.” Making the data clear so that the audience knows how it was conducted, the full set of demographics behind it, and how respondents were qualified should be the responsibility of an editor. CP itself “limits its coverage to reputable polls dealing with topical issues.”<\/p>\n

Plus there’s a whole discussion to be had about how questions are framed and by whom. CP notes that “Polls are often used by interest groups to support their causes.” And that is done through specific wording of the questions, “designed to persuade large numbers of people to a certain point of view, not to measure opinion.” These are called push polls, partisan polls, or message testing. I’ll leave this here, but I want to make a point that crafting the questions on a poll is crucial to the results and should not be done casually or by people unfamiliar with the science of polling.<\/p>\n

\"Nobody<\/a>I was curious if there was a method to prevent someone from voting multiple times, thus skewing the results and invalidating the poll. So I tried. It has, sort of… it won’t let you vote twice from the same device right away (using either a basic cookie or IP detection method), but I was able to open an “in-private” browser window \u2014 a feature of every modern browser \u2014 and vote twice. And vote on a different device. And vote again on the first device a day later without being blocked. I suspect clearing cookies or the cache, or using a VPN will also allow multiple votes. Again, it is not likely an issue with this particular poll, but the company should take better care to prevent such abuse in the future should they ask something relevant (and if you look through their poll archives<\/a>, they sometimes do ask questions that have social or political relevance… but almost as often ask piffle like that shown above).<\/p>\n

Unique Participant Fraud<\/a> is defined as “Survey fraud that involves individuals who use the same identifying information in order to access a survey multiple times. This can occur either with the intention of accepting multiple incentives or for legitimate reasons (e.g. the participant was not sure that their first survey was correctly recorded, or they simply forgot they had already participated). These are submissions where it is clear that a single individual has submitted multiple responses.”<\/p>\n

Opt-in polls like those on CollingwoodToday are thus merely entertainment with no credibility for data analysis or even for indicators of local demographics. They are akin to publishing horoscopes or reprinting government propaganda (both of which are done in CollingwoodToday).**<\/strong><\/span> But they often get treated otherwise by both the media and their audiences. The Association of Internet Research Specialists<\/a> wrote back in 2015 what remains true today (emphasis added):<\/p>\n

…online surveys can produce inaccurate, unreliable, and biased data. There are four main reasons for this: sample validity, non-response bias, stakeholder bias, and unverified respondents<\/strong>…
\nWhen online surveys are accessible to anyone who visits a website, the researcher has no control over sample selection. These self-selected opinion polls result in a sample of people who decide to take the survey — not a sample of scientifically selected respondents who represent the larger population. In this situation online survey results are biased because people who just happen to visit the website, people who are persuaded with a monetary or other incentive to sign up for the survey, people who have a vested interest in the survey results and want to influence them in a certain way, and people who are driven to the site by others are included in the sample. This results in a double bias, because this distortion is in addition to the basic sample already having excluded people who do not have Internet access.<\/p><\/blockquote>\n

I readily admit it is unlikely anyone would try to manipulate this particular poll since there is no obvious value in doing so outside of targeted advertising. It seems<\/em> innocuous (outside the collection of your internet address, and possibly your browsing history by the apps on the website). But there are plenty of bad actors out there who take advantage of the easy access to online polls. Similarly weakly-designed polls have been abused to create a political or partisan environment for or against issues, candidates, and parties. This is a significant threat to democracy. There’s this thing called a cascade or bandwagon effect<\/a> in which “popular options become more popular because of a poll” and another in which early respondents can set the tone for others.<\/p>\n

CollingwoodToday has published many polls<\/a> whose results could be manipulated by anyone looking to sway public opinion. For example here are some previous questions:<\/p>\n