I was perusing the usual collage of semi-news, trivia, and fluff on CollingwoodToday recently and noticed a poll at the bottom asking “How old are you?” It piqued my interest enough to add my vote. I’m always interested in demographics and statistics, and am curious about changing media engagement, especially now that print media has been scuttled locally. Who is reading it online? And how might that reflect both our community’s demographics and political leanings?
But wait… is voting really that simple? Point and click? No screening questions? Nothing to qualify me, prevent me from voting more than once, determine where I live, anything to verify my status? Ask how often I visited the site? Nothing more to make it data-rich for further analysis? Nope. Just a single question anyone can answer. Okay: click. Now look at the results…
This type of survey is known as an “opt-in poll” or “open-access poll” about which Pew Research noted, “On average, error on opt-in samples was about twice that of probability-based panels… much of the error on the online opt-in samples is due to the presence of “bogus respondents,” who make little or no effort to answer questions truthfully.” Pew found the percentage of bogus respondents was fairly low in their study: “about 4% to 7%, depending on the source” and while it might not be terribly relevant to this particular poll, it could have a significant impact on others, such as political leanings polls.*
The website has a disclaimer: “The CollingwoodToday.ca poll is a sampling of public opinion intended solely to allow our readers to express themselves on issues of the day. Its findings may not be representative of the general population of Collingwood or other areas.” The latter sentence is quite the understatement. Without appropriate qualifiers, a poll cannot represent any geographic area. But can it actually represent the age groups themselves? Again no, since there is no way to verify the information or the respondents. (And asking someone’s age or employment status is not a “sampling of public opinion” because no opinion is being asked. But that’s just local media, eh? Imagine what they could be if they had an editor!)
The results page has the date the poll was added to the site, but not the date of the latest results. That’s a serious omission because a reader has no way of knowing from the results page if the poll is still open, if and when it closed, or the results are those from that one specified date (from what I can tell, many, possibly most, of these polls are still open, but responding depends on having it randomly appear on whatever page you happen on). Nor is there any indication of the frequency of responses during the open period (did most respond when it was first posted, spread over the entire time, at the end, or bunched in discrete segments that might be paired with particular dates or news items?). Again, imagine what local media could accomplish with the data if they had an editor!
The Canadian Press Stylebook (18th edition) has five pages on how media should treat polls. I realize it’s quite a stretch to expect local media to read let alone heed any style guide, but they make a point on page 131: “Public opinion polls are like snapshots: it’s not always obvious what they’re showing.” Making the data clear so that the audience knows how it was conducted, the full set of demographics behind it, and how respondents were qualified should be the responsibility of an editor. CP itself “limits its coverage to reputable polls dealing with topical issues.”
Plus there’s a whole discussion to be had about how questions are framed and by whom. CP notes that “Polls are often used by interest groups to support their causes.” And that is done through specific wording of the questions, “designed to persuade large numbers of people to a certain point of view, not to measure opinion.” These are called push polls, partisan polls, or message testing. I’ll leave this here, but I want to make a point that crafting the questions on a poll is crucial to the results and should not be done casually or by people unfamiliar with the science of polling.
I was curious if there was a method to prevent someone from voting multiple times, thus skewing the results and invalidating the poll. So I tried. It has, sort of… it won’t let you vote twice from the same device right away (using either a basic cookie or IP detection method), but I was able to open an “in-private” browser window — a feature of every modern browser — and vote twice. And vote on a different device. And vote again on the first device a day later without being blocked. I suspect clearing cookies or the cache, or using a VPN will also allow multiple votes. Again, it is not likely an issue with this particular poll, but the company should take better care to prevent such abuse in the future should they ask something relevant (and if you look through their poll archives, they sometimes do ask questions that have social or political relevance… but almost as often ask piffle like that shown above).
Unique Participant Fraud is defined as “Survey fraud that involves individuals who use the same identifying information in order to access a survey multiple times. This can occur either with the intention of accepting multiple incentives or for legitimate reasons (e.g. the participant was not sure that their first survey was correctly recorded, or they simply forgot they had already participated). These are submissions where it is clear that a single individual has submitted multiple responses.”
Opt-in polls like those on CollingwoodToday are thus merely entertainment with no credibility for data analysis or even for indicators of local demographics. They are akin to publishing horoscopes or reprinting government propaganda (both of which are done in CollingwoodToday).** But they often get treated otherwise by both the media and their audiences. The Association of Internet Research Specialists wrote back in 2015 what remains true today (emphasis added):
…online surveys can produce inaccurate, unreliable, and biased data. There are four main reasons for this: sample validity, non-response bias, stakeholder bias, and unverified respondents…
When online surveys are accessible to anyone who visits a website, the researcher has no control over sample selection. These self-selected opinion polls result in a sample of people who decide to take the survey — not a sample of scientifically selected respondents who represent the larger population. In this situation online survey results are biased because people who just happen to visit the website, people who are persuaded with a monetary or other incentive to sign up for the survey, people who have a vested interest in the survey results and want to influence them in a certain way, and people who are driven to the site by others are included in the sample. This results in a double bias, because this distortion is in addition to the basic sample already having excluded people who do not have Internet access.
I readily admit it is unlikely anyone would try to manipulate this particular poll since there is no obvious value in doing so outside of targeted advertising. It seems innocuous (outside the collection of your internet address, and possibly your browsing history by the apps on the website). But there are plenty of bad actors out there who take advantage of the easy access to online polls. Similarly weakly-designed polls have been abused to create a political or partisan environment for or against issues, candidates, and parties. This is a significant threat to democracy. There’s this thing called a cascade or bandwagon effect in which “popular options become more popular because of a poll” and another in which early respondents can set the tone for others.
CollingwoodToday has published many polls whose results could be manipulated by anyone looking to sway public opinion. For example here are some previous questions:
- Do you have an overall favourable or unfavourable view of Pierre Poilievre?
- The new passport design features images of nature and daily life, instead of Canadian history. What do you think of it?
- What party would you vote for if an Ontario provincial election was held today?
- Should doctors and healthcare companies be allowed to charge for appointments with general practitioners?
- If a federal election were held tomorrow, which one of the following parties would you vote for?
- In your view, what is the most important issue facing your municipality right now?
- How happy are you with the results of the municipal election?
- Do you know who you will be voting for in the upcoming municipal election?
- With the death of Queen Elizabeth II, do you think it’s time for Canada to cut ties with the monarchy?
- Do you agree with the federal government’s decision to drop COVID vaccine mandates for domestic and outbound international travellers?
- Are you pleased with the outcome of the provincial election?
These may be important questions, but with an opt-in poll easily manipulated by respondents or bots, and no qualifications for voting or even a way to determine the geographic origin of respondents, there is no way to determine what the results signify — if anything. Or if they have been gamed to produce a specific ideological target. They could be mostly Russian trolls skewing the results to aid Putin’s agenda. Or, in local instances, supporters or opponents of a particular project plumping the results by voting numerous times. How can readers tell? And how can they avoid being swayed by them? ***
As a Quirk magazine article about polling in the US noted (emphasis added),
There are many, though, who argue that political pollsters have become the handmaidens of their client-masters and that the results of their polls are either deliberately manipulated and/or misreported in order to help the client candidate attract more votes. If true, this would give new meaning to the poll as a “strategic political weapon,” as well as poking holes in the “myth” that pollsters are neutral, apolitical beings.
And it doesn’t have to be actual people doing it. Apps and bots can be programmed to sweep online sites to affect polls, posts, and discussions. The US elections in 2016 and 2020 were targets of intervention by foreign states looking to ensure Trump’s election then re-election; how well they accomplished this is still being debated but their efforts are well-documented. A 2023 story in SciTech Daily warned that “AI-Powered Bad Actors: A Looming Threat for 2024 and Beyond,” noting:
A study forecasts that by mid-2024, bad actors are expected to increasingly utilize AI in their daily activities… The authors predict that such bad-actor-AI attacks will occur almost daily by mid-2024—in time to affect U.S. and other global elections.
In Ontario, media constantly published polls predicting a CONservative win before the last provincial election. This firehose of polls may have created voter apathy and discouraged supporters of other parties from voting because they felt there was no point since the race had already been won. As a result, we saw the lowest voter turnout in our province’s history. The CONservatives won the majority with only 22% of the whole electorate despite a previous record of nepotism and scandals, but still taking only 41% of those votes cast. A similar tactic is being played out these days federally, with rightwing media (more than 90% of Canadian media is in CONServative ownership or control) constantly publishing polls showing support for the CPC and their far-right leader PoiLIEvre (aka Skinny Trump). Goebbels would have been proud.
Polls can be, and are being, used as ideological weapons. As a 2018 article in The Conversation noted (emphasis added):
But it isn’t just biased audiences that can skew online polls. Some are susceptible to manipulation by bots. Worse, there exists a black market for buying online votes: a vote can be bought for less than two roubles (£0.03) on Russian online marketplaces. This allows individuals or organisations to easily fudge poll outcomes that can be used to influence citizens’ voting decisions.
Manipulation of online surveys can be problematic in two ways. First, when many bots or paid respondents take part, it can change the results of the poll. Ideally, this dynamic can be tamed by robust technological authentication. But, more subtly, manipulation could also open the door to a phenomenon dubbed “cascades” by social scientists. By planting a few initial votes that make it look as if voters favour a particular outcome, manipulators could offset a dynamic that eventually shifts the poll result in the desired direction.
The bottom line here is to be wary of any online polls that are not created by reputable polling organizations. Even when the pollsters are reputable, they or their clients may have an ideological agenda in framing it in a particular way; consider both the client (who paid for the poll?) and how the results are presented in the media. As for opt-in polls, try not to treat them as any more relevant or informative than a crossword puzzle or cartoon.
Notes:
I previously wrote about online polls on this blog: Internet Surveys: Bad Data, Bad Science and Big Bias (2013) and Are Internet Polls Valid? (2012).
* Wikipedia also mentions “voodoo polls” (aka pseudo-polls) a term which it notes:
…is a pejorative description of an opinion poll with no statistical or scientific reliability, which is therefore not a good indicator of opinion on an issue. A voodoo poll will tend to involve self-selection, will be unrepresentative of the target population, and is often very easy to rig by those with a partisan interest in the results of the poll.
** Horoscopes are pseudoscience claptrap, yet many media still carry them to cater to an audience of gullibles, generally the same audience for celebrity gossip and anti-vaxxer propaganda. Richard Dawkins famously castigated newspapers for publishing horoscopes, saying “astrology is neither harmless nor fun, and that we should fight it seriously as an enemy of truth.” But in our Trumpian, post-truth world where leaders lie openly and brazenly, falsehoods have a lot more prominence in our media. Dawkins also wrote, “Astrology not only demeans astronomy, shrivelling and cheapening the universe with its pre-Copernican dabblings. It is also an insult to the science of psychology and the richness of human personality.”
As the Smithsonian Magazine noted, horoscopes have become …”along with standards like the crossword, newspaper ‘furniture’ … It’s an unusual form of language and form of relationship and as such, it lends itself well to a kind of attachment.” Emphasis added:
So what’s going on? Why are people willing to re-order their love lives, buy a lottery ticket, or a take a new job based on the advice of someone who knows nothing more about them than their birthdate?
One reason we can rule out is scientific validity. Of all the empirical tests that have been done on astrology, in all fields, says Dr. Chris French, a professor of psychology at London’s Goldsmith College who studies belief in the paranormal, “They are pretty uniformly bad news for astrologers.”
…in 2011, British rockstar physicist Brian Cox came under fire from astrologers for calling astrology a “load of rubbish” on his Wonders of the Solar System program on BBC. After the BBC fielded a bunch of complaints, Cox offered a statement, which the broadcaster probably wisely chose not to release: “I apologize to the astrology community for not making myself clear. I should have said that this new age drivel is undermining the very fabric of our civilization.”
As for government propaganda reprinted verbatim in local media without any effort to verify it or question its claims, I covered that previously: Why Local Media Has Failed Us.
*** Local readers may remember in 2012 when the late Enterprise-Bulletin newspaper used just such an unscientific, opt-in poll on its front page. It helped promote the partisan agenda of the Saundersonites who demanded our council give the local YMCA a $35 million handout. At the time, it was commonly said in the community that the poll results had been plumped up significantly by YMCA supporters voting multiple times. Ironically (or is it hypocritically?), the EB also published an editorial that year about its own use of opt-in polls in which it said, “We post those polls to gauge how the community feels about one issue or another, but otherwise there is little to no scientific basis to them.”
I’m not sure what method, if any, the Town of Collingwood uses to qualify survey participants, or who writes their questions. When I tried to access results from a previous survey, the secretive, user-hostile town’s inaptly named “engagement” website only gave me this: “CLOSED: This survey has concluded,” with no information about respondents, demographics, or results. But that would require town hall to be open and transparent, and have a proper communication policy. I expect neither will happen this term.
When I did find survey results (like this report), it didn’t provide more than the most basic demographic information (sex and age grouping) for respondents and not even confirming that respondents were actually Collingwood residents (a notable oversight that calls into question the credibility of the results).
Not only that, but it appears the survey was promoted only to a select audience, rather than the entire town: “The Town promoted the survey through 29 social media posts, articles in the Town’s newsfeed and newsletters and mentions of the survey in the Mayor’s radio interviews.” In other words, you would find out only if you had 1) a device with an internet connection and were online between May 21 and July 2, 2) social media accounts on those two platforms and were online on those platforms during those dates, 3) had joined, liked or befriended the town’s account in order to see their posts (which I assume were split between the two platforms, so perhaps 14 or 15 on each platform over 42 days), or 4) listened to the weekly local radio interviews during work hours and were able to write down the information for access later. I can recall no “newsfeed” arriving in my inbox at that time nor any printed “newsletter,” in my mail. Again, a flaccid and lazy communication policy. So it’s just another opt-in poll — yet the town used it to determine policy for an Official Plan. Collingwood deserves better.
Words: 3,051
