Back in 2012, I wrote a blog piece about internet polls and surveys, asking whether internet polls and surveys could be – or should be – considered valid or scientific. I concluded, after researching the question, that, since the vast majority lack any scientific basis and are created by amateurs – often with a goal to direct rather than measure public opinion – that,
Most internet polls are merely for entertainment purposes. They harmlessly allow us to believe we are being engaged and are participating in the process, and they make pollsters happy to receive the attention. They are, however, not appropriate tools for making political or social decisions unless they are backed by rigid, scientific and statistical constraints.*
Earlier that year, an editorial in a local newspaper wisely drew similar conclusions (emphasis added):
It’s said that there’s lies, damn lies, and statistics. You could also throw Internet polls into that mix. …
But anyone who takes the results to heart, or attributes any level of credibility to them, is horribly mistaken. We post those polls to gauge how the community feels about one issue or another, but otherwise there is little to no scientific basis to them.
And unlike a poll that’s usually conducted by the likes of Ipsos, Leger and Gallup – who use scientific principles to conduct many of their public-opinion polls – the results of an Internet poll can easily be ‘plumped’ by one side or the other enlisting family, friends and associates to vote, regardless of their level of understanding of an issue.
I wanted to follow up my earlier piece with some more information from the professionals in the polling and statistical analysis fields, as well as some journalistic comments. The reasons internet polls are unscientific and lack credibility has been addressed by many universities, professional polling companies and associations.
The National Council on Public Polls weighed in on this question, stating (emphasis added):
While different members of NCPP have different opinions about the potential validity and value of online surveys, there is a consensus that many web-based surveys are completely unreliable. Indeed, to describe them as “polls” is to misuse that term.
The NCPP then suggests a list of ten questions for journalists to ask to help clarify the results and the scientific methods used to create and assess the poll:
- Is the internet-based survey designed to be representative, and if so, of what population? If not, it is not worthy of being reported.
- What evidence is there that the sample is representative of the population it claims to represent? Unless the internet-based survey can provide clear evidence that the sample is representative by demographic and/or other relevant information it is not worthy of being reported.
- How was the sample drawn? Many internet-based surveys are just “call-in” polls or are asked only of people who happen to visit a particular web site. These surveys usually do not represent or make any pretense to represent any other population, and are not worthy of being reported.
- What steps does the organization take to prevent people from voting more than once? Any poll which allows people to vote twice, or more often, is not worthy of being reported.
- How were the data weighted? Survey data may contain biases from a variety of causes. The magnitude of these biases and random errors are usually unknown to the researcher. Even so, weighting may minimize these biases and errors when there is a strong relationship between the weighting variable and data in the survey. If there is not a strong relationship weighting may make the survey results worse. Demographic weighting of internet-based surveys is essential but is not sufficient. Some firms, in addition to demographic weighting, are weighting on other variables in an attempt to reduce the biases of online data.
- What is the evidence that the methodology works and produces accurate data? Unless the organization can provide the results of their other internet-based surveys which are consistent with other data, whether from the Census or other surveys, the survey results are not worthy of being reported.
- What is the organization’s experience and track record using internet-based polls? Unless the organization can demonstrate a track record of obtaining reliable data with other online surveys, their online surveys should be treated with great caution.
- What is the organization’s experience and track record as a survey researcher using traditional survey methods? If the organization does not have a track record in designing and conducting surveys using the telephone or in-person surveys, it is unlikely that they have the expertise to design and conduct online surveys.
- Does the organization follow the codes of conduct of AAPOR, CASRO, and NCPP (whether or not they are members)? If they follow none of these, they are probably not a qualified survey research organization. The more of these Codes they follow, the more likely their data are to be reliable and be trusted.
- Is the organization willing to disclose these questions and the methods used (as required by the codes of conduct referred to in #9 above)? If the organization is unwilling to disclose, or unable to provide, the relevant information the survey is probably not worthy of being reported.
The NCPP also has a list of 20 questions journalists should ask about all poll results. They state in the introduction (emphasis added):
The only polls that should be reported are “scientific” polls. A number of the questions here will help you decide whether or not a poll is a “scientific” one worthy of coverage – or an unscientific survey without value. Unscientific pseudo-polls are widespread and sometimes entertaining, but they never provide the kind of information that belongs in a serious report. Examples include 900-number call-in polls, man-on-the-street surveys, many Internet polls, shopping mall polls, and even the classic toilet tissue poll featuring pictures of the candidates on each roll.
One major distinguishing difference between scientific and unscientific polls is who picks the respondents for the survey. In a scientific poll, the pollster identifies and seeks out the people to be interviewed. In an unscientific poll, the respondents usually “volunteer” their opinions, selecting themselves for the poll.
The results of the well-conducted scientific poll provide a reliable guide to the opinions of many people in addition to those interviewed – even the opinions of all Americans. The results of an unscientific poll tell you nothing beyond simply what those respondents say.
With just over a year left to go in this term, I’d like to take a few minutes to consider all the accomplishments of this council over the past three years. They are not inconsiderable, and worth celebrating, I think you’ll agree.
Plus we can now host those aquatic competitions that are so important to the swimming community. That will prove a benefit to the whole community, economically and in public relations. The Collingwood Clippers held their open house at the new pool, Sept. 5, and presented the town with the anticipated donation cheque for the pool upgrades: $158,000. Thank you, Clippers!
The Province of Quebec is proposing to ban the wearing of any and all religious headgear (including hijabs, turbans and yarmulkes, as well as other religious symbols) from public facilities and public service – affecting teachers, hospital workers, daycare workers, nurses, civil servants and (we assume) politicians.
And what about displays of the Flying Spaghetti Monster? Evolvefish? Or the red atheist “A”? Atheist symbols aren’t religious, so would they be exempt under this new law? 
