10/6/13

Internet Surveys: Bad Data, Bad Science and Big Bias


Falacious reasoningBack in 2012, I wrote a blog piece about internet polls and surveys, asking whether internet polls and surveys could be – or should be – considered valid or scientific. I concluded, after researching the question, that, since the vast majority lack any scientific basis and are created by amateurs – often with a goal to direct rather than measure public opinion – that,

Most internet polls are merely for entertainment purposes. They harmlessly allow us to believe we are being engaged and are participating in the process, and they make pollsters happy to receive the attention. They are, however, not appropriate tools for making political or social decisions unless they are backed by rigid, scientific and statistical constraints.*

Earlier that year, an editorial in a local newspaper wisely drew similar conclusions (emphasis added):

It’s said that there’s lies, damn lies, and statistics. You could also throw Internet polls into that mix. …

But anyone who takes the results to heart, or attributes any level of credibility to them, is horribly mistaken. We post those polls to gauge how the community feels about one issue or another, but otherwise there is little to no scientific basis to them.

And unlike a poll that’s usually conducted by the likes of Ipsos, Leger and Gallup – who use scientific principles to conduct many of their public-opinion polls – the results of an Internet poll can easily be ‘plumped’ by one side or the other enlisting family, friends and associates to vote, regardless of their level of understanding of an issue.

I wanted to follow up my earlier piece with some more information from the professionals in the polling and statistical analysis fields, as well as some journalistic comments. The reasons internet polls are unscientific and lack credibility has been addressed by many universities, professional polling companies and associations.

The National Council on Public Polls weighed in on this question, stating (emphasis added):

While different members of NCPP have different opinions about the potential validity and value of online surveys, there is a consensus that many web-based surveys are completely unreliable. Indeed, to describe them as “polls” is to misuse that term.

The NCPP then suggests a list of ten questions for journalists to ask to help clarify the results and the scientific methods used to create and assess the poll:

  1. Is the internet-based survey designed to be representative, and if so, of what population? If not, it is not worthy of being reported.
  2.  What evidence is there that the sample is representative of the population it claims to represent? Unless the internet-based survey can provide clear evidence that the sample is representative by demographic and/or other relevant information it is not worthy of being reported.
  3. How was the sample drawn? Many internet-based surveys are just “call-in” polls or are asked only of people who happen to visit a particular web site. These surveys usually do not represent or make any pretense to represent any other population, and are not worthy of being reported.
  4. What steps does the organization take to prevent people from voting more than once? Any poll which allows people to vote twice, or more often, is not worthy of being reported.
  5. How were the data weighted? Survey data may contain biases from a variety of causes. The magnitude of these biases and random errors are usually unknown to the researcher. Even so, weighting may minimize these biases and errors when there is a strong relationship between the weighting variable and data in the survey. If there is not a strong relationship weighting may make the survey results worse. Demographic weighting of internet-based surveys is essential but is not sufficient. Some firms, in addition to demographic weighting, are weighting on other variables in an attempt to reduce the biases of online data.
  6. What is the evidence that the methodology works and produces accurate data? Unless the organization can provide the results of their other internet-based surveys which are consistent with other data, whether from the Census or other surveys, the survey results are not worthy of being reported.
  7. What is the organization’s experience and track record using internet-based polls? Unless the organization can demonstrate a track record of obtaining reliable data with other online surveys, their online surveys should be treated with great caution.
  8. What is the organization’s experience and track record as a survey researcher using traditional survey methods? If the organization does not have a track record in designing and conducting surveys using the telephone or in-person surveys, it is unlikely that they have the expertise to design and conduct online surveys.
  9. Does the organization follow the codes of conduct of AAPOR, CASRO, and NCPP (whether or not they are members)? If they follow none of these, they are probably not a qualified survey research organization. The more of these Codes they follow, the more likely their data are to be reliable and be trusted.
  10. Is the organization willing to disclose these questions and the methods used (as required by the codes of conduct referred to in #9 above)? If the organization is unwilling to disclose, or unable to provide, the relevant information the survey is probably not worthy of being reported.

The NCPP also has a list of 20 questions journalists should ask about all poll results. They state in the introduction (emphasis added):

The only polls that should be reported are “scientific” polls. A number of the questions here will help you decide whether or not a poll is a “scientific” one worthy of coverage – or an unscientific survey without value. Unscientific pseudo-polls are widespread and sometimes entertaining, but they never provide the kind of information that belongs in a serious report. Examples include 900-number call-in polls, man-on-the-street surveys, many Internet polls, shopping mall polls, and even the classic toilet tissue poll featuring pictures of the candidates on each roll.

One major distinguishing difference between scientific and unscientific polls is who picks the respondents for the survey. In a scientific poll, the pollster identifies and seeks out the people to be interviewed. In an unscientific poll, the respondents usually “volunteer” their opinions, selecting themselves for the poll.

The results of the well-conducted scientific poll provide a reliable guide to the opinions of many people in addition to those interviewed – even the opinions of all Americans. The results of an unscientific poll tell you nothing beyond simply what those respondents say.

Continue reading

10/5/13

The Eyes Have It


Age-related macular degenerationThis summer my mother was diagnosed with macular degeneration. There is no cure. It is irreversible. It simply progresses. Science has some hope for future cures, and has some treatments to slow the progress, but a cure likely won’t come soon enough for her.

At 93, one expects that the body will fail, that organs and parts won’t work as well, will lose efficiency, will fail. But this is particularly tough on her. It was clear during our visit today that this diagnosis troubles her.

My parents were both voracious readers, and they passed along a love for books, reading and learning to me from an early age. Reading mattered, reading was important in our family. They shared that with me, it was part of our family DNA.

My father passed away eight years ago and my mother, in her nursing home, still reads every day. She reads for entertainment, for company, for relaxation, for amusement, and for learning. Or rather, she did, until this summer when the problem manifest itself and her reading was curtailed.

Now she struggles to read. She has to use a bright lamp over the book, and has taken to large-print books to still be able to read. But it’s a temporary solution as the AMD spreads inexorably.

She does crossword puzzles too, to keep her mind sharp. They’re harder to do now, because she can’t see the page as clearly. AMD affects the centre of the retina, spreading outwards.

She can see her TV screen if she sits up close, but the laptop screen is that small amount too distant, and besides, she can’t make out the keyboard very clearly. It was hard enough trying to peck out email messages with one hand. Now the computer sits unused.

Losing her ability to read easily is a blow to someone who has lived a tough life, suffering medical problems that have left her wheelchair-bound for the last decade. She certainly didn’t need any more complications.

Yet, despite all her trials and tribulations, her mind is still as sharp as a tack and her memory is remarkable – better than mine. She can recall details of her life, of her childhood right up to recent events, with astounding clarity. I envy that.

Continue reading

10/4/13

Digital Attachments


XCOM sniper (not Dmitrri)It’s tough to lose a solider. Especially one like Dimitri. A fine sniper, with a good kill record. I had trained him for so long, raised him from a lowly private to sergeant, then to lieutenant. He was equipped with the best gear. His accuracy had improved to a deadly asset. He was a cornerstone to my tactical approach.

He was also an investment in time and materiel. And as such, he was headed for greatness. Captain, maybe major.

Until the aliens got him. That was nasty.

Three of them swarmed his position, flanking his protection and taking him down with close melee attacks while the rest of the squad was busy defending citizens, too far to help.

Not a pretty sight.

The same battle took out Matt, the heavy weapons corporal who blasted whole blocks with his rocket launcher. Matt was caught in the blast of an exploding car outside a mall where the aliens had landed. Damn, I hadn’t counted on that when I moved him up to an overwatch position. But the aliens set the car on fire and that was that.

Our assault got caught in an ambush. We won, eventually, but it was a long fight with every inch bitterly contested. Coming back to base we were a solemn group. Two dead. Not a good thing.

Now the squad looks awfully thin, two down with rookies in their place. Big shoes to fill. And it’s not getting any easier out there, with the aliens ramping up their own technology, and getting tougher and smarter all the time. Winning this war won’t be easy. Matt I could almost afford to lose, being relatively new, but Dmitri was my best sniper.

I need to start training someone, fast. But who?

Of course it’s a game (XCOM: Enemy Unknown to be precise). Playing it this week has made me ponder the nature of attachment, in particular our attachment to characters in games or online. Why does it matter to us when a digital character “dies”? Or how he/she “lives”? How do we get so attached to virtual beings?

After all, it’s not like real life or death. Just a game. But yet…

Losing Dmitri irked me, but it also bothered me on a deeper level. Not simply because I had customized him, changed his suit colours, his facial hair, and imagined a background for him. He was mine. Or me. I’m not sure which. There was an emotional link. Not the easiest thing for a person who values logic and skepticism.

When the aliens gutted Dmitri, I was torn between restarting at the last save-game position and playing the deus ex machina role to save him, or letting the narrative run as it played out. Starting again felt like cheating. Letting him die felt like I had failed him. It. Dmitri wasn’t real, of course. But he/it felt like he was, at times. The narrative won, but not without misgivings.

Continue reading

10/2/13

I Didn’t Know That…


History of EnglandOne of the great delights of learning is to be able to read or hear something new, something unknown, something that challenges the mind or your previously formed ideas and opinions. Something that fascinates and delights you. That “ah ha!” moment.

Last week I stumbled across a website called History of England and I felt like that when I started to read through it. Better yet, I spent an hour downloading the 104 free podcasts of his history (plus the eight or so supplementary ones) to listen to while I walk my dogs.*

The site is a blog created by David Crowther, who also reads the pieces for the podcasts. Crowther modestly calls himself a “part time history enthusiast,” but his writing is as good as many of the histories I’ve read.**

I discovered the site when I was searching for some data on the Middle Ages for my post on the Unknown Monk meme last week. I started reading, then reading some more, and suddenly it was several hours later.

Crowther’s succinct profile is:

Interests: Well, History, obviously. But also a dedicated allottment owner, though at the most important times it’s difficult to get down there enough. Then very keen on walking, whether with the dog or something more major. Play tennis, bit of golf; armchair Rugby & cricket fan. Supported the Leicester Tigers since . . . a long time ago.

With some breaks for personal time, Crowther produces a weekly podcast – an amazing amount of work and dedication I admire and respect. I know how tough it can be to do this sort of work with any regularity. But this stuff requires a lot of background work: reading, culling images, cross-checking.

Plus he fills his blog with maps, text and images to supplement the podcasts. It’s a wonderful place to simply explore. England’s history is so rich it never fails to captivate me. Somewhere in that timeline, my ancestors lived and breathed, fought, worked the land… where, or course, I don’t know, but probably in the north near my father’s home of Oldham.

I started listening to Crowther’s podcasts on Monday and I’ve finished a mere eight of them – each is about 30 minutes. I’ve just finished the second on Alfred the Great and am in the late 9th century. Really intriguing guy – and learned, not just one of the era’s typical warrior-kings. Literate – in fact he not only taught himself Latin (and translated Latin into the vernacular), but wrote some of the earliest written works in the vernacular.

So far the stories been full of surprising information about the early English – and the successive invasions of the Angles, Jutes, Saxons, Vikings and Danes after the Romans pulled out of Britain in 410 CE. Ripping stuff, and told with a light hand and a dry sense of humour. He reads very well, with a good speaking voice, measured and easy to follow.

It’s an era I know damned little about – actually no one does, really, because until Alfred there was little written, or at least little that has survived. It’s not called The Dark Ages for nothing. But it turns out to be a rich, fascinating time for all that. Kings with odd names, warriors, battles, politics, internecine squabbles, church and state, family feuds… the stuff of good history.

One of those “ah ha!” moments was his talk about Offa, King of Mercia in the 8th century CE. I’d heard the name, but wasn’t really aware of his place or importance. Now I know enough to want to delve deeper. I expect a trip to Chapters or Amazon in the near future will include a search for books on this period.

I’m hooked. And I have 100 more to go!

By the time I get to the end, I expect he will have added many more, so I can look forward to many enjoyable hours. His 104th podcast – the latest as of this writing – only brings us to the mid-14th century. He hasn’t even reached my personal favourite – the late Tudors and early Stuarts. I can hardly wait for him to delve into Henry VIII, Elizabeth and Shakespeare. That should be getting close to lecture 200, I suspect.

~~~~~ 

* My usual listening fare has been audio courses from The Great Courses, which I still enjoy listening to. Their individual lectures are 45-60 minutes each, which means I sometimes can’t finish one when walking the dogs. Sometimes I listen to music copied from old 78 recordings instead.

**  And perhaps better than many – with history as a major interest of mine, I’ve read thousands of books over the last few decades and not all of them are as spellbinding as Crowther’s modest work.

10/1/13

The (sometimes violent) urge to write


Scribble, scribbleAs of this writing, I will have published 253 posts since I began this blog at the ending week of December, 2011. Two hundred and fifty three posts in 21 months. Just over one post every two-and-a-half days, on average. Plus 30 or so still in draft mode. Another half-dozen scribbled in word processing notes or notebooks.

And that doesn’t include the six years of blog posts – a list of 91 pages – on my previous blog site (still available in archive format, although some formatting issues have developed after some code updates).

“Scribble, scribble, scribble,” as Prince William Henry, Duke of Gloucester and Edinburgh said to Edward Gibbon. *

Approximately 500,000 words on this blog as published, public material. Uncounted numbers on my other sites, forum and blog, in draft or other formats.

That’s a lot of writing – and it still doesn’t include the writing I have done for my Municipal World articles or books (more than 75,000 words in two published books, one submitted and in editing at 45,000, and the fourth still being written – about 20,000 so far), my Machiavelli book (more than 75,000 words), and a novel I started more than a year ago (approx. 50,000 so far).

Or the writing I’ve done for an upcoming convention talk, my websites, and the innumerable Facebook (and the pages I maintain), Twitter, LinkedIn and forum posts. The ukulele and harmonica reviews, the motorcycle essays, the blog pages, and pre-blog material, the tequila guide. Or numerous emails to staff and fellow councillors, the work I did for a local political party – including crafting their newsletter.

Continue reading