{"id":11598,"date":"2014-09-21T16:26:09","date_gmt":"2014-09-21T20:26:09","guid":{"rendered":"http:\/\/ianchadwick.com\/blog\/?p=11598"},"modified":"2021-05-15T07:45:02","modified_gmt":"2021-05-15T11:45:02","slug":"no-data-are-better-than-bad-data","status":"publish","type":"post","link":"https:\/\/ianchadwick.com\/blog\/no-data-are-better-than-bad-data\/","title":{"rendered":"No Data Are Better Than Bad Data"},"content":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_11598\" class=\"pvc_stats all  \" data-element-id=\"11598\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewBox=\"0 0 502 315\" preserveAspectRatio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/ianchadwick.com\/blog\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p><a title=\"Avoiding survey bias\" href=\"http:\/\/fluidsurveys.com\/university\/avoiding-survey-bias\/\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" src=\"http:\/\/fluidsurveys.com\/wp-content\/uploads\/2013\/08\/AVOIDBIAS-672x300.png\" alt=\"Avoid bias\" \/><\/a><br \/>\n<strong>The full name of an\u00a0article I read today is,<\/strong> &#8220;The Fallacy of Online Surveys: No Data Are Better Than Bad Data.&#8221; It&#8217;s from 2010 and very good. You can find it on the <a title=\"Responsive Management\" href=\"http:\/\/www.responsivemanagement.com\/news_from\/2010-05-04.htm\" target=\"_blank\" rel=\"noopener\">Responsive Management website<\/a>. It makes some key points about the invalidity of online surveys:<\/p>\n<ul>\n<li>For a study to be unbiased, every member of the population under study must have an equal chance of participating.<\/li>\n<li>When online surveys are accessible to anyone who visits a website, the researcher has no control over sample selection. These self-selected opinion polls result in a sample of people who decide to take the survey &#8212; not a sample of scientifically selected respondents who represent the larger population.<\/li>\n<li>Non-response bias in online surveys is complicated by the most egregious form of self-selection. People who respond to a request to complete an online survey are likely to be more interested in or enthusiastic about the topic and therefore more willing to complete the survey, which biases the results.<\/li>\n<li>Unless specific technical steps are taken with the survey to prevent it, people who have a vested interest in survey results can complete an online survey multiple times and urge others to complete the survey in order to influence the results.<\/li>\n<li>Because of the inability to control who has access to online surveys, there is no way to verify who responds to them &#8212; who they are, their demographic background, their location, and so on.<\/li>\n<\/ul>\n<p>I&#8217;ve said this all before. The article concludes:<\/p>\n<blockquote><p>As a result of these problems, obtaining representative, unbiased, scientifically valid results from online surveys is not possible at this time, except in the case of the closed population surveys, such as with employee surveys, described earlier. This is because, from the outset, there is no such thing as a complete and valid sample &#8212; some people are systematically excluded, which is the very definition of bias. In addition, there is no control over who completes the survey or how many times they complete the survey. These biases increase in a stepwise manner, starting out with the basic issue of excluding those without Internet access, then non-response bias, then stakeholder bias, then unverified respondents. As each of these becomes an issue, the data become farther and farther removed from being representative of the population as a whole.<\/p><\/blockquote>\n<p>There&#8217;s also a<a title=\"Colorado State\" href=\"http:\/\/warnercnr.colostate.edu\/docs\/hdnr\/hdfw\/2010\/Mark_D_Duda_Fallacy_Online_Surveys.pdf\" target=\"_blank\" rel=\"noopener\"> good slide show on internet surveys here<\/a> that goes over the basics presented in the article above. <a title=\"Self-selection in online surveys\" href=\"http:\/\/www.cbs.nl\/NR\/rdonlyres\/EEC0E15B-76B0-4698-9B26-8FA04D2B3270\/0\/200814x10pub.pdf\" target=\"_blank\" rel=\"noopener\">A 2008 paper<\/a> addressed just issue with online surveys: self-selection. The author,\u00a0Jelke Bethlehem, wrote:<\/p>\n<blockquote><p>&#8230;web surveys are a fast, cheap and attractive means of collecting large amounts of data. Not surprisingly, many survey organisations have implemented such surveys. However, the question is whether a web survey is also attractive from a quality point of view, because there are methodological problems. These problems are caused by using the Internet as a selection instrument for respondents.<br \/>\nThis paper shows that the quality of web surveys may be seriously affected by these problems, making it difficult, if not impossible to make proper inference with respect to the target population of the survey. The two main causes of problems are under-coverage and self-selection.<\/p><\/blockquote>\n<p>The author concludes:<\/p>\n<blockquote><p>It was shown that self-selection can cause estimates of population characteristics to be biased. This seems to be similar to the effect of nonresponse in traditional probability sampling based surveys. However, it was shown that the bias in selfselection surveys can be substantially larger. Depending on the response rate in a web survey, the bias can in a worst case situation even be more than 13 times as large.<\/p><\/blockquote>\n<p>In other words: most online surveys are bunk. You might also recall I wrote about online surveys in past posts. I won&#8217;t repeat what I said then, but here are the links to those posts:<\/p>\n<ul>\n<li><a title=\"Are Internet Polls Valid?\" href=\"https:\/\/ianchadwick.com\/blog\/are-internet-polls-valid\/\" target=\"_blank\" rel=\"noopener\">ianchadwick.com\/blog\/are-internet-polls-valid\/<\/a><\/li>\n<li><a title=\"Internet surveys: bad science, worse data\" href=\"https:\/\/ianchadwick.com\/blog\/internet-surveys-bad-data-bad-science-and-big-bias\/\" target=\"_blank\" rel=\"noopener\">ianchadwick.com\/blog\/internet-surveys-bad-data-bad-science-and-big-bias\/<\/a><\/li>\n<li><a title=\"Conducting a casino survey\" href=\"https:\/\/ianchadwick.com\/blog\/conducting-a-survey-about-a-casino\/\" target=\"_blank\" rel=\"noopener\">ianchadwick.com\/blog\/conducting-a-survey-about-a-casino\/<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_11598\" class=\"pvc_stats all  \" data-element-id=\"11598\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" version=\"1.0\" viewBox=\"0 0 502 315\" preserveAspectRatio=\"xMidYMid meet\"><g transform=\"translate(0,332) scale(0.1,-0.1)\" fill=\"\" stroke=\"none\"><path d=\"M2394 3279 l-29 -30 -3 -207 c-2 -182 0 -211 15 -242 39 -76 157 -76 196 0 15 31 17 60 15 243 l-3 209 -33 29 c-26 23 -41 29 -80 29 -41 0 -53 -5 -78 -31z\"\/><path d=\"M3085 3251 c-45 -19 -58 -50 -96 -229 -47 -217 -49 -260 -13 -295 52 -53 146 -42 177 20 16 31 87 366 87 410 0 70 -86 122 -155 94z\"\/><path d=\"M1751 3234 c-13 -9 -29 -31 -37 -50 -12 -29 -10 -49 21 -204 19 -94 39 -189 45 -210 14 -50 54 -80 110 -80 34 0 48 6 76 34 21 21 34 44 34 59 0 14 -18 113 -40 219 -37 178 -43 195 -70 221 -36 32 -101 37 -139 11z\"\/><path d=\"M1163 3073 c-36 -7 -73 -59 -73 -102 0 -56 133 -378 171 -413 34 -32 83 -37 129 -13 70 36 67 87 -16 290 -86 209 -89 214 -129 231 -35 14 -42 15 -82 7z\"\/><path d=\"M3689 3066 c-15 -9 -33 -30 -42 -48 -48 -103 -147 -355 -147 -375 0 -98 131 -148 192 -74 13 15 57 108 97 206 80 196 84 226 37 273 -30 30 -99 39 -137 18z\"\/><path d=\"M583 2784 c-38 -19 -67 -74 -58 -113 9 -42 211 -354 242 -373 16 -10 45 -18 66 -18 51 0 107 52 107 100 0 39 -1 41 -124 234 -80 126 -108 162 -133 173 -41 17 -61 16 -100 -3z\"\/><path d=\"M4250 2784 c-14 -9 -74 -91 -133 -183 -95 -150 -107 -173 -107 -213 0 -55 33 -94 87 -104 67 -13 90 8 211 198 130 202 137 225 78 284 -27 27 -42 34 -72 34 -22 0 -50 -8 -64 -16z\"\/><path d=\"M2275 2693 c-553 -48 -1095 -270 -1585 -649 -135 -104 -459 -423 -483 -476 -23 -49 -22 -139 2 -186 73 -142 361 -457 571 -626 285 -228 642 -407 990 -497 242 -63 336 -73 660 -74 310 0 370 5 595 52 535 111 1045 392 1455 803 122 121 250 273 275 326 19 41 19 137 0 174 -41 79 -309 363 -465 492 -447 370 -946 591 -1479 653 -113 14 -422 18 -536 8z m395 -428 c171 -34 330 -124 456 -258 112 -119 167 -219 211 -378 27 -96 24 -300 -5 -401 -72 -255 -236 -447 -474 -557 -132 -62 -201 -76 -368 -76 -167 0 -236 14 -368 76 -213 98 -373 271 -451 485 -162 444 86 934 547 1084 153 49 292 57 452 25z m909 -232 c222 -123 408 -262 593 -441 76 -74 138 -139 138 -144 0 -16 -233 -242 -330 -319 -155 -123 -309 -223 -461 -299 l-81 -41 32 46 c18 26 49 83 70 128 143 306 141 649 -6 957 -25 52 -61 116 -79 142 l-34 47 45 -20 c26 -10 76 -36 113 -56z m-2057 25 c-40 -58 -105 -190 -130 -263 -110 -324 -59 -707 132 -981 25 -35 42 -64 37 -64 -19 0 -241 119 -326 174 -188 122 -406 314 -532 468 l-58 71 108 103 c185 178 428 349 672 473 66 33 121 60 123 61 2 0 -10 -19 -26 -42z\"\/><path d=\"M2375 1950 c-198 -44 -350 -190 -395 -379 -18 -76 -8 -221 19 -290 114 -284 457 -406 731 -260 98 52 188 154 231 260 27 69 37 214 19 290 -38 163 -166 304 -326 360 -67 23 -215 33 -279 19z\"\/><\/g><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/ianchadwick.com\/blog\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p>The full name of an\u00a0article I read today is, &#8220;The Fallacy of Online Surveys: No Data Are Better Than Bad Data.&#8221; It&#8217;s from 2010 and very good. You can find it on the Responsive Management website. It makes some key points about the invalidity of online surveys: For a study to be unbiased, every member of the population under study must have an equal chance of participating. When online surveys are accessible to anyone who visits a website, the researcher has no control over sample selection. These self-selected opinion polls result in a sample of people who decide to take \u2026&#8230;&#8230; click below for more \u2193<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[4,261,204,110],"tags":[71,61,506],"class_list":["post-11598","post","type-post","status-publish","format-standard","hentry","category-collingwood","category-critical-thinking-2","category-internet-social-media","category-media-issues","tag-internet","tag-politics","tag-polls-surveys"],"a3_pvc":{"activated":true,"total_views":8083,"today_views":0},"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"jetpack-related-posts":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/posts\/11598","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/comments?post=11598"}],"version-history":[{"count":5,"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/posts\/11598\/revisions"}],"predecessor-version":[{"id":26166,"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/posts\/11598\/revisions\/26166"}],"wp:attachment":[{"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/media?parent=11598"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/categories?post=11598"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ianchadwick.com\/blog\/wp-json\/wp\/v2\/tags?post=11598"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}