10/2/12

Conducting a survey about a casino


Last night, Collingwood Council debated a motion about a possible casino in Collingwood, made by Counc. Joe Gardhouse that read,

WHEREAS the Ontario Lottery and Gaming Corporation (“OLG”) has requested individual Municipalities to respond to their RFP not later than November 16, 2012;
AND WHEREAS Collingwood has been identified as a potential host site for their Gaming facilities expansion within the C-7 zone;
AND WHEREAS the potential monetary and ancillary benefits to the taxpayers and businesses of Collingwood are significant;
AND WHEREAS this opportunity deserves and requires Municipal due diligence and public input as required by the OLG RFP;
THEREFORE BE IT RESOLVED THAT Council of the Town of Collingwood instruct Staff to identify suitable locations and/or zones for the potential gaming site not later than October 15, 2012;
AND FURTHER THAT Council schedule a Council/Public Information Meeting regarding the potential economic and social impact to the community not later than October 30, 2012;
AND FURTHER THAT an opinion survey/poll or other public forum be conducted to include a cross-section of citizens and businesses for their feedback on the gaming facility expansion to be completed not later than November 12, 2012;
AND FURTHER THAT Council shall notify the OLG of the Town’s position by not later than the November 16, 2012 deadline.

This was broken up into four motions – one about locations (defeated),the survey (passed five-four; I was one of those against this step), the public meeting (unanimous) and the notice to OLG (unanimous).

(The Whereas preamble statements were not voted on. I would have taken exception to the statement there there are significant “monetary and ancillary benefits to the taxpayers and businesses of Collingwood” had that discussion happened. What about the detrimental effects? What will they cost us?)

To put this in context, the Ontario Lottery and Gaming Commission (OLG) has identified this general area as one of the preferred locations for a new casino (basically a warehouse with 300 slot machines – Georgian Downs has 1,000, Rama has 2,500). Collingwood is one of four regional municipalities that signed a memorandum of agreement (MOU) this summer to collectively support a casino in Wasaga Beach, if the Beach wanted one. The Beach would share the revenue with its neighbours in return for our support.

Statistics cartoonThis motion essentially suggests Collingwood would ignore the MOU and apply on its own for a casino without the regional sharing or cooperation. If Collingwood was to get the casino, I would argue we should still share the profits with the other consignees, because to do otherwise would not be honourable.

Let’s look at what’s involved with the survey Council approved last night.

We want to get 400 qualified answers (a qualified sample for a population of 20,000 with 0.95 confidence level is 378. See www.berrie.dds.nl/calcss.htm. Our population with part-time residents is closer to 25,000 according to our staff estimates, and we get many visitors, so I’m rounding up a bit to simplify the math below.

An unqualified response is worthless. And a smaller sample is equally worthless because it reduces the validity of the results. We need a good level of confidence in the results to determine if they actually reflect the community’s wishes.

But what part of the community is bring asked? The respondents need to mirror our own local demographics. What if 75% of respondents are between 18 and 25, and the majority of that group wants a casino… is that more significant that if the remaining 25% unanimously reject it? How do you weight the results?

We also need to know in advance what’s the bar for acceptance or rejection? 51%? 60%? 66%? 75%?

What if it’s a 50-50 split? What is the criteria for saying yes or no? Is a simple majority enough (50% plus one) to decide either way?

We didn’t get into that, last night, but it is a significant topic for heated debate. And we can’t proceed to a decision without having established the criteria.

Someone with experience and education will have to design the survey – establish the methodology, then create the many questions necessary to qualify and quantify the results. You need a professional pollster to do this, probably someone with a university degree in statistics or public relations. It can’t be handed off to someone without at least some experience in poll and survey design and in data analysis.

Demographics cartoonStaff and council will have to be interviewed first to determine the appropriate methodology, the nature of the questions, the sort of result expected, the community profile, etc. And then the designer will need to get final approval for the proposed questions. So say at least a week for the design and interview phase.

Someone will also have to design a template for recording the answers. This is probably the easiest part: we could either buy pre-packaged survey software. We could have a staff or contract person code a rather complex, large Excel sheet, too, or an Access database template.

Based on my experience with Excel and database programming, that shouldn’t take more than four or five days once the questions have been compiled and approved. It’s quicker to purchase something and just enter the questions, and let the software crunch the numbers as they get tabulated. That might reduce the time for the template preparation to a day or two.

Let’s also assume the time per person to identify a potential candidate from a tax list or phone book, call them, ask them the qualifying questions to be sure they are in the target audience (not children, not employed by the municipality or a spouse of such an employee, or are employed by the OLG, are residents not visitors, own or rent property in the town, etc.), ask the necessary demographic questions (age, gender, income bracket, marital status, employment status, etc.), and then ask the actual survey questions is 10 minutes per person.

That’s not very long – about half of what professional pollsters usually allocate per call.

Add another five minutes per person for the caller to fill in the forms on the computer, check the person off the list, record the data in the paper record and so on.

That’s a mere 15 minutes per person now. That may be reasonable time for a survey of this sort, but it leaves no margin for error or contingencies.

There’s no leeway for wrong calls, for answering callers’ questions, for explaining the nature of the survey when asked, hangups, busy signals, no responses, pens running out of ink, computers crashing, invalid respondents, etc.

StatisticsThat’s 6,000 minutes to get 400 qualified answers: 100 hours. At the average 7.5 hours per day, that’s 13.33 days non-stop. Let’s give the caller and extra 1.7 days over that period as contingency for unanswered calls, bathroom breaks, talking to staff and supervisors, making photocopies, getting pens, making backups, getting a coffee… say 15 days minimum for one person, non-stop, to get a reliable sample with any statistical accuracy and validity.*

Four working weeks total time so far. Assuming, that is, there are no delays, and nothing goes wrong, that the washrooms aren’t busy, pens don’t run dry, and there’s no lineup at Tim’s. Plus a week to prepare the survey.

Obviously we want it sooner than that, so let’s triple the number of workers to get it in one week: three people working non-stop for five days, calling residents, asking questions. Again, no contingency or leeway has been allocated – we expect 100% efficiency.

Then, once we have our 400 qualified respondents, we need someone to collate the responses and produce a report on the answers, breaking down the responses by the demographic criteria (are you a full or part time resident, are you currently employed, age, gender, family income bracket, etc.). And get it printed for council and staff, and make a public presentation to council. We may be able to get the report in a week after the survey has been completed, if we push hard.

That report and analysis will require someone experienced in statistics and demographics to analyse and break it down; maybe we can use the same person we used to design the survey.

So we’re at a minimum three weeks from the design stage to the final report, and no time or effort wasted in between. We need at least three full time staff and one professional pollster/statistician/report writer to do it in three weeks.

Who will we use to do the calling and which professional statistician will we get to write the report?

If we use staff for the calls, who trains them and how long will it take to train them? Which three people on staff have at least one week of uninterrupted time they can dedicate to this? (If we do have staff with that much free time, I want to raise this at budget time!)

Let’s say we also want to ask a qualitative, not quantitative question like, “Why do you think a casino would be good (or bad) for the town”? Something that requires more than a simple yes or no answer. That would add another several minutes per caller to take and record the answer. My time estimates above are very, very conservative, by the way. As it says on www.dobney.com/Research/MR_basics.htm:

In terms of cost, most market research is charged on a time basis plus a management/design fee. If you took a general face-to-face population study of 1000 people. You might allow 15 minutes for the interview, 20 minutes finding/contact time, 10 minutes for processing each questionnaire – so 45 minutes per interview or 750 hours of time (100 days), on top of which would be added time for questionnaire design, production and dispatch, interviewer briefing and management, creation of tables, analysis and presentation. Typically on a straightforward survey these elements should add about 15-20 days, although at a higher daily rate.

Can we use internet polls or Facebook instead, as Counc. West suggested? Sure – if you want the results to be entirely meaningless, and statistically and demographically worthless – simply a pointless “feel-good” exercise that gets trashed and humiliated in every media. It would be just as accurate to check the Magic-8 Ball for an answer as to conduct a typical online poll.

(You can read my personal comments on the validity of internet polls here: www.ianchadwick.com/blog/are-internet-polls-valid/ )

T shirtWe could hire programmers to write a secure website that has login and authentication controls on voters, and some data collection and analysis programs. We would still need to have qualifying questions and poll questions written by a professional to be sure people’s responses were valid, and someone equally qualified to write up the report later. It would add time to the exercise and what’s to guarantee that we will get the necessary number of qualified responses within the time frame before the deadline? None.

Or we could simply buy the services of one of the many online survey companies to collect and collate the data for us. We still need the professional to design the questions and create the final report. The disadvantage here is that to get a qualified response in an adequate sample may take months, because there is no incentive for users to complete the survey. Plus we’d have to advertise and promote the survey site to try and get residents to visit it. Radio ads, print ads… the costs keep mounting.

Whatever we chose, it will cost us in both time and money to achieve anything meaningful. To get those results before we have to make a report to the OLG (Nov 16) is unlikely.**

So I ask what I asked last night: does anyone have the slightest inkling of how much will this survey cost the municipality and how much staff time will be put into it? What if it cost taxpayers $50,000? $60,000? Or more? Will it be worth the money?
~~~~~
* Say one in two people who answer the phone proves to be unqualified (i.e. a child or non-resident) and it takes five minutes to determine that and do the paperwork to record that call. That would add another 2,000 minutes to the total time to get 400 qualified respondents. Or 33.33 hours (another 4.4 working days). If only one in four is unqualified, but one in four is a no answer or busy signal (1 minute to record this), then it would add 22.8 hours or just over three days of work.
** I have already expressed my opposition to the process – I believe we should follow through with the MOU and support Wasaga Beach’s bid to be the host community.