Wednesday, November 17, 2010

Beware of Poll Predictions Based on Poorly Representative Samples

I am going to take a moment of publisher privilege and write about a topic that is not specifically atomic, nuclear, or energy related. As I read about election predictions based on polling numbers, I am reminded of a story I learned during one of my statistics classes.



We had just learned the mathematical formula for determining the required sample size for a specified 'error rate'. It turns out that if you test 1000 randomly selected representatives of a given group, you will be able to predict the behavior of that group to a precision of +- 3%.



To illustrate the importance of the descriptive words - 'random' and 'representative' - the instructor told the famous story of the polls that predicted that Dewey would handily beat Truman in the 1948 election. That election took place in the early days of political polling. The poll was taken by telephone at a time when telephones were still luxury items. Since Dewey's supporters tended to be in the higher income classes compared to Truman's, the telephone polls included a systematic sample bias that made their results wildly inaccurate in predicting the actions of the people who voted.



There are a number of observers who are pointing out the fact that the 2010 elections might contain a different kind of telephone bias, since there is a growing population of people who do not own traditional wired telephones. Similar stories ran during the 2008 election season. According to The Pew Research Center, a recent phone sample that specifically includes cell phones in addition to land lines shifted a few points towards candidates from the Democratic Party.



Based on my own admittedly non-random sampling, I suspect that there are other technological developments that are making phone polling less and less representative of the voting population. Though I generally take the time to vote, I NEVER take the time to talk to a pollster. Like most of the people that I call, I only answer the phone if I recognize the caller ID. I let voicemail handle the sales people and the robodialers. I never return those calls.



In the rare instance when I am distracted enough to answer the phone without looking at the caller ID, I hang up without comment as soon as I recognize an autodialer, a poll or a sales pitch. (I have been told I am being rude when I quickly hang up, but I figure I am saving the person on the other end time. Is that rude or not?)



The final technological issue that is becoming more important is the growing use of texting rather than talking. Both of my twenty something daughters are more likely to answer a text from a friend than a telephone call from a stranger - quite frankly, so am I. Are my responses to attempts to intrude in my life common enough to cause an additional bias in the polls towards answers given by people with more time on their hands and less interest in using modern technology?



I am not sure what political positions the people who use cell phones, texting, and caller ID favor, but my prediction is that the final results of the upcoming election will be substantially different from the results predicted by using polls produced from surveying people who actually answer randomly placed calls and take the time to answer questions from surveyors.