Attention Researchers: Would You Touch this Election with a 10 Foot Poll? by Steve Runfeldt, Guest Writer
"If enough of us refuse to answer, the polling data will become so unrepresentative and unreliable even the media would have to admit it was useless." — Ariana Huffington on her blog, The Huffington Post shortly after the New Hampshire Democratic Primary.
On January 8, Hillary Clinton won that race by 2 points, a 10 point shift from what most polls had predicted. Watch any news program today and you are likely to hear a political pundit deriding the validity and reliability of political polls, and by extension, surveys.
Andrew Kohut, president of the Pew Research Center, wrote a column in the NY Times, titled "Getting It Wrong", saying, "All the published polls, including those that surveyed through Monday, had Senator Barack Obama comfortably ahead with an average margin of more than 8 percent." To his credit, Kohut defends the overall reliability of polling methodology and suggests factors for further study.
Some of the factors that have been suggested include:
Sampling problems — Some have suggested that certain demographic segments of the voting population are underrepresented because they do not cooperate with pollsters, suggesting that these people tend to be lower income, less well educated and more racially biased.
Media interpretation and bias — Some have suggested that the poll results were in fact accurate, but that the media is poorly educated in how to read and interpret poll results.
Racial bias — Some have suggested that many white voters are more liberal when speaking to pollsters than they are when alone in the voting booth. Would online polls be more representative?
Gender bias — Some have suggested that women would not want to seem biased toward Hillary Clinton, or men might be embarrassed to admit that they were going to vote for a woman, so their actual votes would be underrepresented.
Voting machine error — Some bloggers believe that the polls were correct, but that the voting machines were in error.
Bias in political polling — Much of the public opinion simply does not trust political polls. Certainly push polling and other disreputable practices further this distrust. I once was asked to help a local campaign with a poll when one activist suggested, "Why don’t we just tell them we took a poll. Who would know?" I had to tell her, "I would know."
So, what do you think was going on in New Hampshire?
And what do you do in commercial market research, when the actual results differ significantly from what you or the client expected? Do you challenge your results, or the expectations? Have you ever encountered a client who just did not trust surveys?
Looking forward to your responses.
Steve
P.S. The CNN/ UNH poll (http://i.a.cnn.net/cnn/2008/images/01/05/top10.pdf ) predicted a tie +/- 5%. The actual difference was 2 points. Maybe not all of the polls were entirely wrong.
###
Steve Runfeldt (Senior Account Executive for Quantitative Research) came to Schwartz Consulting Partners in September 2007 with a total of 27 years of research experience. His expertise is in innovative research design, statistics and analysis. He has a BA degree in Psychology and Anthropology from Brandeis University, graduate work in Behavioral Sciences, Genetics and Neurobiology at The Rockefeller University and Comparative Psychology at Georgia State University. Steve has worked as project manager, statistician and director of Internet research at Elrick & Lavidge (now TNS), principal and VP of Research at Customer Sat.com and founder and CEO of Justaskthem.com.
Steve is recognized as a pioneer and innovator in the field of Internet survey research, having introduced some of the first methods for online sample control, real time online reporting and customer relationship management. As head of justASKthem.com he designed and managed one of the first online customer satisfaction management systems which enabled AT&T WorldNet to become the industry leader in customer service satisfaction. Other clients with whom Steve has worked include AMD, American Century, Cover Girl, Pac Bell, Price Waterhouse, Procter and Gamble, Roper Starch, SBC, Siemens, the U.S. Navy, the World Bank, and others.
In 2005 Steve developed a new method using Flash technology that allows websites to collect consumer feedback through a short 3 question inline feedback application. When installed on a web shopping cart this method has achieved as high as a 70% response rate.
As a member of the Marketing Research Association Internet Ethics Task Force, Steve helped to write the association’s Internet market research survey guidelines. He has been a keynote speaker, panel moderator and workshop presenter for groups such as the Marketing Research Association, Emory School of Business, University of Georgia Marketing Research Program, the Institute for International Research and the International Quality and Productivity Center.
February 19th, 2008 at 10:07 am
Marketing research and specifically polling, is based on two people talking to each other; an interviewer and a respondent. All the science, theory, mathematics, and logic can and will change as these two people talk.
February 19th, 2008 at 1:48 pm
I find this Presidential campaign to be fascinating. Predicting an outcome seems to be a daunting task. Polling is certainly the best way to predict but I don’t believe that everyone’s decided yet. I agree that many people could tell you one story about who want but change in the booth. Especially when it’s a black man and a women. My personal prediction is Obama. He is very charismatic, intelligent and diplomatic. He is the Jackie Robinson of politics and I believe he’d be good for America and the world.
February 20th, 2008 at 11:12 am
Actually, I made a post to aapornet that addressed this issue. In effect saying that pollsters should use a well tested approach from commercial research to accurately measure preference. Here’s an excerpt from this post:
….there is one major reason the polls are having a hard time. Because pollsters work for clients (such as the press or TV network, a candidate, and their own private polls,) they have to report on public decisions – yes-no, good-bad, like-dislike. So in election polls they ask voters who they prefer. When a person says they prefer Obama or Clinton they are given full credit by the poll – 100% — of being for that candidate. In reality, especially in this Democratic primary, few people are 100% for any candidate. I heard many voters thinking out loud about how they could vote for any of the two leading candidates, Clinton or Obama. I hear conflicted aspects of each person’s decision. A polling technique used in commercial research that is not beholden to decision driven clients would have not been fooled by the Democratic primary process. This approach is called the “constant sum” decision model.
In this approach a voter would be asked preference in two parts — which candidates would be “considered” if they were voting – and among those considered, how would 100 (or 10) points be divided between them (called “constant sum.”) Average the percentage of points given to each by the survey sample, and you get a very close estimate of preference for any candidate overall. No undecided here although you can identify respondents who are partially undecided.
But the real improvement is in the detail given by each voter in the survey. Take two voters considering between Clinton and Obama. Person A gives Obama 100 points and Clinton 0 points, and Person B gives Obama 60 points and Clinton 40. In the traditional preference question both would be for Obama, but if a small thing happened in the campaign that impacted Person B, there might a small switch in preference to Clinton 55/Obama 45. In elections like this small things have big impacts. It is likely that in the early stages of the primary elections there are a lot of voters who were having a hard time deciding between alternative candidates that are acceptable. Movement can occur quickly and must be measured with a micrometer, not a yardstick. (Lately as preference is getting stronger most of the polls have been right.)
This constant sum approach enables the pollster to concentrate on the intensity of preference for each candidate for each voter surveyed and more accurately represent the solidity of the support for the candidate. So the polling industry should not be measuring preference of a voter, but degree of preference.
If they did this they would find that the polling public would better appreciate an approach that takes into account their ambivalence. Results from elections would be more accurate to predict and voters and issues more “in-play” would be better understood.
February 20th, 2008 at 1:52 pm
Harry,
I agree with you. Rating candidates on a preference scale seems to me to be more likely to more accurately predict the actual outcome.
I wonder why you suggest the constant sum approach over a simple Likert scale rating, e.g, How likely do you think you are to vote for x? Not at all likely, Not very likely, Somewhat likely, Likely, Very Likely.
I think that Likert scale questions are easier for respondents to answer. Did you have a reason for suggesting constant sum?
I wonder if there is anyone on this blog who conducts political polls who can address this.
Thanks,
Steve
February 21st, 2008 at 4:02 pm
I believe many independent voters (40% in NH) saw Obama as the likely winner and, when the curtain closed, decided to cast their support in favor of McCain who they felt could use their vote. That is, Obama lost many independent votes to McCain, not Clinton. Had the polls been closer for Obama/Clinton, or further for McCain/Romney, I believe the results would have more closely mirrored the polls. This may be a case of micro-motives and macro-behavior; independent voters may not have shifted their votes had they known doing so would have affected the outcome. So I don’t believe the pollsters got it wrong.