[Note: While this article focuses primarily on quantitative research, the basic points also apply to qualitative research]
Whenever anyone talks to me about Respondent Cooperation, I invariably tell them that we should place more emphasis on Respondent Satisfaction. I say this because I believe that if we continue to focus on the former and ignore the latter, we risk losing our audience (our respondents).
Let’s face it: completing the average survey is about as exciting as filling out a loan application. Most surveys are visually unappealing and unexciting.
Respondent Cooperation rates have dropped for a number of reasons, but a major one is that completing a survey is too much like taking a test… not an experience most enjoy. Sure, there are other factors affecting Respondent Cooperation, but we tend to ignore that participating in most surveys is just not much fun.
This is primarily the result of a lack of imagination and creativity.
The lack of imagination and creativity in the research process is the result of numerous factors. I will pinpoint a few:
- Most researchers are nerds. After all, who but a nerd enjoys studying human opinions and behavior? The clinical approach we take in our work may get us the answers we are looking for, but uses a process and techniques that simply aren’t very entertaining.
- We write/ask questions in an unbiased manner which will get the answers required, not in ways which will amuse or engage the respondents.
- We tend to hire only researchers–not creative people–to assist us.
- We are too often restricted by budgets and timelines which prevent us from doing something “different.”
- We are creatures of habit. Most research companies have a style, and have developed a catalog of standard questions. It is very easy to throw together a survey from an inventory of questions which aren’t very interesting.
Let’s look at some methodologies.
Phone. Telephone research is effectively on its deathbed. Participating in a survey over the phone is as exciting as listening to a talk radio show in a language you don’t understand. Interviewers ask questions in an unbiased manner and don’t emphasize words or use intonation which will bias a response. Phone surveys are thus not very engaging, which in turn contributes to low cooperation rates.
Phone surveys are also frequently way too long. We live in the age of sound bites and text messaging (limited to 160 characters), not in an age of 20-30 minute questionnaires. As a result, phone research is dying , and will probably never work again as a great method for research.
Paper. Paper surveys all too often are really like a test with all the baggage and negative feeling attached. Paper surveys are, for the most part, badly designed, poorly laid out, and are usually printed in black ink on white paper. Talk about DULL! What’s the problem with using color or incorporating graphics? We should at least attempt to engage people.
Online. This is the medium with the most potential to engage respondents. Unfortunately most online surveys are too much like paper. They simply are not very engaging. This is because the tools used to program the surveys were developed to meet the needs of the researchers not amuse the respondents. What we have to keep in mind is that while we don’t currently have tools that let us design entertaining and fun surveys, these features certainly can be incorporated into existing software. All we have to do is ask for it. Any technique that you see being used on a Web page or an online advertisement could be incorporated into a Web survey. We need to make better use of movement, sound and color. It is time to start thinking of an online survey more like a good Powerpoint presentation.
The Bottom Line.
Researchers need to hire some/more creative people to assist them, and we need to start to make surveys more fun. Otherwise, we stand the risk of having no participants.
Those are a few of my thoughts.
- Do you agree with me?
- What are your thoughts?
I look forward to hearing from you!
###
Peter Van Brunt, PRC
Mr. Van Brunt has been involved with doing quantitative marketing research since 1974. In 1982 he started his own research company, ReData, Inc. Over the years, Mr. Van Brunt has been an active volunteer in the Marketing Research Association. He was one of the founding members of the Mid-Atlantic Chapter of the MRA, served as the second Chapter President, and remained active on the Chapter Board for a number of years. At the National level, he has served on numerous Councils, Committees, and Task Forces. He served on the National Board of Directors as a Director at Large, and was elected national President of the Marketing Research Association for the year 2002-2003. He was instrumental in MRA starting Professional Research Certification, and he currently chairs the Review Committee for Researcher Certification. He is PRC certified at the Expert Level. In 2006 he was presented with MRA’s Honorary Lifetime Membership Award.
He holds a BS in Mathematics from Bucknell University.
Josh Mendelsohn says:
I wholeheartedly agree. In fact its one of the reasons I came to Invoke over other research firms. We track satisfaction from our participants and their likelihood to participate again (both over 95% on 45-90 minute research sessions). Most companies not only don’t ask it, they don’t care.
Researchers need to remember, good surveys result in good data. Bad surveys result in bad data and the misuse of millions of dollars.
Larry Brownell says:
Peter,
I whole heartedly support some of your points specifically:
We need to enforce time limits for each form of research survey
We need to better utilize interactive media like the Internet & cells
We are to restrictive and prescribed.
I also agree with the creative need, but within limits. By creating an overly entertaining survey, you may be biasing the results. Those who like the personality reflected and those who don’t will respond differently. Emotional responses to things that entertain us are very subjective and I would suggest there is no universal for humor or other entertaining qualities and that by trying to interject these elements you are basically creating the same bias you would, if you were only interviewing interested or disgruntled consumers.
I agree with the principle of your idea, but I have yet to see anything that is universally entertaining, or may be it is just my humor that fails that test.
Caroline says:
The market research industry appears to have built its business model based on the belief that people like to spend their time taking surveys. That might have been the case when market research was new, but not so today.
Researchers seem to fail to understand the modern consumer’s value on time, expecting to get a chunk of the respondent’s time and attention without providing any significant compensation in return. The modern consumer is fairly well educated in marketing practices and knows that his opinion has monetary value, yet the researcher often expects to get five or ten minutes of time for nothing and even greater chunks of respondent with very minimal compensation.
If the research industry would follow the practices of the free market and begin to fairly compensate respondents for the research time it seems likely that they will find a more receptive audience.
Vic Walsh says:
Yes, questionnaires need to be shorter and more relevant. That should help boost respondent satisfaction.
On the other hand, research is a science, and the challenge will be to balance creativity with validity. We live in a world in which the respondent base has grown up on a steady diet of Sesame Street and similar shows. People want to be entertained, even in school. Some topics will require a huge amount of creativity and budget to satisfy that demand. Then how much of the research dollar will be spent on creativity instead of on the science? How will the researcher make a study about financial planning or buying tires fun?
Kelley Styring says:
Peter,
I would love to see a high calibre research company hire a kick a** game developer to pull together an over-the-top methodology. I’ve seen some interesting animated food development tools coming from Socratic and I like the invoke tools very much. I’d like to see more of this because it engages the respondent’s creative side and probably delivers even more insight than the dull Q&A we typically do. If you know of anything like this out there, please let me know.
Kelley
Ian Kiernan says:
Peter,
I 100% agree that we need to make the survey experience more exciting/enjoyable/entertaining for the respondent.
My company focuses on data collection and reporting. We program high volumes of online surveys targeting many different audiences including consumers, b2b professionals and healthcare professionals around the world. Much of the work that we do is quite complex.
We also have invested heavily in our technology to improve the survey experience for the respondent. As an example, we regularly utilize Flash technology to improve how questions are presented to respondents, drag & drop solutions, full screen video, automated probing of open-ends, etc.
So, from my standpoint, the technology is ready, available and accessible to companies within our industry.
The bigger issue for us has been the actual design and length of the surveys that we are being asked to program. We are continually asked to program 20 – 45 minute surveys that are very, very boring, with repetitive questions, large grids of attributes to rate, etc. And, in my opinion, there is no technology that will make these surveys more exciting.
As an industry, my view is that we need to continue to educate and push back on our clients to reduce the length of the surveys. If we can get the average survey length down to 10-15 minutes (or less), then I think we will (a) improve the quality of the data (b) improve the survey experience for the respondent and (c) give ourselves an opportunity to utilize the latest and greatest technology to bring our surveys alive.
Steve Runfeldt says:
I also wholeheartedly agree, but with the caveat that we need to be careful to attend to the statistical underpinnings of quantitative research.
Politicians often disregard the “unbiased” aspect of polls ending up with push polls that get them the results they are looking for. We need to be very careful of this sort of thing.
On the other hand, many of the sampling procedures we employ were developed specifically for phone, paper or door to door surveys, where the sample itself is the cost limiting factor. Online surveys do not necessarily need to have cost limiting samples. If the survey or feedback instrument is short, engaging and fulfilling to the respondent, we can use other methods. We conducted a short survey a while back that received a 70% response rate. The survey was 21 questions, but no respondent was asked more than three at a time. We got good solid results. The sample of questions was statistically balanced and we were able to get most of the crosstabs that you might get in an ordinary survey – with no multicollinearity.
People love to communicate online. They will open up about any topic, including personal information that they might never reveal on the phone. Browse through MySpace or Facebook, or any of the thousands of blogs. People do not have a resistance to telling all online.
So, why do we have such trouble getting these same people to respond to online surveys. I think that Peter has a good part of the answer here.
I would love to participate in a working group get together to explore alternative ways to collect statistically valid information from online consumers. Anybody interested?
Michael Mitrano says:
Nicely put, Peter. The longer and more boring we make surveys, the less representative those who have the patience and motivation to sit through them will be.
Chelsea Linsley says:
I agree with you as well. I think the benefits of interactive surveying are becoming more widely recognized by research buyers on the whole, however, I think it’s important to note that current offerings available in the market have their deficiencies. Increased cost, increased turnaround for programming, and the limitations of working within standard HTML surveying environments are all factors to consider. There is a need for greater flexibility in current interactive solutions.
As for the creative aspect – why not leave that up to the buyers? By switching to a system that allows for integration of any custom-designed content (such as Flash), buyers can exercise complete control over survey design. Integrating animation, video, or custom questions should be easy.
Any leading programming house should at this point see the value of offering an interactive component on-line. The fact is that engaged respondents provide better quality data and interactive surveys help keep respondents engaged.
Merrill Dubrow says:
Steve,
Thanks for your comments. Peter sounds like you have a tremendous interest here and an invitation from Steve Runfeldt to participate in a working group get together to explore alternative ways to collect statistically valid information from online consumers.
what’s the next step?
Merrill
Peter Van Brunt says:
Merrill,
I be happy to talk with Ed Sugar about putting together a program for a conference. Do you think people would be interested? –Peter
Merrill Dubrow says:
Peter,
Seems like there would be interest to me. Not sure I would start with a program, I think i would start with a small committee, brainstorm a little bit and than approach Ed about presenting it at a conference.
Seems to me everyone knows there is an issue and this could be a way to make some significant progress.
Ed?
Merrill
Ed Sugar says:
Peter –
Bullseye! You have touched on a subject very close to my heart.
Here are my comments from a posting on this blog back in December:
“We should be designing and developing methods to collect data that excite and inspire respondents to participate. Just like any great producer of consumer goods (Apple, NIKE, Toyota), this profession needs to continually find that “WOW!” factor that is going to keep our respondents engaged and forthcoming with their insights and opinions, without sacrificing integrity or quality. Participating in a marketing research study should be more like a trip to an amusement park rather than having your mother give you a haircut.”
I would be glad to work with you, Steve and anyone else to develop a program, a workshop, a webinar, a task force, a white paper, anything that will place this subject on the profession’s front burner.
Merrill Dubrow says:
Ed,
Thanks for you comments and agreeing to help. Steve/Peter – sounds like you have an audience – let Ed know what you are prepared to do and next steps.
Merrill
Patrick Glaser says:
Creative formats inject some excitement into the research process, but it also contradicts the sterile research paradigm that we’ve all been taught to follow. You’re adding noise into the equation.
The question is how do we manage the tradeoff between engaged cooperation and some added noise, versus low (potentially satisficing) cooperation and cleaner data? I think the answer is some much needed research-on-research comparing the performance of different formats.
Peter Van Brunt says:
Patrick,
I agree with you, but I am on the side that thinks we are all working with archaic techniques and sampling schemes that no longer necessarily fit the new paradigm. Like you, I think we need to do research on the research process. There is no guarantee that standards and methods which were long ago developed necessarily fit today. I agree with Steve’s comments. Our profession needs to develop new tools and techniques which better fit today’s research environment. I am convinced that we can have entertaining surveys which won’t bias responses. We just need to figure out how to do it.
Olivier says:
This topic is also very close to my heart. Like most people involved with online sample, I am reminded every day of how challenging it is to find new fresh people to take most surveys. Fortunately, I have also had the chance to experiment with a few forms of shorter and more entertaining surveys that convinced me that there are simple solutions within-reach. Yes, there are ways to make surveys shorter, better written, more entertaining. Some involve tools already available, some just involve time, talent and testing, or more sample. What we need is a way to convince clients to spend the extra dollar to use those techniques… Unfortunately, it is challenging to put a price tag on sustainability.
Steve Gentile says:
Peter, you raise some interesting points.
As someone who entered this field by way of documentary filmmaking and corporate communications, and as a writer and photographer, married a fine artist, and I ride a Harley, etc etc – I might say that although I may be one of the more creative types in this business, I have met some very right brained folks in my almost 20 years.
Who we are in background and experience and ethnography is what makes us what we are in whatever our vocation. It is our responsibility to stretch our arms and mind to the world, large and small. If the world were all creatives, we’d yearn for beancounters and they would be popular (i know heaven forbid).
In the big picture of our work, whether quantitative or qualitative, my encouragement and slant has always been to know your audience and build your house on that – a boring CPA will never “get into” a creative questionaire, whether on paper, on line, in person, or on the phone.
Your research is only as dull as you want to make it so if you don’t want dull, sharpen the pencil (or use a crayon). I have heard you speak several times Peter; you are not a dull guy in the least bit, so please take it easy on yourself.
Jeremy Bromberg says:
Besides agreeing, what I really like about Peter’s piece is that he (not personally) accepts responsibility for respondent cooperation and satisfaction. We’ve seen, including in prior topics introduced by Merrill, pointing at the respondents as a problem. This focuses on what we can and should control.
And for those who expressed concern above about bias, thoroughness, etc.: you are right to include these on the list of obstacles to be addressed. Hopefully folks will focus more on how to address those obstacles instead of simply accepting them as reasons not to try!
Bob Graham says:
“All researchers are nerds.” Speak for yourself, Peter. Such a blanket generalization does not begin to describe me and my colleagues. Yes, some certainly are, but there are also some of us who are creative and look for ways to communicate the findings and insights of the projects we work on rather than the intricacy of the data. Actually, I think it’s more accurate to say “All market researchers are expected to be nerds.”, and I use this perception as my secret weapon. One comment I have heard everywhere I have worked is “You’re not like other market researchers…” and I smile, for I consider this to be the ultimate compliment. As a matter of fact, it’s also my “secret weapon” for gaining involvement from my clients.
My background is not in statistics or the social sciences or business but in the Humanities. Throughout my career, I have met other researchers whose backgrounds are in Literature, History, Philosophy, etc. and I have found them to be among the most insightful colleagues I know, not hired guns, but thought leaders.
Creativity is fine, up to a point, but in research, as in all things, there needs to be a balance. Those of us who have worked for advertising agencies know this all too well, as we’ve watched creatives try to re-interpret the data to make it favor their ideas.
Being a good research leader (as opposed to a good researcher) involves a capacity to say “no”. To resist the charge to put everything into one test and focus instead on what issues we are trying to address or problems we are trying to solve. I have found that when goals and parameters are established at the beginning of the study and the survey instrument is written to focus on those issues, respondents don’t get bored. Respondents get bored when a questionnaire is long, unfocused, and contains questions about everything but the kitchen sink (and believe me, respondents can tell instantly when they are answering a poorly designed questionnaire). Clients get bored when a researcher (often a young and inexperienced one) presents the research and has a slide for every finding including 5 separate ones on the kitchen sink.
Christopher King says:
My favorite story is about a researcher in Hawaii who created an survey at a unattended stand-alone kiosk in the Honolulu Airport (this story is about 15+ years old). He was asking questions to tourists for the Hawaii Visitors Bureau. He added some music and moving hula dancers to the survey. After a week of collecting data, he was puzzled by the results. He finally sat down and observed his kiosk for a few hours only to discover that even though his screening questions required adults, it was children who were taking the survey and they were having a great time.
Additionally, today’s internet surveys are still affected by bandwith issues for many and the reluctance to download any client-side programs. There is a delicate balance between “being taken seriously” vs. too much graphical/gaming content vs. interview length vs. boredom vs. respondents that are awake.
We have been conducting several research on research studies looking at simple ideas such as inclusion of a picture of an interviewer during a survey and MaxDiffs questions instead of rank-order questions. Such techniques have improved respondent “satisfaction” as well as test / retest validity.
Professor Vicki Morwitz just presented a paper at this years ART (Advanced Research Techniques – AMA) on the value of priming questions to improve respondent attention. The more granularity of answers you give respondents in the first few questions can influence how careful respondents are in answering the entire survey.
I agree interviews must improve and engage the respondents. Sometimes simple measures can be effective. I have seen many studies where it is obvious that the questionnaire was put together by committee and every department put in their two cents. The result of such questionnaires is respondent abuse. We can do better as an industry.
Dan Womack says:
I’m catching up on several blog posts at one time so I’m going to try to combine my response to Today’s (Friday) post about 6 words with my response to this great topic:
We created it, we can fixit.
Okay, so “fixit” isn’t one word but count me in on any group that evolves from this. I think it’s a hurdle we can clear if we can get momentum.
I continuously preach simple and fun in my organization. We still have work to do, but in less than a year I’ve gone from fighting to reduce the length and complexity in the work we do to having internal clients first telling me that we have to “keep it under 10 minutes.” I know it’s working when I hear my words coming back at me.
Steve Runfeldt says:
“‘All researchers are nerds.’ Speak for yourself, Peter.”
This reminds me of one of my first visits to Silicon Valley as an online researcher. It was sometime toward the end of the last century. We were meeting with a client and describing how our process worked. I asked the client if she was online (in 1997 you had to ask), and proceeded to bring up our online demo.
She remarked, “So, you’re the geek.” I didn’t know how to take that.
That evening at dinner a friend asked how my meeting went that day. I told her about the “geek” comment and she exclaimed. “That’s great. This is Silicon Valley. The geeks are the ones who make all the money!”
I haven’t worried about being called a “geek” or a “nerd” ever since.
Peter Van Brunt says:
It just goes to show you, that once you are misquoted, you then are frequently misquoted. I never said “All researchers are nerds” If you scroll to the top of this, you will see what I wrote. If however, you would like to think that we all are nerds, feel free to do so.
I would like to thank all of you who have expressed a willingness to work on this issue. I am meeting with Ed Sugar in August about putting together a program/workshop, and will be in touch with those you indicated a willingness to help.
Bob Graham says:
I stand corrected on the “All researchers are nerds.” misquote, and apologize.
As for suggesting that I might want to believe that all researchers are nerds, I think I made it clear in my post where I stand on the matter. I do not even think that “most” (as you originally stated, Peter) researchers are nerds and that the best researchers certainly are not. I’m a bit disappointed that you chose to focus on the misquote rather than the content of the message which was written in good faith. It seems incongruous to me to call for creativity on the one hand, and nitpick on the other.
Furthermore, I take offense at your statement that “that once you are misquoted, you then are frequently misquoted”. I see only one reference to the statement after my original post. In my book “once” is not a synonym for “frequent”.
[…] Peter Van Brunt writes in his article Attention Researchers: Lets Make Market Research Fun that researchers need to focus on the experience of the respondent when designing surveys and make an effort to make the research process more enjoyable. […]