I think it’s safe to say that no matter what the research project objectives are, one thing that every custom research project has in common is respondents. I think it’s also safe to say that every client WANTS and NEEDS the right respondents for their research project. It doesn’t matter if it is a qualitative or quantitative study, they want the right respondents. That’s for sure. They are paying for and deserve respondents that qualify, are truthful and rely on companies to ensure that their partners’ or vendors’ quality standards are very high. People who don’t know our industry that well always question the quality of respondents. Since I have been in the industry for over 25 years I know many people who have quality front and center of everything they do and really would like for them to share some best practices with readers of the blog.
Ok now that I have bored you a little bit on the obvious, the question is, "What do data companies really do in terms of quality?"
- How do panel companies recruit new panelists?
- What traps are they putting in place to catch people who might be falsifying data?
- How do focus group facilities ensure that Fred Smith is really Fred?
- How do they make sure that the recruiter isn’t leading the respondent?
- How do telephone interviewers get trained to make sure that they are following the screener/questionnaire exactly the way the client wants them to?
- What are the quality standards for coding and translation companies?
I look forward to hearing from Chris D, Craig S, Steve S, Roseanne L, Amy S, Todd B, Harriet N, Ann T.B, Carl I, Michael H, Ed S, Peggy O and many, many others–all of whom I have known for many years and have a lot of quality standards in place.
Michael Halberstam says:
Hi Merrill,
ISA has many different business units including telephone data collection. Everyone in the company, from data collectors to senior executives to programmers, begins with the same initial training. That starts with a thorough review and understanding of our Vision , Mission and Values which are posted around all of our offices. More specifically, these new employees go through our 10 hour data collector training program. In fact, several of our clients send their new research employees through our training so they can get in touch with the telephone data collection process. We offer this to all clients. Everyone must do live calling before they graduate.
To address your specific question, the training program, which is now over 27 years old and incorporates many of the MRA and other association guidelines, goes into great detail on following instructions. Classes are small and are very interactive.
Follow up via constant monitoring, recording of surveys and periodic additional classes are standard practice.
Amy Shields says:
While I (like many of my friends and colleagues) could write a book on ethical recruiting practices, many of the quality control practices that Advanced Focus applies are based on the sound standards and guidelines recommended by all of the prominent associations, with specific processes and policies that go above and beyond. One bad apple really can spoil the bunch and it’s the responsibility of all data collectors to uphold high standards that promote the legitimacy of the profession as a whole. Nothing upsets me more than to hear about (or witness first hand) unethical recruiting.
What I personally believe makes the biggest difference is hiring the right people (from the ground up) and, more importantly, obtaining buy-in, beginning with the employment interview. I agree with Michael about employees understanding the company mission, but also believe the same for potential employees. A significant part of our interviewing and training process is focused on the INDUSTRY; the important role we play; the active role our company takes in protecting and promoting the profession and how unethical recruiting practices undermine that effort; what the ramifications for our clients, our company and our profession are when proper processes are not carried out, etc. I think that when an employee takes pride in what they do, who they work for, and the industry that they serve, their output in automatically one of higher quality. Of course, this initial process does not replace the thorough training program we have, or the quality control measures that we take before/during/after recruitment; it’s just something that adds legitimacy to the important role of our recruiters in the research process.
Anne Brown says:
Hi Merrill,
While we have quality checks in place in all aspects of our business, let me comment on the following:
On line surveys undergo extensive testing pre launch, we soft launch and then edit and clean the data to look for errors. Once we go full field, we look at length, straight lining, a trap question and inconsistencies.
In coding, we use a state-of-the-art coding system that permits instant access of supervisors to check coders work.
For translation we use a three step process (translator, editor, proofreader) with teams industry or business sector specialized.
Steve Schlesinger says:
I would love to add a bunch of original thoughts and ideas but basically i need to echo what Amy said before me. If you lean on the best practices set forth by the leading industry associations and couple that with excellent hiring and training, you deliver a quality product/service to your clients.
To ensure quality, the key is validation. For qualitative research validation comes from screening, rescreening during confirmation, ID authentication and then a final rescreening on-site. Along with these steps comes the monitoring of recruiter/respondent interactions by supervisors. With these measures in place, quality is ensured.
peggy O'Connor says:
We have a Quality Control Department that works out of our corporate headquarters and reports directly to the Vice President of Client Services. Using our proprietary Stealth Monitoring capability this group is completely independent from operations. They monitor every single interviewer at least twice a month. They can monitor any interviewer in any facility at any time without a supervisor being aware of it. This helps with our system of checks and balances.
Open end/other specify editing – all open ends and other specifies are reviewed by our editing department. They work closely with the phone center supervisors to ensure the interviewers are probing for clarity and additional information as well as using the other specify options correctly. Their feedback helps floor supervisors zero in on specific individualized trainings.
We specialize in recruiting to web. The Supervisor at the completion of the recruit (and prior to hanging up) verifies the qualifications and willingness to participate in the Web interview.
Our supervisors use hand held portable phones and can stand by the associate and help them during the interview. It also allows them to cut in if appropriate.
We record every interview so we or the client can spot check the quality. If a problem comes up it allows us to review the interview in question. We keep the recordings for 90 days after job completion.
Malcolm Williamson says:
You can have the highest quality respondent in the world and if your screener is bad, don’t waste your time and money.
Craig says:
Merrill -
Thanks for covering this important topic in your blog. I could write pages and pages on this. But for now, I’d like to take the discussion in a different direction. I could comment on all of the tools, processes, etc that my company uses to ensure data quality (which are significant), but I believe the discussion needs to extend further into the end-client community. We need to help end-clients understand that the quality of services provided to end-clients by researchers (and suppliers to full-service agencies) is not the same across the board. Many agencies and suppliers are investing in quality, while others are not. I believe that end-clients DO care about quality, but not all are aware of some of the key “nuts-and-bolts” elements that drive quality. By offering further education of those quality-driving elements in the various aspects of our businesses, end-clients will increasingly differentiate between high-quality and low(er)-quality services. And one would hope that, when faced with an educated choice between high quality at a slightly higher price versus known lower quality and a lower price, the client will place value on the quality work and better understand the value of those services. This, in turn, allows suppliers to continue to invest in high quality services which is good for our industry at large.
OK, I’m stepping off my soap box now. I look forward to others’ thoughts on this.
Carl Iseman says:
Merrill…I concur with all that my colleagues before me have said…I would only add that along with setting high standards for hiring recruiters and continually evaluating the quality of work they produce is the need to not over-use one’s database. Especially today with the economic climate being what it is, marketing research can be an important adjunct to one’s income. Because of this we’ve always placed strict limits on the number of times we’ll contact respondents within our database, whether they qualify or not. Constantly purging and adding new respondents to the database gives our clients fresh consumers who’ve not be “trained” by responding to multiple screeners.
Finally, as Malcolm said, an excellent screener is a precursor to excellent recruiting. We work closely with our clients to make certain that the screener will yield the expected result ..it’s our partnership responsibility!!
Melanie says:
He, Merrill. I agree with everything said before me, especially about writing pages on how we address quality and with Steve’s comment about validation, but the main point I’d like to add is that quality is more than a set of tools or a few processes and techniques. It’s a way of business – a continual mindset that is always looking for weaknesses and problems and then crafting solutions for fixing them. As an industry, we’ve figured out a lot over the last few years and I’m proud of all we’ve accomplished, but things are changing every day and we still have a lot to learn.
My favorite quality quote is by William Foster and I’ve use it in presentations for years: “Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives.”
Chris DeAngelis says:
Merrill –
A lot of good thoughts on this have been shared already. I think we all agree that respondents are an important ingredient in research quality and a primary stakeholder in the research process. Finding them where they are, when they wish to participate and keeping them engaged are a challenge everyone shares. To my way of thinking, representation is the heart of the matter. Research quality depends on a diverse set of respondents that are more than just demographically and geographically representative.
For online surveys the veracity of the respondent in terms of them being who they say they are can be addressed with third party validation processes. The respondent can be compared with other participants in the survey using digital fingerprinting and similar technologies to confirm the respondent is unique within the survey. The design of the survey, the respondent burden involved all affect the outcome based on how the respondent engages with the event. We can design checks and implement other best practices to make sure fatigue and satisficing are minimized if not eliminated.
There are so many aspects of research that impact quality: Finding the right respondents for proper representation, verifying they are who they say they are, ensuring they are unique, controlling their survey experience to keep it engaging and thanking them and rewarding them intrinsically or extrinsically depending on their motivations to make sure they as respondents know they are part of the value that’s created by research and will continue to respond truthfully and enthusiastically to future surveys which ask for their thoughts.
Chris D.