Qualitative Before Quantitative Research
Friday, June 22, 2007 at 08:01PM How many times have you given up on taking a survey because none of the multiple choice answers provided fit your situation or because the questions themselves were based on assumptions that didn't apply to you or because they failed to ask the questions that were most important to you? And why do some surveys seem to go on and on, asking question after question to the point that you want to quit?
The answer is that many surveys are misused. They are used as all-purpose feedback tools - and often they are the only feedback tool a company uses. A lot of companies do an annual customer satisfaction survey. (Some also do an employee satisfaction survey.) They look at this as their one and only chance per year to get good customer input so they load the survey with every question every department can think of.
And no question can ever be dropped or changed from the annual survey, of course. That would break the chain of comparability between results of different years. Participation drops every year, but we can make that up with incentives, reminders, and by sending it out to pester ever more customers each year.
Nobody knows what to make of most of the data, but we get an overall customer satisfaction score that hovers in the 80s every year and that sounds pretty good so we must be doing okay.
These familiar situations arise when well-intentioned people try to use a quantitative tool like a survey before they've gathered enough qualitative information. The surveys don't work because the people designing them fundamentally don't know what to ask.
Quantitative Research
Quantitative research involves measuring things. The tools used for this include surveys, web analytics, data mining, modeling statistical analysis and predictive analytics. Concrete things like conversion rates, revenue, lift, demographics and even customer satisfaction can be reduced to numbers and compared to industry norms, past metrics, goals, etc.
Quantitative methods are perfect for getting at the nuts-and-bolts what, where, when and how many questions needed to measure your progress against your goals, competitors or standards. Hence the annual customer satisfaction survey. There are even statistical methods that can help you prioritize features based on the value different customer segments place on them and how much they change their likelihood to buy. Sounds good, right?
But how do you decide what your goals are in the first place? What good does it do you to measure something if you don't know what to measure, why you are measuring it, or what to do with the resulting numbers? And how do you know the list of features you want your customer survey to help you prioritize has the killer features your customers really want in it? And worst, what do you do with this nagging feeling you are missing some critical insights into how your customers think?
Qualitative Research
Qualitative research is all about exploring. The tools used for this include focus groups, interviews, on-site shadowing, and even playing games with your customers. The result is not numbers but lists of insights about customer needs, goals, features, obstacles and values, as well as descriptions of personas, tasks, processes, relationships, mental models and value propositions.
Qualitative methods get at the why and how questions you need answers to before you can formulate the right quantitative questions. The most familiar example I can think of from the world of product management is feature prioritization. Doing a number of interviews of focus groups can help you generate a list of feature ideas that respond directly to the problems you see your customers having in the real world.
This kind of exploratory research where you are trying to learn about your customers' lives or businesses and hearing about or observing their problems first hand is where you discover needs you never knew about and where really innovative product and feature ideas are born. You might find out they don't need more features at all but better documentation or training or support or whatever. You might find out they think your product is worth way more than you charge. You might find out they couldn't care less about the new product idea you are about to spend a year developing. Without the kind of insight that comes from getting to know individual customers and how they think you can't find the hidden nuggets, the ideas that will really make your product a must-have.
Okay, now you've done the interviews or focus groups or onsite visits you needed to do to really put yourself in your customers' shoes. You've learned all about how they think and you have a lot of ideas about products or features that could solve the most painful problems they face every day. How do you know which of these brilliant new insights are the most valuable, the ones that will drive people to buy, the ones you should implement first?
And Back Again
That's where the survey comes in. Once you have the proper list (of features, in this example), you can do your survey and quantify how many potential customers in which segments are motivated by which of these features. Good statistical analysis of a properly executed survey can help you size the market for different products or features. You might find, for example, that feature X is valued by nearly everyone in your target market but valued below some other features that are different for different groups. That might suggest you need different product versions or add-ons for each group.
This qualitative before quantitative approach works with the annual customer satisfaction survey as well. Before you do your first survey, bring some customers in for informal discussions to explore their priorities and needs. Then you'll be able to develop a survey that measures whether you are satisfying their needs and not just your need to brag about customer sat numbers.
Innovation Games
I attended a seminar last night sponsored by the Boston Product Management Association (BPMA) on a an interesting (and fun) method of qualitative research called Innovation Games. Mara Krieps of Pivotal Product Management walked us through a dozen different games you can play with your customers to learn more about them, their values and their need, and get valuable feedback on your product.
Drop by the PM Exchange Forum to learn more and share thoughts about games you can play with your customers.
Reader Comments (3)
qual can be used really effectively to follow up quant research too - a great way to explore those tricky to explain findings and work out what people were thinking when they answered the questions (cognitive testing)
You might want to take a look at Dedoose. It's a very new web-based tool for qualitative research and analysis and I think right up your alley. It does a spectacular job integrating quantitative data for mixed methods analysis as well. Check out the video's and give it a try for free over at Qualitative Research and Analysis using Dedoose
thanks for sharing.. I like this, because it related to my study