User>Driven

The consulting firm Booz Allen Hamilton does a study of innovation each year at the largest corporations in the world hoping to describe what works and what doesn't. In their 2006 study I think they tried a little too hard to add value by breaking the successful strategies they observed into different categories. All of them really came down to intelligently listening to your market, but they nonetheless presented some good principles and at least one good story of corporate User>Driven behavior.

One key principle they uncovered was that it was more important what process you use to develop products and services than how much you spend on it. "As in years past," they said "we found no statistically significant connection between the amount of money a company spent on innovation and its financial performance." What they found instead was getting close to customers was the key thing.

"It's engineers and marketing product managers spending hours and hours on job sites talking to the guys who are trying to make their living with these tools," said John Schiech, President of the DeWalt division of Black & Decker, one of the subjects of the study.

DeWalt has grown from $150 million to over $2 billion in sales of their power tools since 1991 by listening closely to current customers. In field studies, for example, they observed that builders couldn't cut large crown mouldings in one pass using traditional 10" miter saws. They brought the first commercial 12" miter saw to market, priced it a premium, and quickly found themselves with a best-seller.

Plantronics also practices market listening through focus groups and shadowing of both their corporate and consumer telephone headset customers. BAH had no specific tales of successful innovation at Plantronics, but described their process of applying "strategic filters" to help choose which products to bring to market. Those filters are all related to the expected financial performance of the products over the next 1-3 years.

The largest of these companies, Siemans AG, uses these same market listening techniques in their individual divisions but adds a greater layer of oversight, allowing them to pick and choose where to invest based on heir perception of longer term market trends such as the movement to personalized medical care, the need for portability, urbanization and other changes in demographics.

BAH tries to cast these companies in different roles, saying they are driven by customer needs, market needs and technical innovation, respectively. I don't see that in these examples. I think each company is actively researching and responding to market needs and the differences are really just in the time-horizon each is planning for. It should be no surprise that the larger companies are looking farther out.

In the Siemans study, BAH does quote one executive saying "You've got to be somewhat skeptical of what they [customers] see as the technical solution, and instead depend on your own core set of people who can creatively link new technology to the future market." BAH seems to think this is unique to Siemans and other technical innovators thinking long-term. This is no different than what DeWalt did with the miter saw, though. DeWalt's customers didn't ask for a bigger saw, DeWalt observed the current practice of cutting each piece of molding twice and came up with a solution that saved their customers' time. It may have been a more obvious extension of a current product, but the process of innovation in that case, like the others, was driven by observing unmet needs in the market.

This study just reinforced for me a few key principles of good product design:

• Pick your target market, both in terms of whom you are targeting and when you plan to bring the product to market
  
• Determine the needs of that market, ideally by studying first-hand the problems they face or that you believe they will face in your target time-frame
• Design your product to solve those problems in a way that can be brought to market affordably and profitably

Thanks to Steve Johnson at Pragmatic Marketing for the pointer to a concise Wired Wired summary of a Cornell University study (which they apparently picked up from The Atlantic - yes, that is a lot of references) revealing just how bad Americans are at math.

Because we tend to use precise numbers for small amounts and round off very large numbers (lots of zeros), [the study suggests] sellers can actually con consumers into thinking a price is smaller than it is by replacing those zeros with other digits. - from Wired's summary

The effect apparently goes beyond the familiar phenomenon of 99 cents looking more than a penny less than a dollar. When large numbers are involved (think home prices), apparently $249,673 looks like less than $249,000. The study suggests this because our brains are wired for precision with small numbers and rounding off with large ones. Ergo, precise numbers must be smaller, right? (The Cornell folks claim to have eliminated alternative explanations such as precise numbers signaling unwillingness to negotiate.)

The authors then examined more than 27,000 real-estate transactions on Long Island and in South Florida and discovered the same effect at work in real-life deals. In South Florida, having at least one zero at the end of the list price lowered the final sale price by about 0.72 percent compared with houses listed at a similar price, three zeros lowered it by 0.73 percent, and each additional zero lowered it another 0.39 percent. - from The Atlantic's summary

Bear that in mind when you are selling your next house or pricing your product. 0.72% of a $400,000 sale is enough for a couple of plasma TVs for the new abode.

Of course it should be no surprise that Americans are bad at math. Las Vegas would be just another dry, hot southwest town if that weren't so and state governments wouldn't have the lottery to generate revenue from. 

Last week I attended a webinar on developing personas put together by User Interface Engineering. Jared Spool was a bit distractible but explained things in a very approachable way. He described IUE's research methodology for creating personas, which involves 6-12 site visits, extensive note taking, analysis, persona development and incorporation of the personas into the development process.

Here are a couple of take-aways I thought you might find interesting.

Personas Are Not Fiction

A while back I wrote a piece about personas where I talked it about why it's a bad idea to base your personas on a single real individual. Using a real person as a substitute persona might seem like a good idea since you know that everything about that user is for real. The problem, though, is that one individual comes with quirks that may not be representative of the market as a whole. If you interview enough people, you can eliminate their biographical peculiarities by focusing your personas on the characteristics common to the group.

Jared also cautioned against adding a lot of biographical detail to personas, but he came at it from the opposite direction. He warned us to watch out for the temptation to make up background info on your personas. He did recommend giving your persona a name (though not the name of a real research subject for the reasons above) and even finding a reasoanble-seeming photo of that person from stock photography. Both of these are to make the persona more real, easier to remember, and easier to talk about. He said, though, that the color of their eyes, the kind of snacks they like or other details that don't relate to the product you are trying to design are just distractions. He argued that you should eliminate any detail from the persona that you can't connect with a design decision you need to make.

I asked about his inclusion of age as a detail in an example persona they were showing for a medical application and he reasonably replied that age, if representative of the real people you interviewed, could tell you something about the level of computer-sophistication of the user. He emphasized the word "representative," saying that otherwise you were "just making stuff up," and that didn't make for useful design tools.

See a great response Jared posted on his blog to a rant by Jason Fried of 37Signals about personas being "artificial, abstract, and fictitious." Jason sure missed the mark here and I was glad to see so many responders taking him to task.

Clusters Lead to Personas

Another thing I took away from the seminar was a method of extracting data from interviews and turning it into personas. I usually take a lot of notes during customer interviews. When I am finished with them all, I go through and look for key data points like reports people want to see, goals they have, frustrations they have, tools they use, etc. Then I count up how many people use each tool, for example, to get a picture of what people as a whole are using. This is not a scientific, quantitative approach, but it's often directionally helpful.

The folks at UIE have developed a method of clustering these data points about their subjects (they call them "informants" but that's a little too CIA for me) that helps them develop multiple personas. They compare and contrast the individuals they interviewed and plot each on a series of sliders. They pick a subject they interviewed them on, say, computer-savviness, and they plot all the users relative to each other. They do this for all aspects they can then look for patterns in the charts where individuals seem to be close together on several sliders. Looking at what these clusters of neighbors have in common is how you then generate the descriptions of personas.

One of the key things this method does is to weed out irrelevant detail. It might be that all of the neighbors in a particular cluster are young while other clusters seem to skew older, so that's a good bet for a detail to be included in the persona. It might also be that the majority of your subjects overall drive Fords but if you find there is no consistency as to automotive brand choice within clusters or little contrast across clusters, then auto brand preference is probably not relevant to the personas you are going to write up.

I find I like the rigor this clustering concept seems to bring to the qualitative process of developing personas and I plan to try it out on my next project. Hope it's useful for others as well. 

Jacob Nielsen published an excellent article way back in 2000 suggesting as few as 5 users were enough to adequately test a design. He shows data suggesting you gain 85% of the information it's possible to gain in user testing of a single design with 5 users and that more users provide less and less unique feedback. After that, he says, you might as well revise the design and start with 5 new users. Revise one more time and test with 5 more and you probably have as good a design as you are going to get.

I liked this article not just for it's pragmatic approach to research but also because I think it shows how easy and approachable doing product research really is. Companies often shy away from market and user research, I think, because it sounds intimidating. You need to sponsor customer advisory boards, organize focus groups, compose elaborate questionnaires, and so on, it seems. That sounds hard, takes a lot of time and money and requires expertise. Well, I'll agree with one of those.

In fact, you can learn a lot by just putting your designs in front of a handful of customers and asking them to try them out. You can also call up a half dozen customers and ask them about their biggest frustrations (within your are of expertise) and get a lot of information to feed into your feature prioritization.

I do think you need some expertise to do this right, however. As I have argued recently, someone in the organization must be dedicated to listening to the market - to customers, potential customers and users - to find out what the opportunities are for solving problems and whether proposed solutions are likely to solve them. These people can't be salespeople or sales support people. They will usually be product managers and design or usability professionals.

Once you've invested in hiring these people, though, you don't usually need an enormous research budget to support them. You can get the qualitative input you need with just a few customer interactions. (Though quantitative market sizing data may take a little more effort - see my recent entry on this.)

Is 5 subject enough in your experience? What's the fewest number of subjects you've used and felt like you met your goals? What is the largest number you ever used and why? Post your comments below. 

How many times have you given up on taking a survey because none of the multiple choice answers provided fit your situation or because the questions themselves were based on assumptions that didn't apply to you or because they failed to ask the questions that were most important to you? And why do some surveys seem to go on and on, asking question after question to the point that you want to quit?

The answer is that many surveys are misused. They are used as all-purpose feedback tools - and often they are the only feedback tool a company uses. A lot of companies do an annual customer satisfaction survey. (Some also do an employee satisfaction survey.) They look at this as their one and only chance per year to get good customer input so they load the survey with every question every department can think of.

And no question can ever be dropped or changed from the annual survey, of course. That would break the chain of comparability between results of different years. Participation drops every year, but we can make that up with incentives, reminders, and by sending it out to pester ever more customers each year.

Nobody knows what to make of most of the data, but we get an overall customer satisfaction score that hovers in the 80s every year and that sounds pretty good so we must be doing okay.

These familiar situations arise when well-intentioned people try to use a quantitative tool like a survey before they've gathered enough qualitative information. The surveys don't work because the people designing them fundamentally don't know what to ask.

Quantitative Research

Quantitative research involves measuring things. The tools used for this include surveys, web analytics, data mining, modeling statistical analysis and predictive analytics. Concrete things like conversion rates, revenue, lift, demographics and even customer satisfaction can be reduced to numbers and compared to industry norms, past metrics, goals, etc.

Quantitative methods are perfect for getting at the nuts-and-bolts what, where, when and how many questions needed to measure your progress against your goals, competitors or standards. Hence the annual customer satisfaction survey. There are even statistical methods that can help you prioritize features based on the value different customer segments place on them and how much they change their likelihood to buy. Sounds good, right?

But how do you decide what your goals are in the first place? What good does it do you to measure something if you don't know what to measure, why you are measuring it, or what to do with the resulting numbers? And how do you know the list of features you want your customer survey to help you prioritize has the killer features your customers really want in it? And worst, what do you do with this nagging feeling you are missing some critical insights into how your customers think?

Qualitative Research

Qualitative research is all about exploring. The tools used for this include focus groups, interviews, on-site shadowing, and even playing games with your customers. The result is not numbers but lists of insights about customer needs, goals, features, obstacles and values, as well as descriptions of personas, tasks, processes, relationships, mental models and value propositions.

Qualitative methods get at the why and how questions you need answers to before you can formulate the right quantitative questions. The most familiar example I can think of from the world of product management is feature prioritization. Doing a number of interviews of focus groups can help you generate a list of feature ideas that respond directly to the problems you see your customers having in the real world.

This kind of exploratory research where you are trying to learn about your customers' lives or businesses and hearing about or observing their problems first hand is where you discover needs you never knew about and where really innovative product and feature ideas are born. You might find out they don't need more features at all but better documentation or training or support or whatever. You might find out they think your product is worth way more than you charge. You might find out they couldn't care less about the new product idea you are about to spend a year developing. Without the kind of insight that comes from getting to know individual customers and how they think you can't find the hidden nuggets, the ideas that will really make your product a must-have.

Okay, now you've done the interviews or focus groups or onsite visits you needed to do to really put yourself in your customers' shoes. You've learned all about how they think and you have a lot of ideas about products or features that could solve the  most painful problems they face every day. How do you know which of these brilliant new insights are the most valuable, the ones that will drive people to buy, the ones you should implement first?

And Back Again 

That's where the survey comes in. Once you have the proper list (of features, in this example), you can do your survey and quantify how many potential customers in which segments are motivated by which of these features. Good statistical analysis of a properly executed survey can help you size the market for different products or features. You might find, for example, that feature X is valued by nearly everyone in your target market but valued below some other features that are different for different groups. That might suggest you need different product versions or add-ons for each group.

This qualitative before quantitative approach works with the annual customer satisfaction survey as well. Before you do your first survey, bring some customers in for informal discussions to explore their priorities and needs. Then you'll be able to develop a survey that measures whether you are satisfying their needs and not just your need to brag about customer sat numbers. 

Innovation Games

I attended a seminar last night sponsored by the Boston Product Management Association (BPMA) on a an interesting (and fun) method of qualitative research called Innovation Games. Mara Krieps of Pivotal Product Management walked us through a dozen different games you can play with your customers to learn more about them, their values and their need, and get valuable feedback on your product.

Drop by the PM Exchange Forum to learn more and share thoughts about games you can play with your customers.