Use A/B testing wisely - never in isolation

A recent discussion on a UX forum I participate in turned to the topic of A/B testing.  I really enjoyed the conversation so I wanted to reiterate some of the points I made, and expand on it a little bit as well.  It's not my goal to define A/B testing here but to share my opinion on its use.  I believe that even though A/B testing can be extremely valuable to help identify the best iteration of a site or a particular page, it should never be used in isolation.

Since A/B testing is relatively cheap to do and the results are so compelling, companies are in danger of adopting a "test and learn" culture where pages are just A/B tested with no additional user input.  That would be the wrong way to go.  A/B testing shouldn't be used on its own to make decisions, it should always be used in conjunction with other research methods -- both qualitative (such as usability testing, ethnography) and quantitative (such as desirability studies). 

A/B testing is an important method in the research toolkit because it can give you information that usability testing on its own cannot.  The main goal of A/B testing is to see how business metrics move up and down depending on the version of the page -- click through rates, checkout rates, purchasing rates, etc.  You can't see that with usability testing alone.  But as Kohavi et al. point out in their paper Practical Guide to Controlled Experiments on the Web, A/B testing has some major limitations:

• Quantitative Metrics, but No Explanations. It is possible to know which variant is better, and by how much, but not why.  In user studies, for example, behavior is often augmented with users’ comments, and hence usability labs can be used to augment and complement controlled experiments.
• Short term vs. Long Term Effects. Controlled experiments measure effects during the experimentation period, typically a few weeks.   It is wise to look at delayed conversion metrics, where there is a lag from the time a user is exposed to something and take action. These are sometimes called latent conversions.
• Primacy and Newness Effects. These are opposite effects that need to be recognized. If you change the navigation on a web site, experienced users may be less efficient until they get used to the new navigation, thus giving an inherent advantage to the Control. Conversely, when a new design or feature is introduced, some users will investigate it, click everywhere, and thus introduce a "newness" bias.
• Features Must be Implemented. A live controlled experiment needs to expose some users to a Treatment different than the current site (Control). The feature may be a prototype that is being tested against a small portion, or may not cover all edge cases.  Nonetheless, the feature must be implemented and be of sufficient quality to expose users to it.
• Consistency. Users may notice they are getting a different variant than their friends and family. It is also possible that the same user will see multiple variants when using different computers (with different cookies).

As with most things, it is important to use A/B testing responsibly.   Since every research/testing method comes with its own limitations, a combination of methods is the only way to get the full picture and make the right decisions.

iPhone shows that good design = good usability

TIME Magazine just crowned the iPhone as the best invention of 2007 (see Invention Of the Year: The iPhone).  I know what you're thinking, and I thought the same thing: can we please stop talking about the iPhone now?  Well, almost.  But let me just quote one thing from the article.  The author gives 5 reasons why the iPhone deserves this honor, and the first one is this (emphasis added by me):

The iPhone is pretty

Most high-tech companies don't take design seriously. They treat it as an afterthought. Window-dressing. But one of Jobs' basic insights about technology is that good design is actually as important as good technology. All the cool features in the world won't do you any good unless you can figure out how to use said features, and feel smart and attractive while doing it.

An example: look at what happens when you put the iPhone into "airplane" mode (i.e., no cell service, WiFi, etc.). A tiny little orange airplane zooms into the menu bar! Cute, you might say. But cute little touches like that are part of what makes the iPhone usable in a world of useless gadgets. It speaks your language. In the world of technology, surface really is depth.

I think Apple is showing us all the good usability can look great too.  They understand that it's not just about accomplishing your goal, but also about feeling "smart and attractive" in the process.  I think as usability professionals we often feel constrained by design, as if design stands in the way of making something truly usable.  Let's go beyond that -- let's keep the usability great and make it pretty...

Did the iPod's design evolution influence Apple's stock price?

The Wall Street Journal just published an interesting graph overlaying different iPod releases with Apple's stock price: 

Core77 Design Blog points to this as proof that good design can influence stock price.  This is what they say:

The clarity of their cohesive design language and its evolution over time is also instantly appreciated at a glance. If ever business needed confirmation that design can have a significant effect on your ROI, then this piece of infoporn is it.

At first glance I'd love to agree right off the bat, because I am a strong believer in the ROI of good design.  However, the researcher in me has to take a step back and ask if we can really say this based off this one chart.  As every researcher will tell you, correlation does not imply causality.  Just because 2 things happen at the same time, it does not mean that one event causes the other. 

Forgive me for stating the obvious, but there are hundreds of factors that influence the stock price, and we simply can't prove causality just by looking at this graph.  For example, how do we use design to explain the sudden drop in stock price between the iPod video launch and the redesign iPod shuffle launch?

Having said that, I do think that this poses an interesting question about what effect good user experience has on business metrics and revenue.  I guess if we can find a way to prove that beyond the shadow of a doubt, we won't have to spend half our time convincing people that good user experience is important...

Usability testing on Halo 3

Wired Magazine just published an article about usability testing on Halo 3, the much-anticipated next installment in the Halo video game franchise, and the first Halo game for Xbox 360.  Video game usability has always fascinated me (and I try to get hands-on experience in this area as much as I can...), and this is the first time I've seen a mainstream magazine cover it in such detail.

Halo is a genre-changing first person shooter that brought gaming to a new level with its intricate story-line, cinematic feel and epic soundtrack.  And the creators got there through endless hours of testing...  Here are some excerpts from the article showing how they left no stone unturned:

The room we're monitoring is wired with video cameras that Pagulayan can swivel around to record the player's expressions or see which buttons they're pressing on the controller. Every moment of onscreen action is being digitally recorded.

Midway through the first level, his test subject stumbles into an area cluttered with boxes, where aliens — chattering little Grunts and howling, towering Brutes — quickly surround her. She's butchered in about 15 seconds. She keeps plowing back into the same battle but gets killed over and over again.

"Here's the problem," Pagulayan mutters, motioning to a computer monitor that shows us the game from the player's perspective. He points to a bunch of grenades lying on the ground. She ought to be picking those up and using them, he says, but the grenades aren't visible enough. "There's a million of them, but she just missed them. She charged right in." He shakes his head. "That's not acceptable."

After each session Pagulayan analyzes the data for patterns that he can report to Bungie. For example, he produces snapshots of where players are located in the game at various points in time — five minutes in, one hour in, eight hours in — to show how they are advancing. If they're going too fast, the game might be too easy; too slow, and it might be too hard. He can also generate a map showing where people are dying, to identify any topographical features that might be making a battle onerous. And he can produce charts that detail how players died, which might indicate that a particular alien or gun is proving unexpectedly lethal or impotent.

Pagulayan and his team have now analyzed more than 3,000 hours of Halo 3 played by some 600 everyday gamers, tracking everything from favored weapons to how and where — down to the square foot — players most frequently get killed.

The article goes into many more interesting examples of how they solved user issues with clever design.  Be sure to check it out.  And if you haven't seen the Halo 3 trailer yet, here it is for your viewing pleasure...

Jesse James Garret on usability (and eBay)

The web site e-consultancy recently conducted an interview with Jesse James Garrett, the man who coined the term 'Ajax' and president of Adaptive Path. It’s a great interview and I wanted to highlight some of the things he said. First, on usability…

Usability doesn’t really get at the psychological and emotional context of use. Usability will tell you, from an ergonomic perspective, what people can do with a product, but there is lot more to making a product successful in the marketplace and making a product feel successful in people’s minds. Often, we find that clients come to us, thinking they have a usability problem, but it turns out that their products are pretty usable. The reason that the product is falling short is it is not satisfying an emotional or psychological need.

That is a philosophy that I wish can be top of mind for all designers and researchers -- what's cool isn't always what's best. You have to start by understanding the underlying user needs -- that's why I firmly believe in continuous user research throughout the design process. (I do think that very successful products can be created through genius ideas outside this formalized process -- iPod anyone??? -- but I'm referring to us normal people who need a little more help along the way to make our products work well...)

Then, on a question on if he agrees with a recent survey in which respondents rated Amazon, eBay and Google as the top 3 international sites in terms of usability, Jesse said the following…

It’s interesting to see Amazon and eBay so high on the list, because I think Amazon was delivering a really terrific experience a few years ago, but have found themselves in a land of diminishing returns in the design choices they are making.

If you compare the sheer number of navigational elements on a present day Amazon page with the way it was just a few years ago, they are just starting to load these pages up with features. I think the reason they are doing that is that they are trying to squeeze every drop of revenue they can out of these pages, but I think the overall usability is starting to suffer. It’s becoming so baroque - all of the different features and components they have loaded onto these pages.

eBay has almost the opposite problem, in that because they have this enormous community of people, the sellers, that depend on eBay for their livelihood, there are a lot of people that have really invested in how the site functions. eBay has been slow to change, because they haven’t been able to make changes that would appease this audience of millions of people that don’t want to see the site change.

My take on it is a little different... I think that if you ask regular Internet users about the usability of a site, they don't think of usability the way we do. They immediately jump to "how useful is it to me." And if a site is useful to them, i.e. it fills that underlying user need we talked about earlier, they will figure out a way to use it and make it work for them. This is not to say that bad usability doesn't matter -- good usability is essential for the sustainability of a site that fulfills user needs effectively. They go hand in hand and can't be separated. The point is that the value propositions of eBay and Amazon are so clear and so significant, and the sites so useful, that if you ask users about usability, they will immediately make the connections to these brands in their minds.