The keys to Amazon's success

The charmed life of Amazon's Jeff Bezos is a great article in the latest Fortune magazine about Amazon.com's success.  The key paragraph for me is the CEO's statement of their relentless focus on customer satisfaction -- even it means bad earnings for a few quarters:

For all of Amazon's ups and downs over the past 13 years, Bezos's strategy is one thing that hasn't changed. Customers want three things, he says: the best selection, the lowest prices, and the cheapest and most-convenient delivery. At Amazon, he explains, all decisions flow from those fundamentals. "What's not going to change over the next 10 years is incredibly important - you can build plans that are durable and meet important customer needs," he says, adding, "Ten years from now, customers will still want vast selection, low prices and fast, accurate delivery. In fact, it is impossible to imagine a world 10 years from now where customers will say, I love Amazon, but I just wish your prices would be higher."

A good example of this is Amazon's decision to build some excess warehouse capacity. As Amazon started to grow its business in the late 1990s, some members of the management team argued in favor of building just enough of the giant, automated warehouses - four of the $60 million facilities - to meet projected demand. Bezos decided to build five. "From a financial point of view, we should have built four rather than five," says Bezos, pointing out that in 1999, when the centers were built, "for a company that only had $1 billion in sales, spending $300 million on fulfillment centers is a very big investment."

Instead, he was positioning the company to pursue "more fundamental things": that is, keeping Amazon's customers happy. Remember all the post-holiday news stories at the turn of the millennium about little Timmy and Janie not getting their presents on Christmas because some fly-by-night toy site couldn't handle the holiday crush? Amazon came out smelling like a 1-800-FLOWERS rose. "A lot of companies stumbled and we didn't," said Bezos. "We had an insurance policy against that huge burst of demand" - the fifth fulfillment center.

Which leads up to the most important statement I think he makes:

"A lot of decisions around consumers are like that," Bezos says. "When you do the math it's not clear what will happen."

When you create a great user experience, the business will follow...

UX links of the week (4/10/08)

Interaction Design

Designing Office 2007
Jensen Harris, the guy responsible for user experience in the Microsoft Office group, delivers a near-comprehensive talk on how the Office 2007 UI was conceived. A must-see for everyone!

Sign Up Forms Must Die
You load a new web service, eager to dive in and start engaging, and what’s the first thing that greets you? A sign-up form. We can do better, says Luke Wroblewski, author of Web Form Design: Filling in the Blanks. Via a technique of "gradual engagement," we can get people using and caring about our web services instead of frustrating them (or sending them to a competitor's site) by forcing them to fill out a sign-up form first.

 

Content Strategy

Placing Value on User Assistance
User assistance writers are often the Rodney Dangerfields of the UX world, bemoaning the fact that we don’t get any respect. I think the real problem is that user assistance folks are not particularly good at communicating the ways in which we add value to an enterprise. This column explores two models that show how user assistance adds value and how we can communicate that value to those who pay our salaries—something I would like to encourage other user assistance writers to do.

 

User Experience Research

Extreme User Research
Clients don’t know a thing about their users, and designers think that if they like it, everyone will. Sound familiar? Daniel Lafreniere's 30-minute "extreme user research" plan comes to the rescue for those of us facing this exact situation. With this practical method, you can generate loads of useful data that will have a real impact on design, thus making the website more effective and profitable.

Usability testing: integrating eye-tracking and mouse clicks
Our usability test lab has a software environment allowing for capture, recording, analysis and full interpretation of observable events during a usability test. Moreover, since we are very interested in user behavior, verbal comments, facial expressions and eye movements, as well as keeping a record of keystrokes and mouse movement, we also use other software.

What Adaptive Path Thinks When It Thinks About Eyetracking
Recently, we had a discussion on an internal mailing list about eyetracking, specifically around why we didn’t use it as a research tool…

 

A little off topic…

I'm Over Twitter
I'm so over Twitter. I haven't wanted to admit it to myself, but a couple of things really tipped the scales for me. The first was a Newsweek article from 1995, which famously called the Internet a passing fad.  The fear of being the guy (or gal) is a big part of what drives the technology hype-machine.  Better to keep your mouth shut and appear stupid than to open it and remove all doubt. I feel like it's time to be brave enough to say what I really think.

Design for the Next Billion Customers
Niti Bhan and David Tait, who are specialized on research and strategy for emerging markets, recently collaborated with Experientia on an extensive ethnographic research project in Africa.   Niti and Dave condensed their broader insights in what it means to design for emerging markets in a long article for Core77.

Is the Internet just a fad? Ask 1995!

There's nothing like a little trip down memory lane every once in a while.  And when I recently stumbled upon this 1995 Newsweek article that called the Internet a fad, I couldn't tear myself away from it.  To be honest, my first though was, "Man, this guy must feel stupid for writing this."  But after thinking about it for a while, I realized that the author had a 50/50 chance of being right.  Things could've gone the other way -- a few smart people could've decided to focus their energies elsewhere, and this Internet thing would have never become what it is today...

Below are a few excerpts -- but I encourage you to read the whole article, it really makes you think twice before speaking in absolutes...

After two decades online, I'm perplexed. It's not that I haven't had a gas of a good time on the Internet. I've met great people and even caught a hacker or two. But today, I'm uneasy about this most trendy and oversold community. Visionaries see a future of telecommuting workers, interactive libraries and multimedia classrooms. They speak of electronic town meetings and virtual communities. Commerce and business will shift from offices and malls to networks and modems. And the freedom of digital networks will make government more democratic.

Baloney. Do our computer pundits lack all common sense? The truth in no online database will replace your daily newspaper, no CD-ROM can take the place of a competent teacher and no computer network will change the way government works.

Then there's cyberbusiness. We're promised instant catalog shopping--just point and click for great deals. We'll order airline tickets over the network, make restaurant reservations and negotiate sales contracts. Stores will become obsolete. So how come my local mall does more business in an afternoon than the entire Internet handles in a month? Even if there were a trustworthy way to send money over the Internet--which there isn't--the network is missing a most essential ingredient of capitalism: salespeople.

To the author's credit -- stores have not become obsolete...

Brand Loyalty and the User Experience

I recently attended a brand presentation where the video below was shown. It’s pretty funny, and also (in my opinion) a perfect example of how interactive products and consumer-generated content should fundamentally change our traditional views of customer loyalty. Loyalty in our current environment is fostered through repeated great (user) experiences, not through advertising and coupons…

But even though I like the general point the video is trying to make, I think it stops a little short of the real issue. It is really just saying that we should listen to our customers better. But that's not enough -- we need to understand the customer in ways they don’t even understand themselves, and then build experiences that meet unmet (and sometimes unconscious) needs through repeated, positive experiences that deepen the customer-company relationship.

Uncovering these needs happens not through "Voice of the Customer" research programs, but through more contextual research efforts like ethnography and contextual inquiries (combined with validating quantitative research). I believe this is where traditional Market Research programs have historically fallen short -- although there is evidence that the tide is turning on this topic as HCI becomes more mainstream and user experience research techniques become more accessible.

In my view there can be no more powerful synergy in discovering how to deepen true customer loyalty than a collaborative effort between Market Research and User Experience Research. This view is very much in line with the thinking described in the Adaptive Path blog essay The Long Wow, which I have referenced before.

So in essence, my viewpoint is this: To not realize how important repeated, quality user experiences are to Loyalty would be an egregious misjudgment of what Loyalty really is about with an interactive product.

Netflix - more homework than entertainment?

As Netflix continues to be down today, I'm reflecting on the panic I felt when I saw this message on their site this morning...

What does this mean?  What happens to my queue that I spent so many hours perfecting?  As pathetic as it might seem, I had to calm myself down a little bit...  Anyway, it reminded me of a great article I read a while ago called "Netflix Guilt".  It's about how asynchronous entertainment (the ability to watch what we want when we want without being attached to a TV schedule) might actually be a move in the wrong direction, turning "entertainment" into "homework".  For example:

The digital revolution has introduced us all to the life-altering phenomenon known as asynchronous entertainment. We can now enjoy movies, TV shows and our favorite media sources wherever, whenever we want. But a decade into this monumental shift, the drawbacks are coming into focus. Episodes of “The Daily Show” and “Letterman” pile onto our DVR television recorders like copies of The New Yorker, begging to either be consumed or wastefully discarded. Netflix movies line up on our shelves like airplanes on a runway waiting to take off. And all of those blog postings relentlessly flood into our Web browsers every hour, every day. There’s certainly not time for all of it. Is this entertainment? It feels more like homework.

So as I wait to see if my queue is still going to exist tomorrow, I encourage all of us to reduce the number of entertainment homework we do.  You know, just to preserver our sanity in case a site goes down or our DVR's stop working...

Use A/B testing wisely - never in isolation

A recent discussion on a UX forum I participate in turned to the topic of A/B testing.  I really enjoyed the conversation so I wanted to reiterate some of the points I made, and expand on it a little bit as well.  It's not my goal to define A/B testing here but to share my opinion on its use.  I believe that even though A/B testing can be extremely valuable to help identify the best iteration of a site or a particular page, it should never be used in isolation.

Since A/B testing is relatively cheap to do and the results are so compelling, companies are in danger of adopting a "test and learn" culture where pages are just A/B tested with no additional user input.  That would be the wrong way to go.  A/B testing shouldn't be used on its own to make decisions, it should always be used in conjunction with other research methods -- both qualitative (such as usability testing, ethnography) and quantitative (such as desirability studies). 

A/B testing is an important method in the research toolkit because it can give you information that usability testing on its own cannot.  The main goal of A/B testing is to see how business metrics move up and down depending on the version of the page -- click through rates, checkout rates, purchasing rates, etc.  You can't see that with usability testing alone.  But as Kohavi et al. point out in their paper Practical Guide to Controlled Experiments on the Web, A/B testing has some major limitations:

• Quantitative Metrics, but No Explanations. It is possible to know which variant is better, and by how much, but not why.  In user studies, for example, behavior is often augmented with users’ comments, and hence usability labs can be used to augment and complement controlled experiments.
• Short term vs. Long Term Effects. Controlled experiments measure effects during the experimentation period, typically a few weeks.   It is wise to look at delayed conversion metrics, where there is a lag from the time a user is exposed to something and take action. These are sometimes called latent conversions.
• Primacy and Newness Effects. These are opposite effects that need to be recognized. If you change the navigation on a web site, experienced users may be less efficient until they get used to the new navigation, thus giving an inherent advantage to the Control. Conversely, when a new design or feature is introduced, some users will investigate it, click everywhere, and thus introduce a "newness" bias.
• Features Must be Implemented. A live controlled experiment needs to expose some users to a Treatment different than the current site (Control). The feature may be a prototype that is being tested against a small portion, or may not cover all edge cases.  Nonetheless, the feature must be implemented and be of sufficient quality to expose users to it.
• Consistency. Users may notice they are getting a different variant than their friends and family. It is also possible that the same user will see multiple variants when using different computers (with different cookies).

As with most things, it is important to use A/B testing responsibly.   Since every research/testing method comes with its own limitations, a combination of methods is the only way to get the full picture and make the right decisions.

iPhone shows that good design = good usability

TIME Magazine just crowned the iPhone as the best invention of 2007 (see Invention Of the Year: The iPhone).  I know what you're thinking, and I thought the same thing: can we please stop talking about the iPhone now?  Well, almost.  But let me just quote one thing from the article.  The author gives 5 reasons why the iPhone deserves this honor, and the first one is this (emphasis added by me):

The iPhone is pretty

Most high-tech companies don't take design seriously. They treat it as an afterthought. Window-dressing. But one of Jobs' basic insights about technology is that good design is actually as important as good technology. All the cool features in the world won't do you any good unless you can figure out how to use said features, and feel smart and attractive while doing it.

An example: look at what happens when you put the iPhone into "airplane" mode (i.e., no cell service, WiFi, etc.). A tiny little orange airplane zooms into the menu bar! Cute, you might say. But cute little touches like that are part of what makes the iPhone usable in a world of useless gadgets. It speaks your language. In the world of technology, surface really is depth.

I think Apple is showing us all the good usability can look great too.  They understand that it's not just about accomplishing your goal, but also about feeling "smart and attractive" in the process.  I think as usability professionals we often feel constrained by design, as if design stands in the way of making something truly usable.  Let's go beyond that -- let's keep the usability great and make it pretty...

Broken Experience #4

I don't even know where to begin on this one...  This form presented itself to me when I opened my browser to get Internet access at the hotel I'm staying at.  Here are the list of things I can see wrong with it -- feel free to add if I missed anything...

• Broken image at the top
• "Please choose from the following plans", but there's only one "plan"
• Why is there a Price column?  And why, if it's free, is it only valid for 24 hours?
• "Enter number of days... as long as it's 1!"  What ???  Why is that text box there if I can only enter a "1" in it?
• "Please leave this box blank."  Why is it there if I'm not supposed to use it?

A completely redundant form -- if it's free, just let me in :)

 

Creating loyalty through great customer experiences

Brandon Schauer from Adaptive Path wrote a great article last week called The Long Wow, about how to create customer loyalty through repeated great customer experiences.  It encourages companies to go beyond just measuring loyalty (through overly simplistic measures like Net Promoter Score) or instituting "loyalty programs", and spend time to create experiences that delight customers again and again.  To quote from the article:

True loyalty grows within people based on a series of notable interactions they have, over time, with a company’s products and services. No card-carrying programs are necessary: Apple doesn’t have a traditional loyalty program; neither does Nike or Harley-Davidson. These companies impress, please, and stand out in the minds of their customers through repeated, notably great experiences.

As I read through the article (and you should too!), a couple of recent experience came to mind of how 2 completely different companies either made me loyal from the start, or increased my loyalty by doing something small to help me out.

 

1.  Running Revolution

Nothing is more important for a runner than shoes.  After sticking with New Balance shoes for years, I recently decided to try out something new.  But since I didn't know where to start, a colleague recommended Running Revolution, a small boutique shop close to where I live.  I became a loyal customer long before I even bought my shoes there.

These guys get runners.  There are no prices on the shoes -- for them it's about finding the right shoe for you, and nothing else.  They measured my foot with thermals, brought out pairs of shoes that they knew would be a good fit for me, put me on the treadmill and videotaped me to measure bio-mechanics, and I can go on and on.  Suffice to say that I didn't walk out with the most expensive shoes in the store, but I will never buy running shoes anywhere else again.  They made a life-time customer simply by fulfilling my need for a great running shoe.  They know that I will come back again and again, so they don't have to sell me the most expensive shoe in the store right then and there.

 

2. Amazon.com

Even though I'm already a loyal Amazon customer, the repeated great experiences keep strengthening that loyalty.  Small things count.  While browsing around, I noticed that the Overnight 1 Click button was disabled.  When I scrolled over the button, I got this message:

How nice and and completely unnecessary of them!   They could've gotten an extra $3.99 shipping cost out of me, but instead they refused to let me pay extra.

 

I'm sharing this as examples of what Brandon points out in his article -- that through a continuous focus on user needs and building great experiences, you create loyal customers that are so much more valuable in the long run than if your only goal is to squeeze as much revenue out of them as you possible can, right now.  Unfortunately Wall Street doesn't believe in this unique brand of economic "delayed gratification", but I wish more companies would just start doing it and prove them wrong...

Can a social network become too large?

An article in the latest issue of The Economist explores the fact that there might be less to Facebook and other social network sites than meets the eye...  The basis of their argument is as follows:

[There is an] an important limitation for social networks, such as Facebook, compared with older sorts of network, such as the postal or telephone systems. These benefit from Metcalfe's Law, which says that the value of a network is proportional to the square of the number of its users. In other words, the more people have phones, the more useful they become. This “network effect” leads to rapid adoption and puts up barriers for new entrants.

But unlike other networks, social networks lose value once they go beyond a certain size. “The value of a social network is defined not only by who's on it, but by who's excluded,” says Paul Saffo, a Silicon Valley forecaster. Despite their name, therefore, they do not benefit from the network effect. Already, social networks such as “aSmallWorld”, an exclusive site for the rich and famous, are proliferating. Such networks recognize that people want to hobnob with a chosen few, not to be spammed by random friend-requests.

This suggests that the future of social networking will not be one big social graph but instead myriad small communities on the Internet to replicate the millions that exist offline.

Although I agree that social networking sites are not perfect at the moment, I don't think the reason is that they are too large, as The Economist suggests.  I want to fall back on one of my earlier posts, where I mentioned the importance of "weak ties" between people in separate network clusters to facilitate the flow of information in that network.  If the future of social network sites is really in "myriad small communities on the Internet" that are not connected in any way, each network in isolation will become pretty useless over time, since there will be no new information coming into or going out of that network. 

In order to have value (or social capital), a network needs to be (1) large, (2) open and (3) have (weak) ties between different clusters in that network.  Alexander van Elsas and Rolf Skyberg writes a lot of about the need for open social network sites, and I agree with their views and their thoughts on how this can be implemented.  The article in The Economist is an interesting read and it does raise some serious questions about the monetary value that is placed on Facebook and other sites at the moment, but I just don't think that smaller, isolated networks is the right way forward.

Visualization of quantitative data -- bad examples

I'm a big fan of and RSS subscriber to Smashing Magazine -- they have some great articles on design, and I always find myself reading all the way through most of their posts.  With that said, I have to voice my disappointed in their recent article on chart and graph generators.  It's probably not even their fault really -- I applaud the hard work they did on pulling together a bunch of Flash, AJAX and CSS based chart generators that are available online.  But the fact is that most of these tools are completely useless unless you know how to visualize data properly, and that is, sadly, a skill that is not taught enough.

All of these tools create pretty flashy charts, that’s for sure, but does it really help to tell the story of the data?  Why would you ever need a donut chart or a 3D stacked cylinder chart?  What does that add to the data that a simple 2D bar chart can’t show you?

If you're not familiar with Edward Tufte's work, you should definitely check it out.  He coined the term "data-ink ratio", in which he argues, to quote from Wikipedia, "against the inclusion of any non-informative decoration in visual presentations of quantitative information and claims that ink should only be used to convey significant data and aid in its interpretation."  Below are some examples, from the Smashing Magazine article, of how "non-data ink" is so overbearing that it completely overshadows the data:

There are so many things wrong with these charts, but let me just point out the 3 main issues really briefly:

• Unnecessary usage of area.  The eye is not good at comparing the relative sizes of areas.  On the first graph, can you easily tell if France or Canada is bigger?  We are good at comparing lengths though, which is why bar charts are almost always a better option than pie charts.  (And by the way, what on earth is the hole for on that first chart?).
• Unnecessary and incorrect usage of color.  In the first chart, different colors are assigned to each country.  Since this should have been a bar chart and not a pie chart (as per my first point), different colors aren't needed, and just adds non-data ink to the graph.  If you're going to use color, then don't use highly saturated colors as in these graphs -- it's uneasy on the eyes and in many cases indistinguishable to people who are color blind.
• Unnecessary usage of 3D.  3D effects should just never be used, period.  It clutters up the charts, and also often results in occlusion -- where some data points are hidden behind others.

We need to teach analysts the techniques to create simple and straight-forward charts that let the data shine through.  All these primary colors and 3D stuff have nothing to do with the data.  A great resource on this is Stephen Few's book Show Me The Numbers -- I highly recommended his book to anyone who spends any time making charts for business presentations.  Ok.  I'm glad I got that off my chest...

Measuring the effectiveness of content on e-commerce sites

I've been thinking about the different ways to measure the effectiveness of content/text on e-commerce sites, and more specifically, how to select the best version if you have a variety of different alternatives in front of you, each with its own group of fans who want to get it on the site right away!  Since the "Voice" of a web site can be such an abstract, arbitrary decision, how can we apply methodologically robust research methods to help make these decisions? 

First, I would define "effectiveness" in this context as the optimization of the following 3 concepts:

• Do users understand what you are trying to tell them and what action they should take to be successful in their task?
• Are you invoking the desired emotions with your content?
• Does the proposed content result in higher conversion rates than other alternatives?

It's so important to combine the user perception data (the first two bullets) with business metrics (the last bullet).  From my experience the only way for user experience researchers to affect change is if we can show the positive impact these changes can have on engagement/revenue metrics.

It seems to me that you will be well served by using the following 3 methodologies to measure the relative effectiveness of different versions of the same content.  This is also a really nice way to progressively reduce the number of alternatives down to the best solution:

• Usability testing.  Start with several different version of the content (~10), along with the current version (if it exists).  Ask users in a lab setting what they understand the content to mean, and any other thoughts they have on the way it sounds.  This should help narrow down the alternatives to 4-6 possibilities.
• Desirability testing.  Use the Desirability method, but adjust it for use in large sample online surveys by turning it into a between-subjects experimental design.  In the survey, users are asked to rate the content on different brand and design attributes.  This way you can determine what emotional response the content extracts out of users.  You'd also be able to ask users which version of the content they'd prefer, and why.  This method has the added benefit of large numbers to give you confidence in the statistical significance of the results.
• A/B testing.  Once you've narrowed the alternatives down to 2 or 3, live A/B testing can help you determine which of the alternatives perform better from a revenue or engagement perspective, by looking at differences that can be attributed purely to content changes.  This obviously works easiest when the content is directly related to a revenue-generating task, like the call to action on a checkout page, for example.  But it's not just about revenue -- there are great ways to measure metrics of engagement with the page, which is just as powerful.

Now, I can see 2 issues that make this a pretty difficult task, and it's the reason why the above 3 methods should not be used in isolation.  In combination, they help tell the whole story.

• It is difficult to know what users really read on a page.  In the first two methods you pretty much have to show people what to read -- that doesn't happen when they visit your site organically with no-one looking over their shoulder.  This is why A/B testing is so important as it gives you a sense of how behavior will change based on content.
• It is difficult to isolate the effect of content changes from the other influencing factors on a page.  This is the really difficult part.  How do you know that conversion/engagement improved because of the content and not of some other factor on the page, like visual design changes?  That is why it is important to keep the rest of the page exactly the same, and also why usability and desirability testing is important to bring out the perceptual data from users.

This is of course by no means the only way to do this, but I think it's a good approach that balances methodological rigor with the dangers of not overdoing it.  I'd be curious if anyone has any thoughts or ideas on how to improve on this approach...