I've been thinking about the different ways to measure the effectiveness of content/text on e-commerce sites, and more specifically, how to select the best version if you have a variety of different alternatives in front of you, each with its own group of fans who want to get it on the site right away! Since the "Voice" of a web site can be such an abstract, arbitrary decision, how can we apply methodologically robust research methods to help make these decisions?
First, I would define "effectiveness" in this context as the optimization of the following 3 concepts:
- Do users understand what you are trying to tell them and what action they should take to be successful in their task?
- Are you invoking the desired emotions with your content?
- Does the proposed content result in higher conversion rates than other alternatives?
It's so important to combine the user perception data (the first two bullets) with business metrics (the last bullet). From my experience the only way for user experience researchers to affect change is if we can show the positive impact these changes can have on engagement/revenue metrics.
It seems to me that you will be well served by using the following 3 methodologies to measure the relative effectiveness of different versions of the same content. This is also a really nice way to progressively reduce the number of alternatives down to the best solution:
- Usability testing. Start with several different version of the content (~10), along with the current version (if it exists). Ask users in a lab setting what they understand the content to mean, and any other thoughts they have on the way it sounds. This should help narrow down the alternatives to 4-6 possibilities.
- Desirability testing. Use the Desirability method, but adjust it for use in large sample online surveys by turning it into a between-subjects experimental design. In the survey, users are asked to rate the content on different brand and design attributes. This way you can determine what emotional response the content extracts out of users. You'd also be able to ask users which version of the content they'd prefer, and why. This method has the added benefit of large numbers to give you confidence in the statistical significance of the results.
- A/B testing. Once you've narrowed the alternatives down to 2 or 3, live A/B testing can help you determine which of the alternatives perform better from a revenue or engagement perspective, by looking at differences that can be attributed purely to content changes. This obviously works easiest when the content is directly related to a revenue-generating task, like the call to action on a checkout page, for example. But it's not just about revenue -- there are great ways to measure metrics of engagement with the page, which is just as powerful.
Now, I can see 2 issues that make this a pretty difficult task, and it's the reason why the above 3 methods should not be used in isolation. In combination, they help tell the whole story.
- It is difficult to know what users really read on a page. In the first two methods you pretty much have to show people what to read -- that doesn't happen when they visit your site organically with no-one looking over their shoulder. This is why A/B testing is so important as it gives you a sense of how behavior will change based on content.
- It is difficult to isolate the effect of content changes from the other influencing factors on a page. This is the really difficult part. How do you know that conversion/engagement improved because of the content and not of some other factor on the page, like visual design changes? That is why it is important to keep the rest of the page exactly the same, and also why usability and desirability testing is important to bring out the perceptual data from users.
This is of course by no means the only way to do this, but I think it's a good approach that balances methodological rigor with the dangers of not overdoing it. I'd be curious if anyone has any thoughts or ideas on how to improve on this approach...
