A/B Testing Part 3 - Building a culture of experimentation

In my previous posts, I have talked about the organizational readiness and the technical preparedness to do online A/B testing effectively.  These baseline elements are foundational and need to be in place for any testing and experimentation approach to be successful.

The next stage is actually building a culture of experimentation and testing amongst product creators. There are a number of mental barriers to overcome. I have talked about the need for product managers to start appreciating the need for "testing in the wild" as a useful addition to any prototype or usability testing. Another mental barrier that comes in the way is the fear that a test might amount to nothing and therefore one shouldn't waste valuable dev cycles testing minor improvements and that testing should be "reserved" only for really big changes.

This is a place where it is important to have a schooling in developing product hypotheses and ways in which those hypotheses can be proved (or disproved). In my organization, we spent (and continue to invest) a considerable amount of time discussing the principles of testing and experimentation, and making sure that product managers walk away with a pretty good understanding of the overall "scientific method" - i.e. the need to develop and validate hypotheses through a systematic process.

We spent a good amount of time on the following questions

- What KPI or metric are you trying to influence?

So specifically, which important customer related or business related metric you are trying to impact? This is an important step to focus the experimentation effort on the things that really matter from a product standpoint. So to actually walk through an example, let us say one of the metrics we are going to impact is the "bounce rate" on a website.

- What is your hypotheses on how you can influence the metric?

What are the different ideas that can be employed to lower the bounce rate? What underlying consumer behavior are we trying to change here? By the way, a lot of these hypotheses need to be generated either from data analysis (so, something that shows that repeat visitors have high bounce rates) or from a detailed understanding of customer needs (through techniques like design thinking and empathy interviews). So one could hypothesize that one of the reasons why bounce rate is high is because our website does not effectively recognize repeat visitors on the site. Or that the reason why bounce rates are high is because of too much content on the page. Or that the call-to-action button needs to be of a different color and font to stand out from the page.

One other quick thing to point out. There might be situations where the purpose of the test is basically to generate new behavioral hypotheses and not to necessarily prove existing ones. So take a typical sales funnel with lots of form fields and a few call to action buttons. One could just come up with variants of the font used, the color of the button, the shape and size and test all of them to see which combination of form field size + font + button shape + button color is the most optimal. The results of the test could create a body of learning around what is preferred by customers, which in turn could influence other such funnels in future. This approach is also useful when there is some kind of new technology or feature to be introduced. So imagine doing a banking transaction on a wearable device like Google Glass. Given the newness of the technology, there isn't typically one proven answer and we need to get to the proven answer through experimentation.

- Finally, what is a test one could run in order to test out these hypotheses?

So specifically, what will the control experience look like and what would be the treatment? And by offering the treatment, what is the metric that we are expecting to move or impact? So in the bounce case example, the test could be

It is typical to spend a few weeks or even months just on this part of the journey - which is to inculcate a testing or experimentation culture within the organization. I do want to emphasize the need to get this culture piece right throughout the conversation. It is a known psychological quirk that human beings tend to be far more sure of things that are inherently uncertain. We are "sure" that a customer is just going to fall in love with a product we have built in a certain way, just because it was out idea. It is important to challenge this internal wiring problem, that can get in the way of true knowledge seeking.