We have been making a slow and steady journey towards A/B testing in my organization. No need to really explain the value or the need for A/B testing. Experimentation is quite simply the only way to understand causation from correlation - also the real only way to measure whether any of the product features we build really even matter.
In the past 12 months, we have had some important lessons about testing and importantly, the organizational readiness required before you go and buy an MVT tool and start running A/B tests at large. And these are:
1. Ensuring organizational and infrastructural readiness
2. Building a culture of testing and establishing proofs of concept
3. Continuous improvement from A/B testing at scale
In my opinion, the first and most important step is creating the baseline in terms of organizational and infrastructural readiness. Despite the best intentions of learning from testing, there can be a number of important reasons why testing just does not get off the ground.
A poor measurement framework is one such big reason. An online performance measurement solution such as Adobe SiteCatalyst is only as good as the attention to detail in implementation and the robustness of the implementation. In our case, though the overall implementation was useful in giving some good online behavior measurement, the attention to detail in ensuring every single action on the site was measurable was just not there. As a result, a few initial attempts at testing proved to be failures - meaning the test was not readable and needed to be abandoned. Not only was this wasted effort from testing. This was also a meaningful setback that re-inforced another belief in the organization, that testing is both risky and unnecessary for an organization that gets customer research and usability right. This brings me to the next part of readiness - which is organizational readiness. Which will be my next post.
In the past 12 months, we have had some important lessons about testing and importantly, the organizational readiness required before you go and buy an MVT tool and start running A/B tests at large. And these are:
1. Ensuring organizational and infrastructural readiness
2. Building a culture of testing and establishing proofs of concept
3. Continuous improvement from A/B testing at scale
In my opinion, the first and most important step is creating the baseline in terms of organizational and infrastructural readiness. Despite the best intentions of learning from testing, there can be a number of important reasons why testing just does not get off the ground.
A poor measurement framework is one such big reason. An online performance measurement solution such as Adobe SiteCatalyst is only as good as the attention to detail in implementation and the robustness of the implementation. In our case, though the overall implementation was useful in giving some good online behavior measurement, the attention to detail in ensuring every single action on the site was measurable was just not there. As a result, a few initial attempts at testing proved to be failures - meaning the test was not readable and needed to be abandoned. Not only was this wasted effort from testing. This was also a meaningful setback that re-inforced another belief in the organization, that testing is both risky and unnecessary for an organization that gets customer research and usability right. This brings me to the next part of readiness - which is organizational readiness. Which will be my next post.
No comments:
Post a Comment