Measuring the live impact of design changes on key business metrics is valuable, but often creates a focus on short-term improvements. This near-term view neglects bigger issues that only qualitative studies can find.

A/B testing has more problems than benefits. You should not make it the first method you choose for improving your site’s conversion rates. And it should certainly never be the only method used on a project. Qualitative observation of user behavior is faster and generates deeper insights. Also, qualitative research is less subject to the many errors and pitfalls that plague quantitative research.

A/B testing does have its own advantages, however, and provides a great supplement to qualitative studies. Once your company’s commitment to usability has grown to a level where you’re regularly conducting many forms of user research, A/B testing definitely has its place in the toolbox.

Putting A/B Testing in Its Place