This is a condensed version of my talk for the Bulgaria Web Summit, 31st May 2014. I spoke without real notes so the following is simply the main points.
Introduction
How should a UX team respond to the results of randomised control trials of their designs? Such “A/B” and “multivariate” tests hold out the possibility of finding the best approach from a number of alternatives. But what does this mean for the practice of design itself? Hotels.com has had one of the most advanced experimentation programmes of any website in the world. As manager of the UX Design team there, it was my job to makes sense of quantitative data and apply that to an overall design approach. What did we find out along the way? And in a business infused with numbers, what problems did we face as designers, and how did we show that traditional design methods were still valid?