A/B Testing at Scale - Risks and Opportunities
WED 14:00 - 14:30
This is a 2022 presentation
Booking.com is known for running A/B tests at scale to optimise every small and big change to our products. Guiding product development by monitoring well-defined metrics allows dozens of teams to work on the same product simultaneously and iterate quickly. However, A/B testing methods can easily be misunderstood, unintentionally abused, or become a limiting factor.
In this talk, we present common pitfalls and shortcomings of classical A/B testing we've experienced at Booking, together with some concrete remedies and alternative solutions. After listening to this talk, you can go back to your teams and constructively challenge them.