This is a guest blog post by Experimentation Manager, Sophie Harpur. Sophie has been working towards making the complex world of experimentation accessible to everyone at Skyscanner for the past two years. She loves her role of fostering the experimentation culture at Skyscanner, if you are developing experimentation at your company and are interested in the an experimentation sharing session with Sophie, please get in contact.
At Skyscanner we have a strong experimentation culture – from conducting quasi growth tests, to A/B product experiments. We are always excited to find new ways to challenge our assumptions, and continue to make data driven decisions to optimise offering: for travellers and our partners. Throughout this article I will highlight some of the actions and the best practices we have adopted to drive this and raise the bar of our experimentation standards.
Insight Based Hypotheses
At Skyscanner, we create our hypotheses and test designs based on genuine insights rather than assumptions and "seeing what sticks". These insights could be based on user research or secondary research within the industry. This is particularly effective because our organisational structure enables us to conduct research on a market level, and ensures that we optimise our product for local preferences, requirements and standards.
We also make our hypotheses as robust, testable and falsifiable as possible. This ensures that we have a clear view on what we are trying to prove/ disprove, and know what success looks like. Taking any other approach would be a waste of time, revenue and resources.
We have Quality Experimentation Tooling
At Skyscanner, we are fortunate to have a dedicated in-house experimentation team. To ensure we are conducting effective experiments, they developed our very own A/B experimentation platform: ‘Dr Jekyll’.
Investing in our own experimentation platform, and a team which can provide guidance on experimentation has delivered some important outcomes:
It has given everyone across the business the opportunity to get to the heart of our products and test assumptions
It has increased the frequency of testing – and number of impactful and valuable changes that we are making as a result
It has ensured a consistent standard of data when running experiments, and reduced the result of Type 1 (false positives) and Type 2 (false negatives) errors in our experiment results.
As a result of reduced errors, we have been able to invest in experiment-based optimisations with greater confidence.
We are Transparent with Our Learnings and Analyses
Skyscanner offices now spans across several offices and time zones and sharing our learnings and failures across the regions is a key part of our culture. In experimentation a fail is as valid as a success. A failed experiment can provide insights that you may have not thought of before, and can make you one step closer to a win. Therefore, we have processes in place to ensure everyone across the company can access these insights and use them to optimise or inspire their own experiments.
We Blend Moonshot and Incremental Testing
At Skyscanner we aim to balance the “moon-shot” type tests with incremental and continuous improvement. Iterative testing conducted at scale is the main way that we improve – but if we never leap into the unknown then we will only ever make an improved version of what we already do. Or to put this into an analogy, without “moon-shot” tests, we will never invent the automobile. We will only make faster horses! There are many ways to define innovation, including basic research, disruptive, sustaining and breakthrough innovation. To simplify all definitions out there, Skyscanner defines innovation at 3 different levels in growth;
NOW: previously ran experiments, incremental innovation or optimization in existing channels, e.g., travel trends, best time to book, testing different targeting in ads
NEW: idea/ concept/ experiment that is new to Skyscanner.
NEXT: new concept/ technology in the industry.
Skyscanner strives to be in all 3 groups; innovators, early adopters and the early majority.
Interested in advertising with Skyscanner?
Further reading on the topic :
Check out the live webinar recording of Chad Sanderson, Sephora on Experimentation vs. Optimisation here