A/B testing an ML-powered learning-to-rank algorithm

Personalisation is one of the most impactful applications of machine learning. In this article, we will explore the development and A/B testing of the ML-powered learn-to-rank (LTR) algorithm we helped one of our clients to build.

The solution was created for Secret Escapes, a leading members-only online travel deals company with millions of users worldwide.

Secret Escapes was looking to deliver personalised search results on their website. The goal was to create an LTR algorithm that prioritised search results likely to drive bookings, displaying the most relevant offers first.

This post delves into the A/B testing processes used to evaluate and refine the LTR solution.

Learn more about the foundation for building this ML-powered LTR in Part 1 of this blog post.
See the full case study on the Secret Escapes project.

A/B Testing for a learn-to-rank algorithm

Testing rationale

For the primary metric for the A/B test, we used Search-to-Book, which was the percentage of search sessions with a completed booking. The performance of LTR was going to be evaluated via a frequentist approach, with a two-proportion Z-test at the end of the A/B test to inform the launch decision.

We had two groups:

• Control — exposed to the existing search experience;
• Treatment group — exposed to the search results based on the LTR algorithm we tested.

By default, we assumed that the existing search experience (which the Control group would get) was comparable or better than search with LTR (which the Treatment group would get).

This formed the null hypothesis of our statistical test. The LTR would only be launched if data from the test proved that Search-to-Book for Treatment is higher than for Control.

Using a Minimum Detectable Effect of ~5% and baselines obtained over a two-month period, we estimated the sample sizes required for 80% power and 95% confidence level, which we used in order to determine the test duration.

Evaluating performance

One of the challenges was evaluating the performance of the learn-to-rank algorithm in different territories. Ideally, since we expect discrete territories to react to the Treatment differently, we should run the tests separately for each.

We also considered the success metrics for the experiments. Click-through-Rate (CTR) — the percent of searches with a result clicked — would have been the obvious choice. It would have given us larger sample sizes as compared to search sessions, and therefore higher test power.

However, here CTR does not measure the business goal directly, and we would not have had a clear quantification between CTR and Search-to-Book. This means that we would not have been able to rule out cases where the two metrics would not be moving in tandem.

Therefore, we decided to keep Search-to-Book as the success metric for the experiments and monitor CTR along with other guardrail metrics. We decided to evaluate the combined performance of the territories in the test, with additional monitoring of performance by territory in the post-analysis.

Additionally, despite originally estimating the test duration to be 4 weeks, we decided to run the test for 6 weeks, as it would allow us more time and more data to capture behavioural changes.

Test monitoring

To track the success and guardrail metrics for the A/B test, we set up a Tableau dashboard that would refresh daily. This would allow the team to take quick action if any of the metrics dipped drastically. Before launching the experiment, we agreed internally that we would stop the test if Search-to-Book in Treatment drops beyond 20% compared to Control.

Aside from monitoring business metrics, we created a Streamlit app within Snowflake (i.e. Streamlit in Snowflake) to test and monitor the sample ratio of the Control and Treatment groups. To ensure that the overall session count between Control and Treatment remains at 50-50, we applied a chi-square test. Any deviation from a 50-50 split would warrant a check on the assignment logic of the A/B testing platform.

A/B testing challenges

The rollercoaster of A/B testing

To call the experience a rollercoaster ride would probably be an understatement. However, the rounds of testing we went through are proof of how iterative the process of evolving a data science model could be and how many factors come into play when the algorithm goes live in production. Here, iterations encompass everything from how the model is developed, what it does or does not account for, how the infrastructure is set up, how it performs all the way to business context and initiatives that impact the live testing.

We launched four rounds of testing — let’s have a look at each of them.

Testing round 1

We started the first A/B test in the hopes of achieving the promising results that the offline metrics had shown. The test had been running for 2 weeks before we had to stop it when we registered several consecutive days of negative performance in the Treatment group against Control.

For context, the primary and secondary metrics (Search-to-Book, CTR and margin) were down overall, suggesting that the model was not performing well.
Improvements below were informed based on technical expertise of the team after 2 weeks of monitoring the model in production, rather than findings from data analysis.

Over these 2 weeks, we had identified room for improvement and we started working on that right away.

• Streamlining document searches: We made changes to allow a smooth transition from OpenSearch version 1.1 to 2.5. Our motto for this version change was “Know your data better”. The implication for us was the OpenSearch would no longer accept heavily nested queries, hence we had to streamline our document searches. This led to a decrease in feature logging time by 75%, which would allow us to reduce the model size by 10-15%.
• Tackling user intent: After careful observation, we noticed that we had a rare class issue. In the judgement list, the grade 3 events were close to zero, which meant the user had booking intent but was not going through with a transaction. We needed to sit down with the data team on the client side and identify a better data source for this.
• Feeding correct data: We had some insights into feeding the most accurate data to the model:
  • Same day ALS data was to be used during training, to ensure we used the feature values as they were on a given day. This historic data was not available for all past search events, so this change resulted in a smaller but a much more correct dataset;
  • Missing deals contributed to a 25-30% data loss, since deals were removed from OpenSearch if they had not been active for a long time. This meant that we could not score deal-related features for the documents in the historic search events. To solve this, we put a super index in place, which would retain inactive deals while upserting other ones.

Testing round 2

The second round of testing proved crucial, as Secret Escapes were running a large promotional summer sale campaign. They expected a lot of website traffic during this period, so we needed the model to perform well.
The model started off well and performed to our expectations. However, a few days had passed when it started underperforming, and we dedicated to stop the test.

We noticed that the Treatment group performed better in Search CTR, but Search-to-Book and margin remained low.

This suggested the improvements that we had implemented in the previous testing round had worked to a certain degree — we were able to attract more clicks, but they were not relevant enough to drive higher booking conversions or margin.

Here is what happened:

• We saw a huge increase in latency, which could have affected the user experience and, respectively, the conversion. We are going to explore this point in a minute. We knew an increase in latency would impact booking conversion, but it definitely would not explain all of it.
  The Search CTR was higher, but the booking conversion was lower. The higher Search CTR could be due to the model pushing cheaper deals to the top.
• Based on the findings above, we revised the features set, removing poorly performing ones, adding approximately 20 new ones, which were previously missing from the training data, and shrinking the 100-100 deal and the user ALS features into a single dot product. These new features proved absolute powerhouses, dominating the top 20 in the SHAP analysis.
  Finally, we also agreed we would be retraining the model on a daily basis to make sure it would be able to pick up on new trends. These enhancements proved highly effective in the results of the next testing round.

We knew an increase in latency would impact booking conversion, but it definitely would not explain all of it.

We had higher Search Click-Through Rate but lower booking conversion. The higher Search Click-Through Rate can be explained by the model pushing cheaper deals to the top.

However, the deals pushed to the top of search results were out of trend in context of the Summer sale. In addition, the increase in latency could have also impacted the user's booking experience.

Latency challenges

When we deployed the model to production for a second round of A/B testing, the platform team noticed a considerable increase in latency. The first test had an average query time of 650 ms; this time, it was around 2200 ms.

From a technical point of view, this might not have been a concern. However, slower query time tends to mean lower engagement and conversion, so we knew it could hurt the treatment group’s performance.

So, we set out to find what was increasing latency. At first glance, we had a few suspects:

• The code for the inference using the LTR model had not changed since the last A/B test;
• The clusters were not using the exact same machines, which would have explained a difference against prod in terms of performance, but not compared to the previous A/B test;
• We temporarily disconnected the dynamic update of sales to see if the additional load impacted the search;
• We re-trained the LTR models with different depths and on a smaller dataset;
• Increased cluster size.

Most of the steps above did not considerably decrease latency but we found the one that would solve the issue.

Performance vs complexity

We realised that since the previous A/B test, the number of search events used for model training had increased, expanding the model three times to around 70 MB, while using the same model training logic. This had resulted in a very deep and complex model with many branching trees.

We set out to tune the hyperparameters, looking to create a more compact model and sacrificing a small bit of the model’s complexity for performance. With this change, we were able to shrink the model to approximately 25 MB, while keeping a slightly lower but very similar uplift from the baseline.

When we applied this model in production, we registered a significant decrease in latency, averaging at around 380 ms.

Testing round 3

They say third time is the charm, but we beg to differ. The test had been running for almost 2 weeks and the model was slightly up overall, with Search-to-Book higher in Treatment, but Search Click-Through Rate and Margin being lower.

This is quite different from what we had before. The features and enhancements we had introduced in the previous round of testing had successfully optimised for booking conversion rather than clicks, but this would not translate to higher margin.

The issues we encountered highlight the challenges with operationalisation when introducing the changes in testing round 2, all of which proved huge learning points for the team.

We had observed alternating daily performance and huge fluctuations in Search-to-Book. While some level of fluctuation is expected, the alternating pattern suggested that there might be a bug.

It was at this point that we had a challenging weekend, where the difference between Control and Treatment fell under our pre-determined threshold.

During the test itself, we encountered the following two issues:

• Misalignment in the data logic used for training and inference;
• Instances where the processes in the end-to-end pipeline were triggered before downstream tasks had finished, resulting in the model not picking up new data. We set up explicit dependencies to ensure that no task would run before downstream ones finished;

We quickly had them resolved, after which the fluctuations lessened, but the alternating pattern persisted and this was concerning.

Here, we had to stop the testing for proper A/A test and further engineering investigations. Here is what we found:

1. A/A test
   Given the small magnitude of the Search-to-Book metric, high fluctuations were expected across two samples even without any treatment applied. We analysed the variance of differences in Search-to-Book across two samples, and came up with an action plan for daily monitoring. It included taking action only if we observed two consecutive days of underperformance beyond two standard deviations. This would prevent us from becoming too reactive to the results, and provide clear steps to take in different situations.
2. Engineering investigations
   The main engineering takeaway was related to observability, as we struggled with monitoring errors and detecting issues in the pipeline. After the test ended, we found a bug related to uploading features, which impacted inference and the quality of recommendations.
   We also caught some data indexing issues in the production cluster, where some fields had not been indexed with the proper values, while some other fields for the newly added features had not been mapped in the sales document. This resulted in the model pushing less relevant deals higher up the search results page. The model would push better matching deals as compared to previous testing rounds but, they were not delivering the same results as the deals which were manually curated and ranked by the CMS teams for the Control group.
   This is a classic human vs AI scenario, where the model has to catch up with the knowledge of the CMS team. The findings of our investigation confirmed that any bugs or issues in the pipeline greatly influence the model's ability to do so.

Testing round 4

We went live with the A/B testing for a fourth time, looking to strike the balance between features and performance.

To illustrate this task, let’s try to imagine machine learning as cooking.

Using a less complex LTR model is like making a simple dish using a basic recipe that calls for a few ingredients only. This dish might be decent and relatively quick to prepare, but it lacks the depth and richness that comes from more elaborate recipes.

Using a more complex LTR model with more decision trees is like preparing a gourmet dish using a detailed recipe with many ingredients and steps. Each ingredient is carefully selected and added at the right time to enhance the flavour and texture of the dish, creating a rich and satisfying dish.

While the gourmet dish provides a superior experience, it also takes more time and effort to prepare. This can be an issue in situations where time is of the essence, such as when you have many guests to serve and you need to do it quickly.

Hence, while a more complex LTR model with more trees can offer significant improvements in performance, it also requires careful consideration of trade-offs, particularly regarding inference speed. Just like choosing between a quick meal and a gourmet feast, the decision depends on the specific goals and any constraints that might come with the context.

Cooking the perfect model takes iterations. A lot of iterations, at that. In the image below, we have tried to demonstrate the iterative process we have been through when compiling the feature set (and hopefully training a high-performing model).

Key learnings

When adding new features, we would always retrain the model on the new data. It was as early as testing round 2 when we learnt that when it comes to evaluating offline performance, we should train and test the model using data from all 4 seasons if possible.

We would originally train and test the model on data from winter and spring (due to limited available data), and we observed very different online performance in the 2nd round of testing, as we were approaching summer.

Then, we would evaluate the NDCG metrics before and after each feature addition to see how that would affect the score.

Finally, we would do the SHAP analysis to see whether the newly added features were actually impacting the model performance in any meaningful way. Note that it is usually bad practice to include many non-relevant features, since it might cause some noise and can cause the model to expand in file size, which would also increasing latency, which would be the exact opposite of what we were after.

To go back to the cooking example, a carefully crafted dish of several, high-quality ingredients is better than one that has ingredients thrown in simply for the sake of it.

To sum up

We built an ML-powered learn-to-rank (LTR) algorithm to help Secret Escapes increase booking conversion rates and margin. And this is exactly what the solution did.

Let’s have a closer look at the combined results for the markets of interest, which we call X and Y.

Although the Treatment group experienced a slightly lower Search CTR (-1.4%), the results show a higher margin and an improved Search-to-Book ratio.
More bookings (+4.0%) and a higher margin per booking (+3%) lead to a margin increase. This indicates that focusing on improving Search CTR alone does not necessarily lead to better booking conversion or margin outcomes.

The results we got from regions X and Y align with the overall findings: higher Search-to-Book ratios and margins, alongside a lower Search CTR.

It is worth noting that region X shows stronger uplifts due to greater consistency in the performance of the Treatment group. Given that the X market is more price-sensitive, the LTR model prioritises deals with lower lead prices, even when discounts are reduced.

The LTR algorithm improves overall margin in three primary ways:

• Higher booking conversion within search sessions;
• Changes in user behaviour, resulting in increased booking conversion during non-search sessions;
• Greater spending per booking, i.e., higher margin per booking.

Across the test, we observed an overall margin uplift of +4.7%, attributed to the combined impact of these three factors.

View Part 1 of this blog post where we discuss building the foundation this ML-powered Learning-to-Rank algorithm.

Reach out to tell us about your project and what challenges you face with building ML-powered solutions. We are ready to help you move your project forward.