Skip to content 
ML-driven personalisation to boost conversions
Secret Escapes
We built a Learning-to-Rank algorithm to offer members a personalised order of deals when they searched the website or viewed a page.
Secret Escapes is one of the world’s leading members-only online travel deal companies, offering inspiring travel experiences and hand-picked offers to over 75 million users worldwide across their Group.
Secret Escapes continues to grow, with a dedication to giving members the best range and value in luxury travel deals at accessible prices.
ML-driven personalisation
Learn-to-Rank algorithm
A/B testing
The Challenge
Last year, Secret Escapes’s Data and Data Science teams were looking to leverage Machine Learning to increase revenue generated through the website.
The opportunity they wanted to explore was to give members a personalised order of deals when they performed a search or viewed a page on the website.
Hypothesis to prove
Secret Escapes had the hypothesis that if they showed their members deals that were more relevant for them, conversions would increase. The hypothesis was based on other personalisation models that the in-house team had built over the years that had proved highly successful.
For this project, the goal was to build a Learning to Rank (LTR) algorithm, which is a class of supervised ML algorithms. This would enable the client to personalise the search results for their members, the aim being a 5% increase in conversions.
Why Infinite Lambda and not in-house?
The data science team were already working on projects with equal and/or greater potential upside. Hence, in order to go at speed with Learning to Rank, they chose to partner with Infinite Lambda to get a product in under 3 months.
Project overview
Phases
Phase 1: Building a Proof of Concept (PoC) on the client’s tech stack and demonstrating it works as expected by measuring offline metrics;
Phase 2: Putting the PoC into production and launching an A/B test with success metrics to decide whether to go 100% live or develop further and iterate.
Objectives and key results
- Design and build a functioning PoC showing how a ranker search model can be deployed on the website for A/B testing;
- Track offline and online metrics to measure performance;
- Execute A/B testing on a functional PoC roll-out for selected territories to test and validate the key hypothesis;
- Improve search conversion rates for members in the test regions (region X and region Y).
Defining offline metrics
The client team prioritised defining success and the mechanisms to measure it for the offline model. This was the only way they could make a decision to go live with it and proceed with A/B testing.
They considered the following metrics:
Cumulative Gain (CG)
Sum of relevance scores of items returned
Discounted Cumulative Gain (DCG)
Similar to CG but prioritises higher ranked items
Ideal Discounted Cumulative Gain (IDCG)
DCG of the ideal ranking of items
Normalized Discounted Cumulative Gain (NDCG)
DCG/IDCG
Secret Escapes defined their offline metrics by Normalised Discounted Cumulative Gain (NDCG), which is a common metric in ranking problems in ML literature. Unlike simpler metrics that only consider if an item is relevant or not, NDCG dives deeper to assess how well a model ranks items based on their graded relevance.
To learn more about NDCG, see the article on the Infinite Lambda Blog.
THE SOLUTION
Existing Secret Escapes tech stack
Our starting point was Secret Escapes’ existing tech stack because it provided a great foundation for the development of a new Learning to Rank (LTR) model to be used for searches on the website.
The stack included:
- Cloud provider: AWS — S3, OpenSearch (Elasticsearch), SageMaker
- Data warehouse: Snowflake
- Data transformation: dbt
- Orchestration: Airflow
- CI/CD: Jenkins
Learn to rank algorithm
In Elasticsearch, the LTR framework uses machine learning to make search results more relevant.
Normally, Elasticsearch ranks results based on factors like keyword matching. LTR adds an ML model that looks at different aspects of items and user queries. This model then adjusts the ranking, placing the items it finds more relevant at the top.
In a nutshell, Elasticsearch gives an initial ranking based on fixed rules, but LTR improves this by considering what users actually find useful, based on ML. The result is a more detailed and user-friendly search experience.
LTR judgement lists
Judgement lists serve as a reference for LTR models. They show ranked search results for a specific query, where the rankings typically range from 0 (completely irrelevant) to 4 (perfectly relevant).
For each query, the judgement list might assign a grade the following way:
- The exact match would get a score of 4, indicating it is highly relevant;
- A match that is not perfect but still relevant might receive a grade of 2;
- An item with no relation to the search query would likely receive a score of 0.
You can create the list manually or automatically. We chose to use dbt with predefined grading guidelines. By scheduling jobs in dbt Cloud, we ensured the data would be up-to-date, and we would not need to perform manual updates.
By incorporating the creation of this list into daily pipelines with proper testing helped, we could spot data issues early on. We also integrated it with Slack, so data analysts would get a notification about potential problems.
As the model would be trained on fresh data daily, it was crucial to maintain a clean, up-to-date judgement list for training.
To create the judgement list for the algorithm, we reviewed past search sessions on the Secret Escapes website and graded each search result based on user interaction:
4 = Completed the booking (conversion)
3 = Opened the booking form but did not complete the booking
2 = Clicked on or viewed the deal
1 = No interaction with the result
LTR feature sets
A feature can be anything that helps our model rate each document, such as keyword matches in fields like title, rating or description.
Building the optimal feature set is an iterative process that requires analysing how each feature impacts the model's predictions. By carefully selecting and refining features, we can guide the LTR model towards more accurate rankings. We ended up using nearly 200 features for model training.
Some features answer deal-related questions, such as: Does this hotel have a spa? Is it pet-friendly? Does the per-person price fall in a given range? How relevant is this document to the keyword “London”?
Others include more user-related information, such as: How active has this user been in the past days? How likely are they to book again?
With these features we had already achieved an uplift from the baseline, but to reach the next level, we needed a way to extract more information out of user behaviours than of explicit features.
This is where factorisation, and in particular Alternating Least Square (ALS), came into the picture. Alternating Least Squares (ALS) is a method that breaks down the user-item interaction data into two smaller sets of data: one for users and one for items. This helps to identify hidden patterns that explain how users interact with items.
We stored the user and item data from ALS in an Elasticsearch index, which would allow us to access the values easily when needed. During inference, when the actual prediction is made, we would perform a dot-product operation to calculate how likely a user is to interact with a particular item or deal based on past behaviour.
SHAP analysis
SHapley Additive exPlanations (SHAP) analysis is a method used to understand how individual features, such as price, or location, affect the predictions made by a model. It breaks down the prediction for each case (e.g., a user or a deal) and shows the contribution of each feature to that prediction. This helps to explain why the model made a certain decision, making it easier to understand and trust the model.
In the Secret Escapes project, SHAP Analysis would help us understand what made people more likely to book a deal. It showed us how different features, like price, hotel rating or hotel features, influence someone when making a booking.
Leveraging SHAP, we could understand which features were the most important for getting people to book and whether a feature made a hotel seem more or less attractive. This way, we could see if things like highlighting free breakfast or showcasing great reviews had a big impact on bookings. By understanding these influences, we could make changes to the website to showcase the features that would really drive conversions (bookings).
In the SHAP analysis, we showed that the ALS dot product was the most important factor in improving results, more so than any other feature. This is because the ALS method effectively captures how users interact with items.
After creating a judgement list and scoring documents based on our selected features, we combined these two pieces of data. We used this final table to train the model, helping it understand which features influence deal conversions and how they do so.
The model would learn these patterns and be able to make predictions for new deals a user has not seen.
In the iterative process of model training, we would apply SHAP Analysis with every run to check how each feature would impact the overall performance of the model.
Automated re-training
We used a Jupyter Notebook, running either in SageMaker or locally, for development because it offered interactive feedback and good plotting. However, to ensure reliable and repeatable re-training, we needed a structured codebase to manage the process.
To do this, we modularised most of the training code so it could be used by both the automated pipeline and the notebook. This allowed us to maintain the code in just one place.
To run the modularised code in the automated pipeline, we used Typer, an open-source Python library, to quickly create a Command Line Interface (CLI) application. Each command could then be used as a Docker entry point.
Since we would be training the model daily, we needed an orchestrator for this task. Secret Escapes already had an Airflow instance, and we used that. Once the Airflow Directed Acyclic Graph (DAG) finished running, the updated model would be uploaded to AWS S3, ready for other processes to pick up and load into the OpenSearch cluster.
A/B Testing
Testing rationale
A/B test design
For the primary metric in the A/B test, we used Search-to-Book, which is the percentage of search sessions where a booking is completed. To evaluate the performance of the LTR algorithm, we used a common statistical method called a frequentist approach. This approach involves using a two-proportion test at the end of the test to compare the results of two groups: one with the old search method (Control) and one with the new search method (Treatment).
By default, we assumed that the old search experience (Control) was at least as good as the new method (Treatment). This assumption formed the null hypothesis — the idea that there is no improvement or that the old method is better. The new method would only be adopted if the test results showed that it performed better than the old method.
To make sure our test was reliable, we calculated the Minimum Detectable Effect (MDE), which in this case was about a 5% improvement, and looked at data from the previous two months. This helped us figure out how many users we needed in the test to be 80% sure of detecting a real difference, with a 95% level of confidence. We also used this to determine how long the test should run.
Evaluating performance
One of the challenges we faced was evaluating the performance of the LTR algorithm in different regions. Ideally, since we expected different regions to respond differently to the new method, we should have run separate tests for each region. However, one region had low traffic, so running separate tests would have meant the experiment could take up to 12 weeks.
We also considered the success metric for the experiments. Click-Through Rate (CTR), which measures the percentage of searches where a result is clicked, would have been a good choice — it would have allowed us to work with larger sample sizes compared to using search sessions, which would have given us more reliable results in a shorter time.
However, CTR does not directly measure the business goal, so we could not easily connect CTR to the Search-to-Book rate. This meant we could not be sure if improvements in CTR would also lead to better booking results.
Hence, we chose to keep Search-to-Book as the main measure of success for the experiments, while also tracking CTR and other supporting metrics.
We decided to look at the overall performance of all regions in the test, but also monitor each region separately in the analysis after the test.
Additionally, although we initially planned for the test to last four weeks, we decided to extend it to six. This gave us more time and more data to better capture changes in user behaviour.
Test monitoring
To track the success and supporting metrics for the A/B test, we set up a Tableau dashboard that would refresh daily. This allowed the team to quickly take action if any of the metrics dropped significantly. Before launching the test, we agreed to stop the experiment if the Search-to-Book rate for the new method (Treatment) dropped by more than 20% compared to the old method (Control).
In addition to tracking business metrics, we created a Streamlit app within Snowflake to monitor the balance between the Control and Treatment groups. To make sure the number of sessions in both groups stayed equal (50-50), we used a chi-square test — a statistical method to check for any imbalance. If the groups were not balanced, we would review the A/B testing platform’s setup and fix that.
Testing challenges
We went through four rounds of testing. They demonstrate how iterative the process of evolving a data science model can be and how many factors come into play when the algorithm goes live in production.
In Secret Escape’s project, iterations encompassed everything from how the model was developed, what it did and did not account for, how the infrastructure was set up, how it performed, all the way to business context and initiatives that would impact the live testing.
The step-by-step process of building the foundation for this ML-driven LTR as well as all four rounds of A/B testing are described in detail on the Infinite Lambda Blog.
Testing round 1
- Streamlining searches: Updated OpenSearch to version 2.5, reducing query complexity. This reduced feature logging time by 75% and model size by 10–15%;
- User intent: Identified missing grade 3 events in the judgement list and collaborated with the client’s data team to find better data sources;
- Data accuracy: Used same-day ALS data for training, improving accuracy despite a smaller dataset. Introduced a super index to retain inactive deals, solving a 25–30% data loss issue.
Testing round 2
- Latency spike: Query times rose from 650 ms in the first test to 2200 ms, risking lower user engagement.
- Feature revisions: We removed underperforming features, added 20 new ones, and simplified calculations, which greatly improved performance.
- Model optimisation: The model size had tripled to 70 MB, causing complexity and latency. By simplifying the model and tuning parameters, we reduced its size to 25 MB with minimal performance impact.
Testing round 3
- Data misalignment: Differences in data logic between training and live use caused inconsistencies.
- Pipeline timing errors: Tasks ran before downstream processes finished, preventing the model from using updated data. We added dependencies to fix this.
- Indexing errors: Some fields were improperly indexed or missing for new features, causing the model to rank low-performing deals too highly.
Testing round 4
We launched the fourth A/B test with a focus on achieving the right balance between model complexity and performance. Careful feature selection was key to ensuring the model delivered meaningful improvements without unnecessary complications. Each step in the process was designed to evaluate and optimise the model while avoiding drawbacks like increased latency or file size.
The key steps in the process were:
- Retraining the model with each new feature to assess its impact using NDCG metrics.
- Using SHAP analysis to confirm whether the added features significantly improved performance.
- Avoiding irrelevant features to prevent unnecessary complexity and latency, ensuring the model remained efficient.
This structured approach allowed us to build a streamlined, high-performing model that focused on meaningful improvements.
We had many sessions with Infinite Lambda, and honestly, this was a great experience. Lambda consultants were extremely knowledgeable and had the expertise we needed for this project. Even more importantly, it was a great culture fit; this had been one of our key factors as the consultancy we chose was going to work closely with our tech and data teams to deliver this project.
— Gianni Raftis, Director of Data at Secret Escapes
We had many sessions with Infinite Lambda, and honestly, this was a great experience. Infinite Lambda consultants were extremely knowledgeable and had the expertise we needed for this project. Even more importantly, it was a great culture fit; this had been one of our key factors as the consultancy we chose was going to work closely with our tech and data teams to deliver this project.
— Gianni Raftis, Director of Data at Secret Escapes
The result
Margin uplift
We have successfully confirmed that the ML-powered LTR solution we helped Secret Escapes build increases booking conversion and margin.
Let’s explore the combined results for both regions X and Y.
Search CTR is lower in the Treatment group (-1.4%), likely due to lower discounts shown in top positions. Despite lower Search CTR, there is a higher margin and Search to Book.
The margin increases due to more bookings (+4.0%) and higher margin per booking (+3.0%). This suggests that improving Search CTR might not necessarily improve booking conversion and margin.
| Search to book | Search CTR | Total bookings | |
| Control vs Treatment | +4.6% | -1.4% | +4.0% |
Results for regions X and Y are similar to the overall outcomes, namely: higher Search to Book and margin, and lower Search CTR.
We see higher uplifts in region X, due to higher consistency in performance of T. As the X market is more price-sensitive, the LTR boosts deals with lower lead prices (despite lower discounts).
Territory | Search to book | Search CTR | Total bookings | |
Control vs Treatment | X | +2.3% | -0.9% | +2.9% |
Control vs Treatment | Y | +9.6% | -3.1% | +6.3% |
overall margin uplift
+4.7%
There are 3 main ways the LTR algorithm impacts the overall margin:
- Higher booking conversion in search sessions;
- Changes in user behaviour leading to higher booking conversion in non-search sessions;
- Higher amount spent per booking (i.e. margin per booking).
In the test, we observed an overall margin uplift of +4.7% based on the three main effects, mentioned above.
Successes
Let’s look at some of the main accomplishments for this ML project.
Cross-team collaboration
Infinite Lambda’s team and Secret Escapes’ data science team worked side by side throughout the entire project to develop, test, release and launch the A/B tests. Having a common goal in sight made it easy to align on priorities, make decisions, ideate, and test out assumptions as needed.
Some of the practical ways to enable collaboration included having:
- Daily stand-ups together;
- Joint planning and backlog review sessions;
- Joint technical sessions;
- A common Slack channel for ad hoc comms;
- Clear ownership of each task to be executed or blocker to be removed, as well as investment from people’s time to prioritise urgent matters.
All of this allowed the teams to sync up regularly to raise concerns, pose questions, exchange feedback, bring up topics for further discussion, and make decisions. Both teams were willing to contribute to the conversation and open to new ideas, which made for a supportive and wholesome project environment.
Iterative approach to empower learning and failing safe
We applied an iterative approach when executing on tasks and establishing what works. The joint team managed to prove assumptions quickly in the process, learning quickly and applying the new knowledge for further improvements.
This approach proved crucial for progress at several key occasions:
- Deciding what features should be left as part of the LTR model and how to configure the dimensions;
- Re-training the model and getting it to use the newly created indexing;
- Fine-tuning and trying out a smaller-in-size LTR model to improve performance;
- Deciding the re-ranking window configurations — we tried them out to see if they made any significant impact on performance and latency. When we validated that was not the case, we proceeded to increase the window from the first 100 to the first 500 results;
- Considering if we should re-configure the ranking influence of the LTR vs Elastic default configuration.
Comms with leadership and stakeholders
In order to keep Secret Escapes’ leadership and wider stakeholders in the loop, we established channels for frequent communication. The teams had a fortnightly demo, which was open to a wider group, such as analytics, design, product, leadership and other product teams (e.g. mobile).
They would also regularly exchange updates and reports to the leadership team to communicate both successes and challenges arising at different points.
Next steps
Together with Secret Escapes, we agreed that the focus should remain on optimising search relevance rather than relying on discounts to improve step conversion, such as CTR.
Relevance is more likely to increase a user's intent to book, which has a greater impact on overall conversions. Hence, prioritising an increase in booking intent is more effective than simply driving additional traffic to Search Page Views (SPV).
Secret Escapes is also considering adding features that reflect the relevance of trip types. We have noticed that well over a third of searches involve the use of a trip type filter, with a 1.4% increase in usage observed in the Treatment group. This feature may hold particular significance for users in region Y, where popular trip types such as "spa" and "beach" rank among the top five selections.
For future iterations, some of the actions that the client is considering include:
- Figuring out which territories should be rolled out next and drafting a roadmap for it;
- Defining clear goals for each of the territories in terms of uplift and gross margin;
- Ensuring that the LTR model is production-ready for each of the territories. Researching the specifics of the territory and making adjustments that account for them.
Let’s walk the walk together
Our data expertise translates into cutting-edge, modular solutions that empower people and help organisations thrive in the era of AI.
see Related Stories
We work with modern organisations to strategise and build cutting-edge solutions, help them adopt data & AI innovation, and nurture key competencies for scaling in the long run.
We work with modern organisations to strategise and build cutting-edge solutions, help them adopt data & AI innovation, and nurture key competencies for scaling in the long run.