Data Mesh Adoption: A Business Leader's Perspective on the Benefits and Challenges

As a business leader, you are probably aware of the buzz around data mesh. There is abundant content out there from podcasts to blog posts and case studies.

One thing they all have in common is that they easily become quite technical. You might be lost amidst the buzzwords, such as data contracts, self-serving data platforms, multimodal data products and whatnot and find it challenging to participate in key conversations that map out the future of your organisation.

This article is going to help you stay on top of the data mesh talk, so you can confidently engage with technical leaders and understand how your organisation can benefit from data mesh adoption.

What do we learn from other companies’ data mesh journeys?

Let’s start by looking at other companies that have already implemented data mesh or are in the process of implementing it.

Here are some examples from 4 domains that we are going to focus on:

If you would like to see a full list of organisations from other industries too, check out this list. Here, you are going to find examples from auto, logistics, energy, fashion, government, gaming, travel, technology, software and many more verticals. We can easily see that data mesh is industry-agnostic.

On the list, you will find a number of other examples from the BFSI and e-commerce/retail sectors. It is not a surprise that these sectors usually belong at the more disruptive edge of the market.

Reasons for adopting data mesh

I took the time to go through the available use cases to try to understand why they have started the journey towards data mesh and whether we can see any patterns. I have collected quotes and insights, which you can find at the end of this article.

But to save you some time, I did find a pattern. Let me summarise it for you.

Companies that are scaling or that are on an intense journey of digitalisation cannot afford to waste time or slow down. At the same time, the relentlessly growing number of data-initiative-related requests paired with the complexity around use cases and domain knowledge put immense pressure on their central data team.

And there is a limit to the cognitive load the team can endure. As a result, turnover of the data team increases, knowledge leaks, quality drops and trust in data decreases along with the value it generates.

If you place yourself on the right-hand side of the diagram, you should be mindful of how your organisation might be blocking data-related initiatives. It is highly likely that you have a people and processes problem rather than a technology problem.

Starting out on the data mesh journey

To understand the benefit of data mesh at a strategic level, let’s look at the following diagram:

As companies start small, it all begins with a central approach, which is indeed the right thing to do. By focusing on the technology platform as the main enabler to generate value, organisations build the foundation they need.

As companies scale, however, technology transforms into an organisational problem and there comes a time to rethink the approach to data initiatives. That is what Data Mesh is all about.

At this point, you might be thinking that you have the same pain exactly and asking yourself if you should go with data mesh. Even if this is the right approach for your case, the transformation is a serious commitment that requires vigorous preparation.

All of this should start with a thorough analysis to identify the real problems and bottlenecks in your organisation as these are specific to every business.

What are the challenges of adopting data mesh?

Using the same methodology, I researched the challenges that companies had been facing and identified that they fall in one of three main categories:

• Management buy-in
• Cultural shift
• Adopting governance

These challenges clearly indicate that a lack of top management commitment makes it impossible to start the journey to data mesh.

As I have mentioned in the previous section, there is a point when technology transforms into an organisational problem. This does not mean that technology should not be considered or leveraged at the very start of the data mesh journey. On the contrary, an organisation should make sure to upgrade and take good care of the existing platform.

However, a successful transformation to mesh depends on various non-technical aspects, such as changes in the roles, ownership and general mindset that are embedded into the organisation’s bureaucratic structure. This is a far greater challenge to a transformation than technology itself.

If you are interested in exploring the challenges yourself, navigate to the Appendix at the end of this article.

Moving to mesh in terms of governance is no easy task. Check out the article on adopting federated governance on the Infinite Lambda blog post.

Let’s recap

In this article, we went over the benefits and challenges of data mesh from a business point of view. For companies that are struggling to scale, data mesh could be a solution. Business leaders are best suited to recognise this issue as they are familiar with the people, the processes and the culture at their organisation. If this is the case, it is worth teaming up with fellow business and technical leaders and running an analysis of the organisation’s structure.

When it comes to challenges, the primary ones are not technical in nature but rather come from culture and processes. A successful transition requires solid management buy-in, willingness to embrace the mesh model and a general culture shift to adjust to the changes.

If you decide to set out on the data mesh journey, the next step is to look at the challenges. Keep in mind that the primary challenges are not technical but rather cultural and process related. This requires top management buy-in, a significant commitment to ensure that this journey is started with the right mindset and resources in place. It is only later that technology even comes in as a concern but we have got a separate article to share insights on this topic.

If you are ready to start working on your data mesh adoption strategy, Infinite Lambda can help you lay the right foundations to make sure data mesh benefits your entire organisation. Get in touch to give us details about your project and we will run a data mesh maturity analysis.

Appendix 1: reasons for data mesh adoption

Let's look at those examples and try to understand why they have started the journey towards Data Mesh and whether we can see any commonality. Here are some of the most common cases:

• "Many organisations have invested in a central data lake and a data team with the expectation to drive their business based on data. However, after a few initial quick wins, they notice that the central data team often becomes a bottleneck. The team cannot handle all the analytical questions of management and product owners quickly enough. This is a massive problem because making timely data-driven decisions is crucial to stay competitive." (accessed on the datamesh-artchitecture.com website);
• As many other companies, XYZ is currently in an intense journey of digitalisation, in order to be successful, scalability of the tech organisation is crucial: scalability as enabler of speed, having more engineers responding quickly to the demands coming from business (features in software, insights, reports…). Therefore, the first priority is detecting and tackling the bottlenecks that could jeopardise the autonomy of the different units, otherwise “just hiring people” will not translate into speed.” (full story available here);
• "The other big driver is speed. When you have a centralized data team structure, the journey from an analyst’s question to an answer might travel through seven distinct steps across different members of the team each of whom have a SLA of 24 to 48 hours. That adds up!" (full story available here);
• "With new insight into our users’ experience, we defined a strategy that will enable a greater number of data workers to easily create and own high-quality data-driven systems resulting in smarter product offerings to [...] customers. [...] This powers the network effect of data production and consumption, which increases the productivity of all data workers, resulting in the large variety of high quality data driven systems [required]. (full story available here);
• “To address capacity, scalability and elasticity needs of a large data footprint of over 4PB, we decentralized and delegated ownership of the data stores and associated management functions to their respective business units.” (full story available here);
• Organisational purposes (full story available here):
  • “A central team of highly specialized data engineers has become a bottleneck in delivering all data products we want to build;
  • It is hard to make choices on priorities for the data team: “who do you enable first, advertising or marketing?”
  • It’s impossible for 30 data engineers to understand all applications being built by 600 IT professionals to serve 7000 employees and reach 8 million people on a daily basis.”
• “40 to almost 300 engineers [….] With that growth, the number of data requests and complexity increased up to the point that data engineers required deep domain understanding and suffered from context switching due to reprioritization across the business. In short, the central function became a bottleneck and slowed down innovation. As a result, analytics teams in business domains like marketing and supply chain management started to build their own data solutions. The data pipelines did not meet engineering standards and increased the central team’s maintenance and support. Over time, the knowledge about those pipelines left the company, and as a result, most of the data assets turned into debt, with tacitly decreasing data quality. That ultimately led to an increasing lack of trust in data.” (full story available here).

Appendix 2: data mesh transition challenges

Similarly, here is a list of challenges that the summary was based on:

• “It wasn’t easy. Moving towards a more decentralised ownership was a cultural challenge for our product teams. It took time to help people realise that this change was in everyone’s best interests.” (full story available here);
• “Lack of data product thinking“ (full story available here);
• “To implement that goal, we introduced the role of a Data Product Manager to envision, design, and establish data product thinking as a core part of our business.” (full story available here);
• “Similarly, the vision of our data assets has been strengthened, much thanks to the Data Product Owners that have really embraced their roles and are passionate about finding ways to enable data consumers with their products.” (full story available here);
• “Establishing a data-driven culture is quite important and sometimes even more important than building the right tech stack.” (Barr Moses, Lior Gavish and Molly Vorwerck, Data Quality Fundamentals: A Practitioner’s Guide to Building Trustworthy Data Pipelines; book available here);
• “Changing the culture from doing projects to building products” (full story available here);
• “Prepare your leadership for time to stand up a new platform: they should not expect results two weeks after you start (or even three…).” (full story available here);
• “As with all product development, identify clearly who your users are and what tools they currently use. You may need to transition or extend their tooling, and this may create friction and resistance.“ (full story available here);
• “Technology alone doesn’t make a data mesh. The shift to decentralized data pipelines and data warehouses requires cultural change within the organization.” (full story available here);
• “Technology is not the most important aspect of an implementation, no matter how cool it might sound. Start from knowledge sharing, getting people to understand each other’s role and contexts” (full story available here).