Skip to content 
There are two key phenomena in data engineering that have laid the foundations for data mesh, one of the hottest topics in the industry today.
First, data engineering has been edging closer and closer to mainstream software engineering in terms of practices, something I recently covered in another article. Second, the technological evolution in data has had a significant impact on the mindset of stakeholder, engineers and community.
I would like to start by bringing your attention to the latter of the phenomena and explore just how it has become the basis of event-driven data mesh, the concept at hand.
How technology impacts the mindset and the organisation
Let’s start with an industry example.
Hadoop as technology created the right environment for the data lake concept. Previously, organisations (and mainly the enterprise segment) could afford to invest in an enterprise data warehouse and business intelligence platforms but this was not sustainable with the ever-increasing data generation and an ever-growing consumption demand.
The Hadoop project set the foundations of the concepts of distributed storage and compute, architecting it in an open-source manner, which acted as a catalyst at that time.
At that point, brave companies (having the courage to handle the complexity of operating a 10-10000 node on-prem cluster is admirable) started their own data lake project; cash insensitive ones invested in the Cloudera and Hortonworks distributions.
The promise was clear: “Pump all the data into the lake so that we never lose any information. Mining the insights can be done later. Anyway, this is not my job as a producer.”
This promise, which resulted from a technology evolution step, started to shape the organisations, containing multiple problems.
The problem with trust
Firstly, even though it was a period of technological innovation, it was still early and the data stack was complex. It required specialisations – distributed software engineer, Hadoop engineer, big data engineer – a separate "data" team, so development life cycles were long.
That technological constraint led to a chained organisational form sequenced by technical specialisation. Each step in the chain relied on human interaction and communication. The farther you were from the source, the poorer quality data you had and the more the domain knowledge eroded.
Hence, the trust of the business users in the final numbers (if any at all) decreased, which just dug those data projects' graves and put a question mark on the data projects' ROI conversations, ultimately classifying data projects as second-class citizens. From the management point of view, these were expandable cost centres for the business.
The problem with schemas
Secondly, beyond the organisational impact, it also impacted the mindset of how we treat data. In the beginning, technology seemed like a big enabler for organisations, which is now solving their big problem of capturing and storing all of the data points so that no information is lost.
The data lake concept was indeed solving this problem.
But then it generated another one.
It slowly started to guide and direct different teams towards loosely governed schemas (also known as the schema-on-read approach), including data types, entity definitions and the change/release processes.
The focus was more on the fact that the platform could store the data and less on the quality of the data. The result was an ever-increasing technical debt for businesses to be able to deal with schema matching, reverse engineer quality issues and apply business logic.
This second point can also be valid in the modern data stack context. At Infinite Lambda, we have worked for multiple mature tech-driven organisations using the current technology stack, where the significant ratio of the data team has just been working on untangling the spaghetti of datasets and entities due to the loosely governed schema.
Why? Because the data team did not have a say on the structure of the incoming data. The main principle behind the data platform was that it could digest and sink anything, which resulted in the inefficient operation of the central data teams.
Similarly to the previous points, this makes the development lifecycle very slow, and the quality remains questionable. Earning trust by proving good quality became a complex and time-consuming exercise.
How can data mesh help?
Now that we know how technology helped shape the mindset within organisations and understand the context, let’s look at how data mesh can help.
There are numerous great articles that explain what data mesh is, such as the original papers “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh” (1) and “Data Mesh Principles and Logical Architecture” (2).
In this article, I will not attempt to explain data mesh from scratch but rather to elaborate on how it can bring dress ideas to the context.
This is the point where we should agree that we do not always talk about technology when it comes to data platforms. I am not saying that all technological problems to data have been solved. Certainly, there will always be challenges and customisations that require deep data domain knowledge. Yet, looking at the entire stack, we find it is not as complex as it used to be 10 to 15 years ago. At the end of the day, this is the point of abstraction layers and technological convergence.
Firstly we must admit that we should not always talk about technology when it comes to data platforms. We have seen in the past how technology forced organisations into a suboptimal form, why don't we abstract that away? I am not saying technology is a "solved" problem; there will always be challenges and customisations where we have to use our deep data domain knowledge; however, looking at the whole stack, it is not as complex anymore as it was 10-15 years ago. At the end of the day, that is the point of abstraction layers and technological convergence.
We should not be afraid to consider data as a specialisation of software engineering. Although the latter comes with its tools and practices, it is no longer necessary to separate them into different teams and departments.
Data has grown enough to get a seat at the table, and data-related activities can now be considered a first-class citizen. There is a much smaller gap between data and software engineering, and the job market has started to see that too.
If we can form our application engineering team according to the organisation's domains, we should also be able to involve the rest of the engineering team, including data engineers.
Data mesh ultimately shows us guidelines on applying a decentralised way of working in the data space. Introducing the concept of data products and redistributing ownership accordingly in a federated way will come with an after-effect of reallocating the budget to the highest value data artefacts. This will not only help in the data initiative’s ROI calculation, which is currently a big dilemma in the industry, but will also help data get the attention of the software engineers in the organisation.
Now we just need to find a way to accelerate the integration.
How can event-driven data mesh help?
Many might still argue that data mesh really aims to advise on the analytical plane level. Nevertheless, even the original article mentions the interoperability and the interconnectivity between the operational and the analytical worlds, expressing hope that a trend for convergence is going to start.
I believe that the convergence has begun with the use of event-driven architecture. That is called event-driven data mesh.
In his book Building an Event-Driven Data Mesh: Patterns for Designing and Building Event-Driven Architectures, Adam Bellemare suggests:
Event streams play the optimal role for the foundation of data products because they offer the immutable, appendable, durable, and replayable substrate for all consumers. These streams become a fundamental source of truth for operational, analytical, and all other forms of workloads across the organisation.
I choose to place data in quotation marks because it is no longer a source for analytical work as it is understood by default, but rather it now serves operational systems equally.
Data could become a real coupling point between services, microservices, teams and more. What is the promise of that convergence? It elevates data to a true first-class citizen.
What is next?
There are a lot of questions to ask ourselves at this point, such as How exactly will this convergence continue? and Who should and who should not drive that? In my next articles, I am going to dive into more details about event-driven data mesh with its technical and non-technical aspects and attempt to answer these questions.
When it comes to the steps of the transformation, every organisation is unique, so standardisation is not the way to go. Yet, companies would benefit from conversations around data mesh principles.
Topics such as data governance and data product, regardless of the final structure, can serve as a base for a healthy debate about problematic areas that might put the success of future data projects at risk.
———
If you are considering transitioning from a centralised to a federated governance model, do check out our article on this topic.
And if you do not know where to start with the transformation at your organisation, reach out. Infinite Lambda provides Data Mesh Maturity analyses that help you ask the right questions. Contact us to get started.
