Skip to content 
The rise of cloud computing and cloud-native technologies enabled the emergence of new age companies. A digital-native breed of businesses that truly operate 24/7 across the globe, companies running technology platforms that seamlessly switch between cloud providers in real-time, that are highly available, highly performant services and are always available to their customers.
While the AWSes of the world enabled this shift, today’s customers demanded it. They wish to use digital products and services seamlessly between devices at all times, often on unstable mobile connections from every part of the world. Consumers today do not have the patience or incentive to wait for degraded applications under the demand of a Black Friday sale; they simply move on to the next app, brand and deal.
Finally, while operational resiliency, scalability and portability are all very important, customers have been accustomed to and expect a high degree of personalisation and relevance as well.
It is all of these factors that lead companies to build cloud-native applications and make their businesses data-driven. But to this end, they need to consider event-driven architectures as well.
The case for data-driven applications
Let us take a closer look at the retail sector. We are all used to personalised recommendations and messaging of the “You might like this” type and the reason for their ubiquity is that they are efficient in increasing conversions. Personalisation is one of the numerous reasons for a business to be data-driven. Why should apps also be data-driven though?
Let’s look at another example.
Surely, you have been offered recommendations of questionable value as well. For example, you have just completed an order for a new coffee machine that you are bombarded with similar coffee machine deals. Recommendations that have been valid only a couple of minutes before are no longer good to you or the advertiser.
Obviously, relevance does not only concern the topic and interests but also the timeliness of the content that you serve to customers.
Another example and one that is of far graver consequences concerns real-time fraud monitoring. A fraudster can very quickly max out a credit card if a fraud monitoring system is refreshing only every 24 hours or even every 15 minutes. As long as it is not instant, a large enough window will exist to allow for significant financial damage.
There is also an ever-growing trend to make use of more data and more diverse types of data (structured, unstructured) in new ways in real time. From calculating surge pricing of taxis in London to predicting the occupancy of parking spaces and optimising electricity usage and storage from the grid for a greener planet and less eye-watering electricity bills, the potential is only going to keep growing.
Data apps
Applications built to take advantage of data to complement a data-driven organisation are known as data apps. Data apps are also subjected to those same customer expectations of 24×7 availability, infinite scalability and perpetual upgrades released frequently with a high degree of success. What is critical however is the timeliness of data app operations, requiring real-time data production and processing.
Data platforms that store data at rest for analytical purposes at a later time have proven to be essential to any data-driven business. Yet, those platforms usually require the extraction of operational data from various data sources, much like natural resources are extracted from the earth, their value being realised later through processing.
The processing of the data commodity is the transformation and modelling that takes place in data warehouses, data lakes or lakehouses. However, this creates a hard dependency on the operational persistence models. While that is acceptable for organisations of a certain size and needs, it is far from ideal for most.
The problem with disparate data teams
Microservices’ internal data persistence models are meant to change as the implementation evolves, frequently without the business objectives and processes changing. Therefore, tightly coupled data extraction processes tend to require similar if not more significant changes just to keep up with the evolving upstream data model.
What is more, the teams maintaining the data platforms are distinct from microservices engineers and therefore the changes to the operational data persistence remain unknown until too late in the process, which results in last-minute changes in a rush.
Actually, this is an optimistic scenario; in reality, operational changes will often go to production and the analytical workloads will break as a result, causing downtime and data loss at best but also potentially untrustworthy fragile datasets in the worst cases.
Event-driven architectures
I argue that the only practical way to enable real-time data-driven apps and data at rest use cases is through the implementation of event-driven architectures.
Event-driven architectures (EDA) promote the production of business events and their distribution, processing and storage as opposed to creating architectures which maintain state though synchronous or asynchronous request/response models.
At the centre of an EDA is the event. Events must be business events, easily and instantly recognisable by the business stakeholders of the system. Therefore, event definition, usage and upgrade require special care.
Events vs commands
We should make sure to distinguish between events and commands as the two are easy to confuse but the latter provides far less value. Here is an example:
- Event: I walked into a room; it is dark.
- Command: I walked into a room; switch on the lights.
The first example communicates the event that has occurred and leaves the processing to the relevant processor(s). This effectively reduces the coupling between producer and consumer.
The latter example is a command to turn on the lights, but it leaves the consumer unable to determine the state of darkness in the room, obstructing the passing of valuable information.
The focus on events a.k.a data creation naturally increases the quality of available data to other microservices as well as to a data platform. With EDAs, microservices are now free to iterate on their internal persistence or even do away with it.
Contracts
Software engineers have been using the words API and contract interchangeably for years. That is because an API specification is your contract with other consumers of your services.
In EDAs, APIs are replaced by event streams and it is only natural to consider the event model a contract. Furthermore, in an EDA the single source of truth is the combination of all event streams since the beginning of time, and the microservice data stores are orthogonal artefacts of the implementation. It is therefore obvious that data platforms should ingest those events streams in the stead of operational datasource.
I would argue that such architecture has successfully implemented an emerging concept called data contracts. Data contracts are an agreement between the backend engineers of the operational platforms and the data and analytics engineers consuming these data in order to create high-quality data products.
To enforce the data contracts with EDAs, I propose the following modification to the above diagram:
In the above diagram, we are using Kafka as the message broker to route the events from the producers to the intended consumers. Schema registry is used to maintain the schema correctness and evolution across releases.
Using Google Protocol Buffers (protobuf) allows us to take advantage of automatic code generation in our microservices as well as ingestion pipelines. Once that is in place, if we are using statically typed languages such as Java, we can enforce the data contract at build time in CI/CD. Data pipelines as well as microservices will be able to validate the correctness of the code vs. the protobuf contract and break the release of non-backwards compatible changes.
Such an architecture also enables data apps. An ML model trained from the high quality event streams which are persisted in a central data warehouse can be easily deployed against the real-time stream of events therefore making timely predictions.
Event-driven architectures and the cloud
EDAs have a number of other benefits and are well-suited for cloud-native applications. They promote low coupling between components allowing you to rewrite components with high-degree of success and supporting scaling developer teams. Such loosely-coupled components typically also have high cohesion making them easy to support and develop by small teams.
To meet consumer demand at peak times, technology platforms must be able to scale horizontally and, by using partitioning of the events streams, individual components can be scaled out independently. When an EDA is coupled with event sourcing or other mechanisms to persist a buffer of event streams, durability and correctness are more easily achieved through the replaying of events. Finally, the re-processing of events allows us to implement future use cases we are yet to imagine.
EDA alternatives
It is clear that while EDAs offer a number of great benefits and enable cloud-native applications and data apps, it is not a prerequisite for a successful implementation of data contracts. However, other possible implementations exist using services which expose API to the data platform pipeline.
While one has successfully decoupled the internal microservice data storage, we have added another dependency to each microservice and unfortunately, when faced with a production problem, developers will always prioritise the prod platform and not the data platform. Ultimately, any such implementation of data contracts is still prone to neglect and/or unintentional breakage. Furthermore, in order to enable the real-time use cases discussed above one must move away from batch ingestion to streaming.
Final words
Constantly evolving consumer expectations force companies to move to the cloud and create native applications taking advantage of capabilities that have previously been impossible. To fully realise the benefits of the cloud and compete in the marketplace businesses have become data-driven, but to meet relevancy expectations they also need to create data-driven apps. Data-driven apps are made possible through high-quality data guarded by data contracts and delivered in real time. Through the usage of data contracts, event-driven architectures uniquely enable the safe delivery of cloud-native applications and data apps.
