Real-Time Analytics: The New Approach You Need

Being able to analyse information in real-time is no longer a luxury but a necessity for enterprises that seek to maintain a competitive edge. From monitoring live dashboards to enabling rapid AI model retraining, demand for up-to-the-minute insights has surged.

Meeting the technological expectations that enable real-time analytics requires a delicate balance between latency and analytical power, a balance that traditional data architectures often struggle to achieve.

In this article, I will delve into the latest developments in real-time analytics, exploring how both comprehensive cloud data warehouses and real-time data platforms are evolving to meet the complex demands of modern businesses.

The growing demand for fresh analytics in enterprise

As businesses increasingly rely on data to drive decision-making, the need for timely and accurate analytics has become paramount. Real-time dashboards provide instant visibility into key performance indicators (KPIs), thus enabling organisations to react swiftly to market changes.

Similarly, the AI demands rapid retraining of models to adapt to new data, ensuring that predictive algorithms remain relevant and effective.

There are several factors that fuel this push for immediacy:

• Seeking a competitive advantage: Quick access to data allows businesses to seize opportunities and mitigate risks faster than competitors;
• Meeting growing customer expectations: Consumers now expect personalised and instantaneous interactions, prompting businesses to leverage real-time data to enhance user experiences;
• Improving operational efficiency: Real-time monitoring can preemptively identify and address operational issues, reducing downtime and improving overall efficiency.

Event-driven architectures (and their limitations)

Historically, enterprises have relied on transactional Event-Driven Architectures (EDAs), such as ksqlDB, to handle streaming data. These systems excel at processing high-velocity data streams, enabling use cases like fraud detection and live monitoring. However, they often fall short when it comes to the depth and complexity of analytical power required for advanced AI and business intelligence (BI) tasks.

Transactional Event-Driven Architectures come with a few limitations as well:

1. Analytical depth: While they can handle real-time data ingestion and simple transformations, these systems may lack the computational power and advanced analytical capabilities that complex AI workloads need;
2. Scalability: Scaling transactional systems to handle large volumes of data with low latency can be challenging and resource-intensive;
3. Integration complexity: Integrating with other data sources and analytical tools often requires significant customisation, which images the data pipelines more complex.

Given these limitations, enterprises need more robust solutions that can deliver both low latency and high analytical power, especially as AI and BI applications become more sophisticated.

Comprehensive cloud data warehouses and real-time data platforms: a dual landscape

In response to the growing demands for real-time analytics, the data landscape has seen the emergence of two complementary categories of players: comprehensive cloud data warehouses and real-time data platforms.

The boundary between these two product genres is somewhat blurry. Let’s look at each of them in more detail.

Comprehensive cloud data warehouses

Comprehensive cloud data warehouses (CDWs) like Snowflake and Databricks offer robust data storage and analytical capabilities, traditionally optimised for batch processing and deep analytical tasks.

Characteristics of a comprehensive CDW

• High analytical power: Capable of handling complex queries, large-scale data transformations, and advanced ML tasks;
• Batch processing: Optimised for processing large volumes of data in scheduled batches, suitable for deep analytical tasks;
• Integration capabilities: Easily integrates with a wide range of data sources and analytical tools, facilitating comprehensive data analysis;
• Scalability and flexibility: Provides scalable storage and compute resources, allowing enterprises to handle growing data volumes and diverse analytical needs.

Moving into fresher data

Snowflake now offers dynamic tables which greatly reduce the latency arising from batch ETL processes. Coupled with native Snowflake ingestion app via CDC for PostgresSQL database, this enables a latency of a few minutes on small load and few joins.

Databricks now offers Delta Live Tables which also greatly reduce the latency arising from batch ETL processes.

Real-time data platforms

Real-time data platforms, exemplified by solutions in the Confluent ecosystem (built around Apache Kafka), focus on the ingestion, processing, and analysis of streaming data with minimal latency.

Characteristics of a real-time data platforms:

• Low latency: Designed to process and analyse data in real-time, ensuring immediate insights and actions;
• Event-driven architecture: Facilitates the creation of responsive and scalable applications that react to data events as they occur;
• Real-time integration: Seamlessly integrates with real-time data sources and services, supporting continuous data flow and analysis;
• Scalability: Handles high-throughput data streams, making it suitable for applications that require real-time data processing.

Moving into analytical workloads

Let’s look at several technologies that aim to address the demands of real-time analytics. Each of the technologies below comes with its own strengths and limitations we need to pay heed to:

• ksqlDB relies on RockDB which is a key-value store, and struggle on handling joins and large time windows (e.g. comparing many years of data);
• Apache Flink improves on ksqlDB but does not remove the need for a proper database for more demanding analytical workloads;
• ClickHouse, a database commonly used for real-time analytics workloads and itself being a columnar OLAP database, is known for its high-speed query performance and ability to handle large volumes of data;
• Rockset (recently acquired by OpenAI) is another option to consider. It offers vector search capabilities, essential for AI use cases.

Leveraging Confluent and TinyBird to reduce the learning curve

Managing and optimising real-time analytics platforms come with myriad complexities for the data teams to handle. This is where solutions like Confluent and TinyBird come in, designed to mitigate the challenges by streamlining key processes and abstracting away the underlying infrastructure complexities.

ClickHouse and Rockset are hard to maintain and fine-tune for performance. Apache Kafka instances, and to a lesser extent the AWS MSK, are not trivial to manage for typical data teams, as these are usually used by platform teams for transaction workload.

Here, Confluent greatly simplifies Change Data Capture (CDC) processes by offering an easy UI built on Kafka Connect and Debezium. TinyBird picks up the Kafka topics and easily creates the dashboards with an API, ClickHouse being completely encapsulated.

The risks of a one-size-fit-all approach

Real-time analytics’ and CDWs’ functionalities are increasingly competing with each other. This presents a new opportunity for companies using one single platform to deliver both workloads.

While a one-size-fits-all sounds like an attractive approach in terms of reducing skill gaps and costs, it also comes with risks that you should be aware of and that make it a reliable solution for simple use cases only.

Risks of using a CWD for low-latency analytics

For analytical data teams that traditionally build dashboards and reports on Snowflake or Databricks, the introduction of dynamic tables / Delta Live Tables looks like the shortest path to the world of real-time analytics. After all, the setup costs are minimal compared to introducing a completely new real time platform, such as Kafka, and existing tools like dbt can often be used, skill gaps are easy to overcome.

However, for use cases with critical latency requirements, where lags are unacceptable, such solutions come with a number of risks.

For example, in a Snowflake dynamic table, the lowest latency you can opt for is 1 minute. This is not the end-to-end latency, as it does not include the CDC ingestion time, and the refresh lags. In certain cases of high CPU usage (some non-trivial joins) and large number of rows (>200 million), the lag can quickly go beyond 5 minutes.

Finally, such a setup also comes with significant demands on the warehouse, thereby raising concerns around the costs.

Risks of using real-time analytics for high analytical power workload

Real-time analytics solutions like TinyBird (with its ClickHouse architecture) provide unprecedented analytical power, yet still keep end-to-end latency within seconds. However, this solution is not a substitute for Snowflake / Databricks for its comprehensive analytical capabilities — at least not yet.

Consider that the default limit of Kafka topics being ingested in TinyBird is no more than 5. Although it could be increased by contacting support, it is still very much on the low side when compared to Snowflake, which easily handles thousands of tables. When it comes to complex joins across tables, these are not possible or not going to be performant for production purposes.

Perhaps even more important is the issue of costs that this approach raises. Fast data, the type that is leveraged for immediate decision-making, is generally more expensive in software and infrastructure costs, and comes with a steeper learning curve for developers and administrators in the data team. Thus, moving everything into the real-time domain would substantially increase the cost of running your data platform, as it comes with infrastructure and training expenses.

Tailoring solutions to business needs

Clearly, the optimal balance between latency and analytical power is not a one-size-fits-all endeavour. It requires a nuanced understanding of the specific business requirements, as well as the ability to tailor the data solutions accordingly.

Here are 5 key steps to finding such a balance:

• Define business objectives: Clearly outline the goals of your analytics initiatives. Decide if you are going to prioritise real-time monitoring, predictive analytics, or comprehensive data exploration.
• Assess data characteristics: Seek to understand the nature of your data streams. Ask yourself if your data is predominantly time-sensitive, or if it requires in-depth historical analysis.
• Evaluate existing infrastructure: Review your current data architecture to identify strengths, weaknesses, and areas for improvement in handling both real-time and batch processing needs.
• Select appropriate tools and platforms: Choose a combination of technologies that align with your defined objectives and data characteristics.
• Implement and iterate: Deploy the solutions you have selected and continuously monitor their performance. Iterate on your architecture to refine the balance between latency and analytical power as business needs evolve.

To sum up

In a highly dynamic and relentlessly competitive business environment, staying informed about the latest developments and continuously assessing your data architecture’s alignment with business objectives will be crucial for maintaining a competitive edge and fostering data-driven innovation.

The landscape of real-time analytics is rapidly evolving, driven by increasing demands for fresh and actionable insights within enterprises. As businesses strive to optimise latency and analytical power, the convergence of slow and fast data platforms offers promising avenues to address these complex needs.

By understanding the trade-offs and strategically tailoring the solutions to the specifics of the use cases, organisations can find the balance that empowers them to harness the full potential of their data, both swiftly and deeply.

Get your copy of Infinite Lambda’s book Data & AI: Fast and Slow.

Tap into more data and AI insights on the Infinite Lambda blog.