Skip to content 
Data Observability dbt Packages
What are dbt packages? Software engineers often use modularised code libraries, empowering them to focus on business logic while leveraging preexisting, perfected code for efficiency. In dbt, these powerful code libraries are referred to as 'packages’. They enable efficient problem-solving as shared analytic challenges are common across organisations. Today, we are going to focus on […]
Multitable SCD2 Joins: How to Unify Historical Changes
In the realm of data management, historical changes are conventionally stored in separate Slowly Changing Dimension Type 2 (SCD2) tables. However, extracting point-in-time insights from these dispersed sources requires merging them into a single, unified entity. This guide offers a succinct walkthrough of the process for performing multitable SCD2 joins, presenting two distinct approaches, Direct […]
Data Contracts for Non-Tech Readers: a Restaurant Analogy
Introducing the data kitchen Data contracts have been steadily gaining in popularity within the data community, providing a structured approach to manage and regulate the complex relationships between data producers and consumers. There are plenty of great articles out there explaining the technical aspects of data contracts, yet it remains a challenge for the non-technical […]
dbt Data Quality Tests Implementation
Data quality definition In this blog post, I am going to show you how to leverage dbt data quality tests to build robust pipelines that work for your organisation and case. To do that, we first need to understand what data quality is. Data quality (DQ) is the extent to which the data is fit […]
Towards Greater Composability in Data Platforms
This post is based on a talk of the same name that the author delivered at Data Mash #7 – London Edition in March 2023. If you have been working in the data industry over the last couple of years, you will have noticed an explosion in available tooling that is allowing us to build […]
Event-Driven Data Mesh: Understanding the Fundamentals
There are two key phenomena in data engineering that have laid the foundations for data mesh, one of the hottest topics in the industry today. First, data engineering has been edging closer and closer to mainstream software engineering in terms of practices, something I recently covered in another article. Second, the technological evolution in data […]
From Central to Mesh: How to Transform Governance Right
Governance is an essential part of any organisation. With good governance practices, businesses can manage processes and ensure the company follows strategy and goals. As companies grow, some transition into a decentralised business model and organisation. Using the data mesh approach, their governance model changes from central to mesh or federated governance. This article is […]
How to Implement Data Vault with dbt on Snowflake
Data Vault is a powerful data modelling methodology when combined with dbt and Snowflake Data Cloud. It allows you to build a scalable, agile and flexible data architecture that can be easily maintained and updated. In this article, we are going to provide guidelines and look at some best practices of how to implement Data […]
Data Vault Components: An Overview
Data Vault is a data warehousing methodology that provides a standardised and scalable approach to managing enterprise data. At its core, it is designed to be a flexible and adaptable framework that can store and manage large amounts of structured and unstructured data. The three main Data Vault components are the Hub, the Link and […]
Data Vault: Building a Scalable Data Warehouse
Over the past few years, modern data technologies have been allowing businesses to build data platforms of increasing complexity, serving ever more sophisticated operational and analytical needs. This introduces a need for effectively managing that complexity at scale – enter Data Vault, a holistic framework for creating and scaling an enterprise-grade data warehouse. Read on […]
Data Validation After Refactoring in Snowflake
Oh, well. Your current model is working as expected. The resulting table or query for reporting has good quality data that has been already validated by stakeholders and auditors, and is ready to go to production. If only the transformations could be more performant. Your team looks under the hood and identifies a few steps […]
Sustainability: the Last Frontier in Business Intelligence
The power of the modern data stack in generating actionable insights out of disparate data is well documented. It’s time to apply this to sustainability. As the latest instalment of the IPCC report has made clear, we are heading towards an ecological crisis that will be irreversible if not addressed in the next few years. […]