Skip to content 
Multitable SCD2 Joins: How to Unify Historical Changes
In the realm of data management, historical changes are conventionally stored in separate Slowly Changing Dimension Type 2 (SCD2) tables. However, extracting point-in-time insights from these dispersed sources requires merging them into a single, unified entity. This guide offers a succinct walkthrough of the process for performing multitable SCD2 joins, presenting two distinct approaches, Direct […]
Data Contracts for Non-Tech Readers: a Restaurant Analogy
Introducing the data kitchen Data contracts have been steadily gaining in popularity within the data community, providing a structured approach to manage and regulate the complex relationships between data producers and consumers. There are plenty of great articles out there explaining the technical aspects of data contracts, yet it remains a challenge for the non-technical […]
How to Use Fivetran Upserting to Update Ingested Data
Introducing Fivetran and its AWS Lambda connector If you are a data engineer building your own ELT pipeline, then at some point in your career you must have heard of Fivetran, a data platform that automates your data ingestion process. Fivetran comes packed with useful features. Most of them are connectors that make ingestion as […]
Event-Driven Data Mesh: Understanding the Fundamentals
There are two key phenomena in data engineering that have laid the foundations for data mesh, one of the hottest topics in the industry today. First, data engineering has been edging closer and closer to mainstream software engineering in terms of practices, something I recently covered in another article. Second, the technological evolution in data […]
From Central to Mesh: How to Transform Governance Right
Governance is an essential part of any organisation. With good governance practices, businesses can manage processes and ensure the company follows strategy and goals. As companies grow, some transition into a decentralised business model and organisation. Using the data mesh approach, their governance model changes from central to mesh or federated governance. This article is […]
Using dbt deferral to simplify development
As data irrevocably grows in volume, complexity and value, so do the demands from business stakeholders, who need visibility in good time, whilst minimising cost. This means that as a developer, you are constantly looking for ways to reduce development time, save money and simplify every iteration in your dbt project. If you are not […]
How to Implement Data Vault with dbt on Snowflake
Data Vault is a powerful data modelling methodology when combined with dbt and Snowflake Data Cloud. It allows you to build a scalable, agile and flexible data architecture that can be easily maintained and updated. In this article, we are going to provide guidelines and look at some best practices of how to implement Data […]
Data Vault Components: An Overview
Data Vault is a data warehousing methodology that provides a standardised and scalable approach to managing enterprise data. At its core, it is designed to be a flexible and adaptable framework that can store and manage large amounts of structured and unstructured data. The three main Data Vault components are the Hub, the Link and […]
Data Vault: Building a Scalable Data Warehouse
Over the past few years, modern data technologies have been allowing businesses to build data platforms of increasing complexity, serving ever more sophisticated operational and analytical needs. This introduces a need for effectively managing that complexity at scale – enter Data Vault, a holistic framework for creating and scaling an enterprise-grade data warehouse. Read on […]
Snowpark for Python: Best Development Practices
While machine learning applications have been enjoying a peak in popularity in the last few years, companies still have a hard time integrating these innovative technologies with their cloud data platforms. Fortunately for Snowflake users, Snowpark, a recent addition to Snowflake’s capabilities, aims to solve this problem. Being a Snowflake Elite Partner, Infinite Lambda has […]
Data Validation After Refactoring in Snowflake
Oh, well. Your current model is working as expected. The resulting table or query for reporting has good quality data that has been already validated by stakeholders and auditors, and is ready to go to production. If only the transformations could be more performant. Your team looks under the hood and identifies a few steps […]
Make Data-Driven Apps with Event-Driven Architectures
The rise of cloud computing and cloud-native technologies enabled the emergence of new age companies. A digital-native breed of businesses that truly operate 24/7 across the globe, companies running technology platforms that seamlessly switch between cloud providers in real-time, that are highly available, highly performant services and are always available to their customers. While the […]