Skip to content 
Airflow start_date and execution_date Explained
Despite Airflow’s popularity in data engineering, the start_date and execution_date concepts remain confusing among many new developers today. This article aims to demystify them. Basic Concept Airflow is an orchestration tool, which means that with sufficient permission it is capable of controlling other services in a pre-defined order and timing. Let’s say you want to […]
Creating a PostgreSQL to BigQuery Sync Pipeline Using Debezium and Kafka
Many companies today use different database technologies for their application and their data platform. This creates the challenge of enabling analytics on application data without giving analysts access to the application database. A highly efficient solution is using Change Data Capture (CDC) to sync application data with the data warehouse. Now, we are going to […]
Postgres CDC Solution with Debezium and Google Pub/Sub
If you have ever felt the need to track modifications happening in your database whether it be the creation of a new table, deletion of a record or a schema change, a Change Data Capture (CDC) solution might be a good fit for you. As the name suggests, CDC is a process that captures changes […]
Tableau in the Modern Data Stack
Tableau has been a leader in the business intelligence space in the last decade. It entered Gartner’s leaders quadrant in 2013 and remains one of the platforms with the greatest ability to execute and completeness of vision today. The simplicity of the drag and drop functionality, combined with the lightning speed of the extracts (at […]
How to Reduce Transaction Costs with Both Centralised and Decentralised Ledger Technology: from Bitcoin to DLT
The Ledger behind Bitcoin Bitcoin means a lot of things to a lot of people. Investors cannot get enough of it, while large companies are increasingly adopting its underlying technology. Countless startups and even unicorns are built from its underlying tech. Therefore, it is not an exaggeration to say each participant has a view on […]
An Event-Sourcing Core Banking Platform on Kafka
The Problem Over a year ago in 2020, the team at Infinite Lambda was tasked with creating a digital core banking system for one of our clients. The system had to fit the budget and business needs of enterprises of various sizes, from small-scale start-ups with a few customers to really big banks with millions […]
Prefect Workflow Automation with Azure DevOps and AKS
Welcome to another article on DevOps and DataOps related topics. I am going to show you how we automated Prefect flow deployments on Azure Kubernetes Service with the help of Azure DevOps. In order to be able to follow along, you need to have knowledge in: Azure Cloud Azure DevOps Azure Kubernetes Services (AKS) Data […]
Production-like Airflow Development Environment on Minikube
DataOps is clearly going to revolutionise the traditional data development mindset. This is not a nice-to-have anymore; actually, data projects could fail if DataOps principles are not adopted. Local development environment in data could be extremely complex, costly, and time-consuming, especially if we want to simulate a production-like environment. They do not just have to […]
The Cost of Snowflake vs On-Prem Data Solutions
3 years ago, I was head of data warehouse and BI at a market-leading insurance company, leading the development and operation team. Back then, we used to work with on-prem technology. Now I work with cloud-based platforms, mostly Snowflake, and I am really happy with that because it makes it easier to deliver real value […]
Machine Learning Lifecycle Management with MLflow on Databricks
Machine Learning lifecycle challenges Machine learning is still difficult, but with well-defined APIs abstracting away the complexities of algorithms it has become less complicated. However, managing the Machine Learning Lifecycle is not easily streamlined and remains really difficult and complex for many organisations. Today, we discuss MLflow on Databricks and illustrate the comprehensive framework machine […]
Leveraging Snowflake Data Shares: an E-commerce Case Study
A primer on data sharing Data shares are a cutting-edge new tool developed by Snowflake, allowing any organisation with a Snowflake account to seamlessly and instantly provide data access to other organisations anywhere in the world. Snowflake themselves are leveraging this technology to provide a data marketplace, most recently being deployed by Starschema to provide […]
Swiss Army Knife Airflow on EKS
At Infinite Lambda we have already helped many of our clients to setup Apache Airflow. The tool itself is great, one of the best orchestrators on the data engineering market. Although deploying it on your cloud, and tailoring it to your needs… well, the required engineering hours are still not negligible. In terms of the […]