Skip to content 
Postgres CDC Solution with Debezium and Google Pub/Sub
If you have ever felt the need to track modifications happening in your database whether it be the creation of a new table, deletion of a record or a schema change, a Change Data Capture (CDC) solution might be a good fit for you. As the name suggests, CDC is a process that captures changes […]
How to Reduce Transaction Costs with Both Centralised and Decentralised Ledger Technology: from Bitcoin to DLT
The Ledger behind Bitcoin Bitcoin means a lot of things to a lot of people. Investors cannot get enough of it, while large companies are increasingly adopting its underlying technology. Countless startups and even unicorns are built from its underlying tech. Therefore, it is not an exaggeration to say each participant has a view on […]
An Event-Sourcing Core Banking Platform on Kafka
The Problem Over a year ago in 2020, the team at Infinite Lambda was tasked with creating a digital core banking system for one of our clients. The system had to fit the budget and business needs of enterprises of various sizes, from small-scale start-ups with a few customers to really big banks with millions […]
Prefect Workflow Automation with Azure DevOps and AKS
Welcome to another article on DevOps and DataOps related topics. I am going to show you how we automated Prefect flow deployments on Azure Kubernetes Service with the help of Azure DevOps. In order to be able to follow along, you need to have knowledge in: Azure Cloud Azure DevOps Azure Kubernetes Services (AKS) Data […]
The Cost of Snowflake vs On-Prem Data Solutions
3 years ago, I was head of data warehouse and BI at a market-leading insurance company, leading the development and operation team. Back then, we used to work with on-prem technology. Now I work with cloud-based platforms, mostly Snowflake, and I am really happy with that because it makes it easier to deliver real value […]
Machine Learning Lifecycle Management with MLflow on Databricks
Machine Learning lifecycle challenges Machine learning is still difficult, but with well-defined APIs abstracting away the complexities of algorithms it has become less complicated. However, managing the Machine Learning Lifecycle is not easily streamlined and remains really difficult and complex for many organisations. Today, we discuss MLflow on Databricks and illustrate the comprehensive framework machine […]
Canary and Blue/Green Deployments with Helm and Istio
In this article, we are going to review ways in which we can deploy two versions of our application in production-like environments in Kubernetes and apply two different availability approaches. The first is to distribute network traffic evenly between the two versions (canary) and the second is for 100% of the traffic to either one […]
Focus on the Business with Serverless API Using AWS and GraphQL
In This Article You Will Learn A Bit About GraphQL A Bit About Serverless Why Serverless Goes Well With Graphql? The Simple Architecture We Created The Entry Point GraphQL Lambda Available Queries File The Account Query File Querying With Result What We Achieved Introduction I used to work as a web application developer before I […]
Using GraphQL for Dynamic Query Generation
A couple of months ago, we had to deliver a proof-of-concept for a client for migrating their entire backend database system from Postgres to Snowflake. This included extending the code of a custom built tool, which “translates” JSON to SQL. The functionality of this tool is quite unique - they have various filter options coming […]
How to Win with Data Engineering
I have been doing Data Engineering in some shape or form for the past 10 years. I have failed, many times. I have also succeeded a few times. Good news is, more recently I’ve mostly been succeeding. I attribute this increase in success rate to two things: I work with good people who know their […]
Apache Airflow Deployment on AWS ECS
Apache Airflow has became de facto in the orchestration market, companies like it because of many reasons. It has a nice UI for task dependencies visualisation, parallel execution, task level retry mechanism, isolated logging, extendability; because of the open source community it comes already with multiple operators and on top of that companies can define […]
Why Your Entire Data Infrastructure Should Be in Code
Infrastructure as Code (IaC) is a very popular practice in modern DevOps. There are plenty of resources on the internet on why you should do it. In this post, I’ll focus on why you should do it for your data infrastructure. Contents Data Governance Consistency across environments Maintainability 1. Data Governance Most medium- to large-scale […]