Skip to content 
Manage Your Big Data Project
I have recently finished my project building an analytics platform for one of Infinite Lambda’s clients which inspired me to write this blog on how to ‘manage your big data project’. Working with a number of large tech organisations enabled me to experience many different approaches to implementing new big data projects. Being myself a […]
Full Development Lifecycle for PySpark Data Flows Using Databricks on AWS
In this blog we will focus on creating the project skeleton for a PySpark job, test framework, automating the build with GitLab CI, and deploying the jobs in production with Databricks Docker images on AWS. Phew! That's a lot! This can be considered Part 2 of my earlier blog post where we discussed the collaborative […]
Predictive Analytics on Large Datasets with Databricks Notebooks on AWS
The notebook for this blog can be downloaded here here. Recently I had the opportunity to work with Databricks notebooks with one of our startup clients at Infinite Lambda. I was fascinated by how this service increased the productivity of our team and let us collaborate effectively with developers and also with other stakeholders like […]
Using GraphQL for Dynamic Query Generation
A couple of months ago, we had to deliver a proof-of-concept for a client for migrating their entire backend database system from Postgres to Snowflake. This included extending the code of a custom built tool, which “translates” JSON to SQL. The functionality of this tool is quite unique - they have various filter options coming […]
dbt Testing Tools Gap
As explained in a previous post, we had a unique request from our client to physically (i.e. not logically) delete records from a data warehouse. dbt as our tool of choice for advanced data transformation was used to incrementally load the data. Due to the unusual request and implementation, we wanted to ensure that the […]
Deleting Records with dbt
dbt is a great tool for managing data transformations. It is built from the ground up to support version control and peer review, it enables and encourages best practices such as automated testing, environment separation and modularity. One of the most powerful features of dbt is the ability to only transform and load new and […]
How to Win with Data Engineering
I have been doing Data Engineering in some shape or form for the past 10 years. I have failed, many times. I have also succeeded a few times. Good news is, more recently I’ve mostly been succeeding. I attribute this increase in success rate to two things: I work with good people who know their […]
Streaming Pipeline in Snowflake
In the recent years we can clearly observe the trends in data space. The existing technologies are constantly changing, new products come to the market, where one of the main motivation is the increased demand from the business. Firstly nowadays we are in a better position to choose a performant warehouse (specially if it is […]
Apache Airflow Deployment on AWS ECS
Apache Airflow has became de facto in the orchestration market, companies like it because of many reasons. It has a nice UI for task dependencies visualisation, parallel execution, task level retry mechanism, isolated logging, extendability; because of the open source community it comes already with multiple operators and on top of that companies can define […]
When to Use Snowflake vs Redshift
Between us here at Infinite Lambda, we have decades of experience optimising AWS Redshift warehouses. We love Redshift - it's powerful, predictable, and robust. However, whenever someone mentions Snowflake, our eyes light up. They are both cloud-based data warehouses built for big data. But they are quite different. In this post, I'll share some reasons […]
Why Your Entire Data Infrastructure Should Be in Code
Infrastructure as Code (IaC) is a very popular practice in modern DevOps. There are plenty of resources on the internet on why you should do it. In this post, I’ll focus on why you should do it for your data infrastructure. Contents Data Governance Consistency across environments Maintainability 1. Data Governance Most medium- to large-scale […]
Optimising Redshift Tables for Storage
In this post, I'll share some tried and tested techniques for optimising your Redshift tables, with a focus on storage. Contents: Vacuum Encode Partition Unload (Bonus) Monitor performance 1. Vacuum Redshift's documentation says that the vacuum command "Resorts rows and reclaims space in either a specified table or all tables in the current database." That's […]