prefect-logo-full-gradient

Disrupting dataflow automation standards

triangle down left

What is Prefect?

Prefect is a powerful dataflow automation framework that lets you focus on what’s important, viz. understanding sources and schemas, handling any necessary data transformations and modeling out the destination accordingly. Meanwhile, it handles all the fluff like error management, logging, data serialisation and concurrency for you.

stroke line triangle down right

When should you use Prefect?

From a developer’s perspective, Prefect is perfect for quick and compartmentalised development. You can set up a task in minutes and using it inside an ingestion flow is even faster. Retry handling, dynamic mapping and caching come out of the box and concurrent async execution is easy to achieve.

From a user experience point of view, it is clean and modern. It provides a very good overview of all the flows and their status and allows an in depth view of each step in the ingestion process.

Use case

Overview

Autolus, an innovative biotech company, tasked Infinite Lambda with setting up an orchestrator and framework that are both easy to use, understand and maintain. Scalability and flexibility were also a priority and we were looking for solid performance, proper logging and the ability to self host.

Solution

Our Prefect ingestion orchestration for Autolus is running inside a Kubernetes cluster on Microsoft Azure. It ingests data from various sources, such as Oracle, Postgres and MS SQL and has Snowflake as a destination, using a mix of the Prefect built-in connectors and some custom ones as well. We are also leveraging the concurrency capabilities of Prefect by running tasks with a Dask Executor.

Prefect has been perfect for meeting our clients requirements of ease of use and modification, solid and obvious monitoring, efficient performance and, last but not least, reliable security.

Why choose Infinite Lambda

The experience

Infinite Lambda is a dbt preferred consulting provider, having successfully leveraged the technology in over 15 projects over the last 2 years. We continuously upgrade our skills and stay up to date with the latest developments.

The approach

We emphasise the full transparency at each stage of the project. Working with us, you will take an active part of the development process and will be familiar with everything we do with your system and resources.

The training

We do not shy away from sharing expertise and will organise engaging training sessions to teach you how to use dbt. Once we complete the project, you will be able to take full care of your infrastructure on your own.

Use Prefect with

new-dbt-dark-grey-logo
green-blue triangle pointing right

Let’s talk about how Prefect fits in your data stack. Contact us now!

Get in touch

Who is already using it

Fullstack Data

Bring all your real-time or batch data into a single integrated analytics platform

Cloud Development

Develop flexible cloud processes that are robust, secure and scalable

DevOps

DevOps

Leverage our flexible DevОps services to build and maintain a core cloud infrastructure

training

Training

Provide your team with regular training on the newest tools in BI, DevOps & more