Disrupting dataflow automation standards
What is Prefect?
Prefect is a powerful dataflow automation framework that lets you focus on what’s important, viz. understanding sources and schemas, handling any necessary data transformations and modeling out the destination accordingly. Meanwhile, it handles all the fluff like error management, logging, data serialisation and concurrency for you.
When should you use Prefect?
From a developer’s perspective, Prefect is perfect for quick and compartmentalised development. You can set up a task in minutes and using it inside an ingestion flow is even faster. Retry handling, dynamic mapping and caching come out of the box and concurrent async execution is easy to achieve.
From a user experience point of view, it is clean and modern. It provides a very good overview of all the flows and their status and allows an in depth view of each step in the ingestion process.
Autolus, an innovative biotech company, tasked Infinite Lambda with setting up an orchestrator and framework that are both easy to use, understand and maintain. Scalability and flexibility were also a priority and we were looking for solid performance, proper logging and the ability to self host.
Our Prefect ingestion orchestration for Autolus is running inside a Kubernetes cluster on Microsoft Azure. It ingests data from various sources, such as Oracle, Postgres and MS SQL and has Snowflake as a destination, using a mix of the Prefect built-in connectors and some custom ones as well. We are also leveraging the concurrency capabilities of Prefect by running tasks with a Dask Executor.
Prefect has been perfect for meeting our clients requirements of ease of use and modification, solid and obvious monitoring, efficient performance and, last but not least, reliable security.
Why choose Infinite Lambda
Infinite Lambda is a dbt preferred consulting provider, having successfully leveraged the technology in over 15 projects over the last 2 years. We continuously upgrade our skills and stay up to date with the latest developments.
We emphasise the full transparency at each stage of the project. Working with us, you will take an active part of the development process and will be familiar with everything we do with your system and resources.
We do not shy away from sharing expertise and will organise engaging training sessions to teach you how to use dbt. Once we complete the project, you will be able to take full care of your infrastructure on your own.
Use Prefect with
Let’s talk about how Prefect fits in your data stack. Contact us now!