Skip to content 
Most data leaders do not need convincing that AI is worth investing in. They have seen the demos, the technology is impressive, and the use cases are obvious.
We could say the business case writes itself.
And yet, somewhere between the proof of concept and production, things stall. That stall almost always traces back to the same place: the data.
The challenges with the agentic enterprise
An agent or model is only as useful as what it can actually see, and in most large organisations, that picture is surprisingly patchy. Pipelines were built for specific use cases and never generalised.
Datasets get copied across platforms in ways that made sense at the time but quietly accumulated into a maintenance headache. Data that should be fresh is hours or days old because the ETL job that feeds it runs on a schedule nobody questions anymore.
Picture a supply chain analyst who has been asking for an AI tool to flag inventory risk before it becomes a crisis. The model is ready. The use case is funded and approved. Then someone actually looks at the data: three platforms, partial coverage, a nightly batch job that means the agent would be working from yesterday's numbers. Months later, the project is still in the "data preparation" phase, and the analyst has stopped asking for updates.
That pattern repeats across the enterprise, in different teams and different shapes, and it tends to come down to three underlying challenges:
- Not enough data is available in the platforms where AI runs;
- The same data is duplicated across multiple systems;
- Legacy architectures make modernisation slow and risky.
This article looks at each challenge in turn.
AI Challenge #1: Enterprise data platforms are still incomplete
For decades, organisations built data pipelines the same way:
- Identify a business problem
- Find the relevant data sources
- Build something just big enough to solve that one problem.
The result, over time, is a tangle of purpose-built pipelines that need rebuilding every time requirements shift.
We should not think of this as poor planning. It was actually a rational response to real constraints at the time. Databases were expensive and did not scale well, so teams processed only the minimum data needed for a given workload. Bringing in everything was simply not practical.
Cloud-scale platforms have changed that calculus entirely.
The useful model now is one where every data source continuously replicates 100% of its data into a cloud-native data platform, with changes kept in sync automatically through built-in Change Data Capture (CDC).
That gives you a complete, current, and historical view of enterprise data from a single foundation, rather than a patchwork of partial snapshots.
The effect downstream is significant. Analytics, AI applications, and intelligent agents at the enterprise level can be built far faster when the data preparation work has already been done, not reconstructed from scratch for each new project.
AI Challenge #2: Data duplication is driving unnecessary cost and complexity
Many organisations operate what I like to call "Noah architectures" because they have two of everything.
That typically means multiple data and AI platforms, such as Snowflake for analytics and Databricks for machine learning, running across AWS, Azure, and Google Cloud simultaneously.
The cost of that sprawl adds up quickly. Organisations end up replicating the same data across multiple platforms, which eats into the efficiencies that modern data architectures are supposed to deliver. In some environments, as much as 20% of platform costs go purely to data ingestion and replication.
That leaves organisations with an uncomfortable choice:
A: Consolidate onto a single platform and potentially lose specialised capabilities or
B: Maintain multiple platforms and absorb the ongoing cost and complexity.
Fortunately, the industry has largely recognised that forcing a choice between capability and cost was never a sustainable model.
Managed data lakes, storing all data in the Apache Iceberg format, actually address both options A and B directly.
To prevent capability loss in scenario A, you can opt for a single unified storage layer. Think of a Global Bronze Layer, accessible to any platform.
Scenario B is more interesting, though. Here, we can use different platforms across departments or groups. We can even bring in a new platform that has a unique AI model supporting Iceberg, without having to copy any data again.
In short, this architectural shift has a practical consequence of having ingestion happen once, so the same dataset can be available across your entire stack.
As new platforms emerge or existing ones fall out of favour, the underlying data remains accessible without re-engineering the pipelines that feed it.
That is what sustainable, scalable data architecture actually looks like in practice. You might not choose the stack perfectly or come to outgrow it, but you have a foundation flexible enough to absorb change without starting over each time.
AI Challenge #3: Modernising legacy architectures without disrupting the business
Once the target architecture is defined, the next challenge is execution.
Many organisations still rely on legacy ETL pipelines and architectures that were considered best practice years ago but have since become technical debt. Moving from purpose-built ETL processes to scalable ELT architectures means that transformation logic can run directly within modern cloud data and AI platforms, which is generally more efficient and easier to maintain.
The complication is that these legacy systems are often business-critical. You cannot simply switch them off.
That tension, between the need to modernise and the need to keep things running, makes risk management a central part of any migration. Getting the sequencing right, validating outputs at each stage, and having a structured path through the process matter as much as the technical decisions themselves.
Infinite Lambda's flagship modernisation solution, Flowline, was built with exactly that in mind. It provides a guided, end-to-end path that breaks the process into manageable stages, with validation built in throughout.
Teams can move legacy pipelines to modern ELT architecture incrementally, checking that outputs remain consistent at each step before moving on, which keeps the business running while the modernisation work happens around it.
How Fivetran and Infinite Lambda enable enterprise AI
Migrating to a modern data architecture sounds straightforward until you factor in the speed and the platform variety. At the same time, you also need to keep everything running throughout.
Flowline handles the structural side of that problem. It analyses existing workloads first, which produces both a technical and a project plan before any migration work begins.
From there, for each identified source, Fivetran provides the extract-load pipeline that does the heavy lifting.
Each pipeline is built around a few core capabilities that matter particularly in the context of AI readiness:
- CDC coverage across every source: Traditional ETL was designed around daily or weekly loads, which made sense at the time. Fivetran pipelines can move data up to 1,440 times per day, so downstream agents and decision-makers are always working with current data rather than yesterday's snapshot.
- Optimised data formats: A one-to-one replica of a source system works well for operational reporting, but AI agents and machine learning models depend on historical data to function properly: Fivetran supports both, selectable at deployment time and adjustable later, so you are not locked into a choice made on day one.
- Multi-platform readiness. Many organisations run several intelligence platforms across multiple cloud providers, which normally means replicating data everywhere and paying for it repeatedly. Writing to Apache Iceberg via Fivetran's Managed Data Lake Service removes that overhead, typically reducing ingestion costs by around 20%, while keeping the data accessible to whichever platforms need it.
Handling the logic
dbt Labs handles transformation and filtering logic to ensure consistency across the board. A metric like monthly revenue, for instance, is defined once in dbt and produces the same result whether it appears in a report or gets queried by an agent. Tests and defined metrics keep that consistency stable as the data and the business evolve.
The goal is straightforward: reports and agents draw from the same foundation, so every business decision is based on the correct data at the right time.
One of the more technically interesting aspects of this setup is the dbt Fusion engine and its support for stateful transformations. Rather than rebuilding every table on every run, dbt tracks which tables have changed and only processes those, along with their upstream dependencies. That has an obvious effect on warehouse compute costs, but the business impact goes further.
Let’s see an example
Say inventory is tracked across several channels: a website, a physical store, and a digital channel. The target model is an Inventory table used by both supply chain teams and agents.
When inventory changes in one channel, perhaps just one underlying table, only that table and the models that depend on it are rebuilt. Everything else stays the same.
Because dbt Labs manages all of that state automatically, developers do not need to define programmatic dependencies or write custom logic to determine what needs to run. The transformations execute in the shortest path possible, and everyone downstream gets the latest data without waiting for a full refresh.
The practical path to AI-ready data
A massive budget does not guarantee getting AI right at the corporate level.
The companies that thrive in the agentic age are the ones that get their data in order first, in a way that is controlled, measurable, and honest about risk from the start.
That is the thinking behind the collaboration between Infinite Lambda and Fivetran.
The approach we take surfaces risks early, puts a structured plan in place, and validates outcomes at each stage. The ultimate goal is to keep the same logic and the same data, but be able to operate at the speed that AI actually demands.
A useful starting point is the ODI Data Access Scorecard, which benchmarks vendors based on how easily you can export and utilise data from their systems. Check it out to compare the availability of data/objects, how efficiently data can be extracted from the platform, and the cost associated with accessing it.
Stay tuned for the ODI assessment launch that allows you to take an interactive quiz to assess current architecture against the requirements of modern AI workloads that ODI supports.
About the author
David Millman is a Partner Sales Engineer at Fivetran, where he works at the intersection of data infrastructure and AI readiness. With a background spanning data integration, pipeline architecture, and enterprise modernisation, David helps organisations close the gap between where their data is and where it needs to be for AI to actually work in production.
