Skip to content 
Mandai Wildlife Reserve
Automated,
AI-Enabled Data Platform
Building a Thriving Data Habitat with Fivetran
A fascinating client
Mandai Wildlife Group is the steward of Mandai Wildlife Reserve.
Operating a destination that combines multiple parks, including the Singapore Zoo, Night Safari, and River Wonders, our client connects the fascinating world of wildlife in Singapore with the visitors to the reserve.
Key results and metrics
Data ingestion challenges during modernisation
Mandai engaged Infinite Lambda to modernise their data stack end-to-end by migrating their legacy data system from SQL Server and SSAS Cube to dbt and Snowflake.
Mandai’s ecosystem requires ingesting data from on-prem to SaaS sources. To handle this efficiently, the natural choice was Fivetran’s technology, which would offer the ultimate reliability with automated integration across the client’s diverse data sources.
Strict security and governance standards
Mandai operates under rigorous security controls, with a clear requirement that all data remains within its inbound network. At the same time, the organisation applies strong engineering governance.
Every change must go through a fully automated deployment process, reviewed and managed end to end through CI/CD. Manual configuration was not an option, which placed clear constraints on how the data platform could be designed and operated.
Complex data sources
Mandai’s data landscape is both broad and demanding. It includes a large SQL Server estate, alongside enterprise systems such as SAP, with a very high number of tables and frequent data changes. Production systems generate constant updates, and the team wanted to avoid approaches that would increase operational overhead or require additional storage just to track changes.
Some systems had already reached technical limits that made standard change tracking approaches difficult, while access to data sources had to be granted in stages. This meant the solution needed to support gradual onboarding, without disrupting live systems or slowing progress.
Hybrid Deployment Approach
To meet Mandai’s strict security requirements, we designed a setup where data always stays within their network. We used Fivetran’s hybrid deployment mode, with ingestion agents running inside Mandai’s own AWS environment on a private EKS cluster.
This approach allowed Mandai to benefit from automated data ingestion without exposing source data to the public internet. It provided the reassurance their security and DevOps teams needed, while still enabling a modern, cloud-based data platform that could scale with the organisation’s needs.
Protecting data in transit
Security was built into every step of how data moved through the platform. All communication between Mandai’s environment, Snowflake, and Fivetran is encrypted, ensuring data remains protected while in transit.
- Hybrid deployment to Snowflake: the data path
Data is extracted and processed by the ingestion agents running inside Mandai’s own network. From there, the data is sent directly to Snowflake over an encrypted connection. At no point is customer data routed through external systems.
- Hybrid deployment agent to Fivetran Cloud: the metadata path
Only non-sensitive information, such as configuration details and operational logs, is shared with Fivetran’s cloud service. This allows the team to monitor performance and reliability through the Fivetran interface, without compromising Mandai’s security standards.
Full control over agent configuration parameters
The hybrid deployment approach also gave Mandai full control over how the ingestion agents were configured. This flexibility was important given the scale and complexity of some source systems.
For example, Mandai’s SAP environment contains a very large number of tables, even though only a small subset was required for analytics. By tuning the agent configuration, we ensured the platform could efficiently discover and manage these schemas without unnecessary overhead.
The result was a setup that could handle enterprise-scale systems smoothly, while remaining focused only on the data that delivered real value to the business.
Effective data ingestion: a multi-agent approach
Mandai works with many different data sources, each with its own characteristics. To manage these effectively, we grouped connectors based on factors such as data size, loading method, and whether the source was on-premises or in the cloud.
Each hybrid deployment agent can support up to ten connectors. This is why we created multiple agents with specific configurations to handle these groups and manage the connectors more efficiently.
Automated Multi-Environment Fivetran Deployment
Mandai’s DevOps team requires that all infrastructure changes are fully automated, traceable, and managed through CI/CD, with no manual updates to data pipelines.
To meet this requirement, we used Fivetran’s Terraform Provider, which allows all Fivetran resources to be managed as code. Infinite Lambda implemented a module-driven Terraform architecture that:
- Automates the creation and management of every data pipeline and related component;
- Ensures all changes are version-controlled and deployed through CI/CD;
- Supports seamless deployment across multiple environments, including development, testing, and production.
This approach gave Mandai full confidence that deployments are consistent, auditable, and repeatable, reducing the risk of errors and speeding up delivery.
Each Fivetran component – whether an agent, connector group, destination or connector – is wrapped in a reusable Terraform module. Environment-specific settings for development, testing and production are supplied through a configuration map.
This approach has several benefits:
- Consistent, repeatable deployments: dozens of Fivetran connectors can be created from the same codebase rather than manually configuring each one;
- Elimination of manual errors: removes the risk of drift or unexpected changes from manual updates;
- Secure handling of sensitive data: credentials and other sensitive values are fetched directly from AWS Secrets Manager and mapped automatically to downstream modules.
The result is a robust, efficient pipeline management process that saves time, improves reliability, and strengthens security.
Module structure & change flow
All updates to Mandai’s data pipelines follow a clear and controlled process. Every change is reviewed and approved by the DevOps team before it is applied, ensuring consistency and security across development, testing, and production environments.
This automated, version-controlled approach ensures that:
- Every data pipeline is deployed reliably and consistently;
- Manual steps and errors are eliminated;
- Sensitive information is handled securely.
The result is a modern, resilient platform where updates can be made quickly and safely, giving Mandai confidence in the accuracy and stability of their data.
Flexible loading methods to adapt to specific database configurations
Mandai’s SQL Server environment included systems with different setups and very active production workloads. Some databases were too large for standard update-tracking methods, and there were concerns about storage requirements.
To manage this, we:
- Loaded each database fully at first, then updated only new or changed data;
- Chose the best update method for each source based on how busy it was;
- Grouped databases across different agents to keep the system efficient and reliable.
This flexible approach ensured that even the busiest systems could deliver accurate, up-to-date data without straining resources.
Taming busy databases
- Incremental updates: We set up connectors to track only new or changed records, so updates happen faster and without overloading systems.
- Handling high-volume databases: For the busiest systems, we used specialised connectors that replicate all changes efficiently while keeping production systems stable.
- Managing transaction logs: We put in place a process to archive logs until they were safely processed, preventing storage issues and ensuring backups continued as needed.
One-time ingestion and exceptions
For certain datasets, we used a one-time load approach called Teleport Sync. This was ideal for cases where:
- Data was no longer actively maintained but still needed for reporting;
- Some datasets were required for validation while security approvals for high-volume methods were pending.
We carefully checked that the datasets were compatible, sized appropriately, and accessed securely with minimal privileges.
This ensured Mandai could include all necessary data in their platform safely and efficiently, even for exceptions or temporary needs.
SOLID IaC THAT ENABLES TRUSTED AI-ASSISTED INSIGHTS
We had already built a solid foundation of Fivetran IaC that is well-defined, scalable, and easy to use. At this point, we could have an AI agent safely step in and improve performance.
Leveraging Claude Code’s Agent Skills would allow for significant optimisations, including reducing repeatable work, speeding up the onboarding of new engineers on existing resources, and even spinning up new resources faster under team control.
We configured a specific Claude Agent to operate over this repository with the skill set as its toolkit. Note that this agent does not act autonomously; it proposes and explains, but humans approve and merge.
The core trust problem with AI managing infrastructure, namely how do we know what it has done, whether it is right, and how to reverse it, can only be solved through an Infrastructure as Code (IaC) approach.
Here is what this looks like in practice and what we applied in Mandai’s case:
| Trust concern | How IaC Addresses it |
|---|---|
| Trust concern:Auditability | How IaC Addresses it:Every change is a git diff, which is reviewable before merge |
| Trust concern:Reversibility | How IaC Addresses it:git revert restores the prior state |
| Trust concern:Blast radius | How IaC Addresses it:State is isolated per module, so a mistake in one cannot corrupt another. |
| Trust concern:Ground truth | How IaC Addresses it:The Terraform files are the canonical description of reality; AI reads the same source people do. |
| Trust concern:Gatekeeping | How IaC Addresses it:plan always runs before apply; CI blocks merges that fail lint or security scan. |
- People give context
- AI can propose
- Terraform can preview
- People approve
AGENT SKILLS FOR MANAGING FIVETRAN IaC
- Fivetran-plan: Generate a Fivetran plan for a specific environment. Requires both tfvars files.
- Fivetran-apply: Apply a previously generated plan file.
- Connector-review: Before adding or modifying a connector, read the existing pattern for that connector type and verify that the proposed change is consistent.
- Schedule-review: Verify that every connector has a matching entry in the schedule configuration.
- Schema-config-review: Verify that new tables exposed by a SQL Server connector are explicitly listed.
- Agent-review: Before adding a new hybrid deployment agent, verify the EKS side is ready to receive it.
- Drift-detect: Surface state drift without proposing changes. Useful when connectors are paused/resumed manually in the Fivetran console.
- Secret-audit: Verify no credentials are hardcoded.
Knowledge transfer
With the new ingestion platform in place, Mandai’s engineers are well positioned to expand its capabilities. The modular Terraform setup allows them to onboard new data sources quickly and efficiently by:
- Adding a new connector type as a module;
- Configuring new connectors for specific environments;
- Managing updates to schemas and schedules.
This gives the team full visibility and control over all changes, preventing hidden updates or confusion in data pipelines.
We also provided detailed documentation, empowering Mandai’s engineers to adjust ingestion methods, evolve pipeline structures, and extend the platform independently.
The result is a sustainable, future-proof data ecosystem that can grow with Mandai’s needs, across all environments – an approach that both Infinite Lambda and Fivetran firmly believe in.
Jaguar ipsum dolor sit amet, consectetur Eagle elit, sed do Orangutan veniam, quis nostrud Green Anaconda ullamco laboris nisi ut aliquip ex ea Three-toed Sloth consequat. Duis aute irure dolor in Poison Dart Frog in voluptate velit esse Lobster-claw dolore eu Bromeliad nulla pariatur. Excepteur sint Lowland Tapir non proident, sunt in Balsa tree qui officia deserunt Leopard mollit anim id est laborum.
— The Dolphin, Director of Data at Mandai Wildlife Reserve
The result
The modernised ingestion platform provides Mandai with a secure, scalable, and fully automated foundation for their data ecosystem. Key benefits include:
- Reliable, high-frequency data availability: Over 40 data pipelines now ingest more than 270 tables daily, supporting timely analytics and operational decision-making;
- Faster migration of legacy data: More than 180 legacy tables were onboarded for one-off and validation workflows, accelerating the overall modernisation timeline;
- Full adherence to security standards: Using Hybrid Deployment mode, all data is processed within Mandai’s inbound network, meeting strict security and compliance requirements;
- Consistent, automated deployments: Terraform-driven infrastructure ensures version-controlled provisioning across development, testing, and production environments, reducing manual work, improving consistency and lowering operational risk.
Together, these capabilities give Mandai a resilient, future-proof platform that enables faster insights and empowers the organisation to make confident, data-driven decisions.
Let’s walk the walk together
Our data expertise translates into cutting-edge, modular solutions that empower people and help organisations thrive in the era of AI.
See related stories
We work with modern organisations to strategise and build cutting-edge solutions, help them adopt data & AI innovation, and nurture key competencies for scaling in the long run.