Build the lakehouse before the agent: a data platform blueprint for African enterprises
AI agents fail on siloed data. A practical AWS lakehouse blueprint with S3, Iceberg, Glue, Lake Formation, Athena and Redshift, governed for DPA, POPIA, NDPA.
Every AI agent is a query against your data, and if the data lives in five systems that disagree with each other, the agent will disagree with reality. For African enterprises the sensible order is to build a governed lakehouse first: Amazon S3 with Apache Iceberg tables, AWS Glue for ingestion and cataloguing, AWS Lake Formation for access control, Amazon Athena and Amazon Redshift for query, Amazon QuickSight for reporting and Amazon Kinesis for streaming. With a foundation in place in about four weeks using an accelerator, the agents that follow have something solid to stand on.
The silo problem, in its African form
The systems are familiar: a core banking platform, an ERP, a billing system, a CRM, a mobile money integration with M-PESA or Airtel Money, and a growing collection of SaaS tools. Each holds a version of the customer, the transaction and the product. None of them agrees on identifiers. The mobile money integration knows the customer by phone number, the core banking system by account number, the ERP by a vendor code, and the CRM by an email address that was accurate in 2022.
Analysts cope by exporting spreadsheets. Management copes by trusting the number from whichever system the finance team uses. An AI agent cannot cope at all: asked “what is this customer’s total exposure?”, it either answers from one system and is wrong, or tries to join across systems and is slow, inconsistent and impossible to audit.
The lakehouse is the fix. One governed copy of each system of record, landed continuously, catalogued, reconciled and secured, from which both humans and agents read.
The blueprint
Storage and table format
Amazon S3 is the storage layer. Apache Iceberg is the table format on top of it. Iceberg matters because it gives you transactions, schema evolution, time travel and efficient updates on files in S3, which is what a system of record needs and what plain Parquet files on S3 do not provide. Every major AWS analytics engine reads and writes Iceberg natively, so you are not locked into one compute choice.
Organise S3 into three zones: raw (exactly as received from source, immutable), curated (cleaned, typed, deduplicated Iceberg tables) and consumption (domain data products and aggregates). Lifecycle rules move raw data to colder storage classes after the curated version is validated.
Ingestion with change data capture
Nightly batch exports are how silos stay silos. Change data capture (CDC) streams every insert, update and delete from the source database as it happens. AWS Database Migration Service handles CDC from most commercial and open source databases into S3, and Amazon Kinesis Data Streams carries higher-volume event streams such as transaction feeds or mobile money callbacks. AWS Glue jobs transform the landed changes into curated Iceberg tables.
The result is a lakehouse that is minutes behind the source systems rather than a day, which is what fraud detection, real-time risk and conversational agents actually need.
Catalogue and governance
AWS Glue Data Catalog is the single catalogue. Every table, its schema, its owner and its classification lives there. AWS Lake Formation enforces access: who can see which tables, which columns and which rows. This is where regulatory compliance is implemented, not in each consuming application.
Query and analytics
Amazon Athena for ad hoc SQL over the lakehouse, paid per query. Amazon Redshift for the sustained analytical workloads and for the finance and risk teams who need warehouse performance; Redshift reads Iceberg tables in S3 directly, so there is one copy of the data. Amazon QuickSight for dashboards and embedded reporting.
Serving AI
The curated and consumption zones become the source for retrieval-augmented generation, for knowledge bases in Amazon Bedrock and for the tool calls an agent makes. Feature stores, built on Amazon SageMaker Feature Store or as Iceberg tables with a well-defined contract, supply machine learning models with consistent, point-in-time-correct features. When an agent and a credit model both read the same customer feature from the same table, their answers agree.
Governance mapped to the regulations
Three data protection regimes cover most of the markets we work in. The lakehouse design must satisfy all of them where the enterprise operates across borders.
| Requirement | Kenya DPA 2019 | South Africa POPIA | Nigeria NDPA 2023 | Lakehouse implementation |
|---|---|---|---|---|
| Lawful basis and purpose limitation | Required | Required | Required | Purpose tag on each dataset in Glue Data Catalog; Lake Formation grants by purpose |
| Data localisation | Regulation 26 categories must stay in Kenya | No general localisation, transfer conditions apply | Transfer conditions apply | Region and Outpost placement per dataset classification |
| Access control and minimisation | Required | Required | Required | Column and row-level filters in Lake Formation; no direct S3 access for consumers |
| Data subject rights | Access, correction, erasure | Access, correction, deletion | Access, correction, erasure | Iceberg row-level deletes with audit; lineage via Glue |
| Breach notification | 72 hours to ODPC | Without undue delay to Regulator | 72 hours to NDPC | CloudTrail and Lake Formation audit logs feeding a security pipeline |
| Cross-border transfer | ODPC guidance and Kenyan standard contractual clauses | Adequacy or contractual safeguards | NDPC-approved mechanisms | Replication rules per classification; Open-tier only to non-African regions |
The principle is that governance is a property of the platform. A consuming application, a dashboard or an agent does not need to know the rules, because it cannot see data it is not entitled to see.
Data products per domain
A lakehouse that is one giant pool of tables becomes a swamp within a year. The structure that lasts is domain ownership: the lending team owns the lending data products, the payments team owns payments, finance owns the general ledger products. Each domain publishes a small number of well-documented tables in the consumption zone with a defined schema, a refresh contract, an owner and a quality score.
Consumers, including AI agents, read data products rather than raw tables. When the lending team changes a source system, they update their data product and its consumers are insulated. This is the practical version of data mesh, without the ceremony.
The four-week foundation
Built from scratch, a lakehouse of this shape takes months, mostly spent on undifferentiated work: landing zone, networking, IAM, Glue job scaffolding, catalogue conventions, Lake Formation patterns, CDC pipelines and monitoring. An accelerator packages that work.
Shujaa’s LakeStack accelerator deploys the blueprint above as infrastructure as code, with opinionated conventions for zones, naming, tagging, classification and access. A typical four-week foundation engagement runs:
- Week 1: discovery of source systems, data classification against the applicable regulations, deployment of the LakeStack baseline into the customer’s AWS accounts.
- Week 2: CDC pipelines for the first two or three systems of record, raw and curated zones populated, catalogue and Lake Formation policies applied.
- Week 3: first domain data products built with the business owners, Athena and QuickSight access for analysts, reconciliation against source reports.
- Week 4: Redshift or feature store enablement as needed, monitoring and cost controls, handover with runbooks and a backlog for the next domains.
At the end of four weeks the enterprise has a working, governed platform with real data in it, not a design document. Further domains onboard in one to two weeks each.
Why this comes before the agent
An agent built on a lakehouse gets three things for free: a single, reconciled answer to any question about a customer or a transaction; access control that the agent inherits rather than re-implements; and an audit trail showing exactly what data informed each response. An agent built on direct connections to five silos gets none of those, and the engineering effort to add them afterwards exceeds the effort to build the lakehouse first.
The order is therefore not a preference. It is the shorter path.
What to do next
If your organisation has an AI initiative on the roadmap and your data still lives in silos, the useful first step is a data platform assessment: an inventory of the systems of record, a classification of their data against the regulations that apply to you, and a target lakehouse design with a sequencing of domains. From there the four-week foundation is a defined scope with a defined price.
Shujaa.ai is an AWS Advanced Tier Services Partner headquartered in Nairobi, holding the AWS Data and Analytics Competency. We build governed lakehouses on AWS for banks, fintechs, telecoms and public entities across East Africa using the LakeStack accelerator. See our data and analytics services.
AWS Solutions Architects, Nairobi. Shujaa.ai is an AWS Advanced Tier Services Partner headquartered in Nairobi.
