Why most African AI pilots never reach production, and the five things that change that

Most African generative AI pilots stall because the organisation treats the model as the product, when the product is actually the data, the evaluation, the permissions and the cost model wrapped around it. McKinsey reported in 2025 that more than 40 per cent of African institutions had experimented with or deployed GenAI, yet most of that work remained stuck in pilots. The five disciplines below are what separate the pilots that ship from the ones that quietly expire.

The pilot trap, as we see it from Nairobi

We have sat in enough steering committee meetings across Kenya, Nigeria, South Africa and the Gulf to recognise the pattern. A team builds a chat assistant on a foundation model in six weeks. The demo is impressive. Leadership asks for a production date. Then the questions begin: What happens when the agent gets a loan query wrong? Who approved it reading customer records? What will it cost at 50,000 conversations a month? Can it handle a customer who writes in Swahili and switches to English mid-sentence?

Nobody has answers because the pilot was built to prove the model could talk, not to prove the system could be trusted. The McKinsey finding (McKinsey, 2025) is consistent with what we observe: the technology is rarely the blocker. The operating model around it is.

The infrastructure picture has also shifted. Amazon Bedrock has been available in the AWS Africa (Cape Town) Region since November 2025, which gives teams with data residency obligations a route to run foundation models on the continent. The model catalogue in af-south-1 is narrower than in the US regions, so architecture decisions need to account for which models you can actually call from Cape Town and which still require a cross-region call with the data handling that implies.

The five things that change the outcome

Step 1: A data foundation that the agent can trust

An agent is only as good as what it can retrieve. In most African enterprises the data an agent needs sits across core banking, an ERP, a billing platform and a mobile money integration, each with its own identifiers and its own definition of “customer”. When the pilot used a hand-curated spreadsheet of fifty documents, retrieval looked flawless. Production means retrieval over millions of records with duplicates, stale entries and access rules.

What works in practice:

• Land the source systems into a governed lakehouse on Amazon S3 with AWS Lake Formation controlling who sees what, before building retrieval on top.
• Treat knowledge bases for Amazon Bedrock as a consumer of that governed layer, not a separate pipeline that someone maintains by hand.
• Decide early which fields an agent may never see. Lake Formation column and row filters enforce that at the data layer, not in the prompt.

We wrote a longer blueprint for this in Build the lakehouse before the agent.

Step 2: Evaluation suites, including Swahili ones

A pilot is judged by a few people clicking around. Production needs a repeatable test that runs every time the prompt, model or retrieval index changes. Without it, every improvement is a guess and every regression is discovered by a customer.

An evaluation suite is a set of real questions paired with expected outcomes and a scoring method. For a customer service agent at a Kenyan bank, that means a few hundred queries drawn from actual tickets, labelled by the operations team, covering balance enquiries, failed M-PESA transfers, loan eligibility and complaints.

The Swahili point matters more than most teams expect. Models that perform well on English benchmarks degrade on Swahili, Sheng and code-switched text in ways that are hard to predict. If your customers write in Swahili, your evaluation set must include Swahili prompts with Swahili-speaking reviewers scoring the answers. Build it once and it becomes the gate for every release.

Step 3: Scoped tool permissions and guardrails

Agentic AI differs from a chatbot in one critical way: the agent takes actions. It calls an API, writes to a database, sends a message. That is what makes it valuable and what makes it dangerous.

The discipline here is least privilege applied to the agent as if it were a junior employee:

Agent capability	Pilot approach	Production approach
Reading customer data	Broad database credential	Scoped IAM role, row-level filters via Lake Formation
Calling internal APIs	Shared API key	Per-tool credentials, allow-listed endpoints, rate limits
Content and topic control	Prompt instructions	Amazon Bedrock Guardrails for denied topics, PII masking, grounding checks
Logging	Console output	Every tool call recorded with inputs, outputs and the user context

Guardrails are not a substitute for good design, but they are the layer that catches prompt injection, off-topic drift and accidental disclosure of personal data. In a jurisdiction governed by the Kenya Data Protection Act 2019, having that layer and being able to show the audit trail is part of being able to answer the regulator.

Step 4: Human-in-the-loop for irreversible actions

Not every action needs a human. Looking up a balance does not. Reversing a transaction, approving a credit limit increase, deleting a record or sending a regulatory notification does.

The practical pattern is a confidence-and-consequence matrix. Actions that are low consequence and high confidence run automatically. Actions that are high consequence route to a human approval step, with the agent’s proposed action, its reasoning and the supporting evidence presented in a queue. AWS Step Functions is a natural fit for orchestrating this: the agent proposes, the workflow pauses, a person approves or rejects, and the workflow resumes.

Teams that skip this step either ship an agent that can do nothing useful, because legal insisted on removing every action, or one that does something expensive at 2am. The human-in-the-loop design is the compromise that lets both sides say yes.

Step 5: A cost model from day one

Cloud cost is the leading concern among African technology leaders, and agentic workloads are a new and unfamiliar cost shape. Token consumption scales with conversation length, retrieval scales with context size, and an agent that loops through five tool calls to answer one question costs five times what the pilot estimate assumed.

Build the model before the pilot ends:

• Measure tokens per conversation, tool calls per conversation and conversations per day during the pilot, and extrapolate to production volumes.
• Compare model tiers. A smaller model with good retrieval often outperforms a large model with poor retrieval, at a fraction of the cost.
• Cache aggressively. Prompt caching and response caching for repeated queries change the economics materially.
• Set budget alarms in AWS Budgets and put a hard ceiling on daily spend for the agent’s account, so an unexpected loop cannot run unbounded.
• Price it in the currency your finance team reports in. A model that looks fine in US dollars may look different after a currency move, and finance will ask.

The finance conversation is easier when the architecture team arrives with a per-conversation cost, a forecast and a plan to reduce it, rather than a surprise invoice.

A note on what the Cape Town region changes

Before Amazon Bedrock arrived in af-south-1, teams with strict residency obligations had to either accept cross-region inference or stay off foundation models altogether. That is no longer the case for many workloads. The constraint now is model availability: not every model in the Bedrock catalogue is available in Cape Town, so the architecture decision is which workloads need in-region inference and which can tolerate a call to Europe or the US with appropriate data handling. Routing by data classification, rather than picking one region for everything, is usually the right answer. Check the current regional availability on aws.amazon.com before committing a design.

What to do next

If you have a pilot that has stalled, the diagnostic is quick. Ask which of the five disciplines above you have in place. Most stalled pilots have one or none. The fix is rarely a different model; it is a four to eight week engineering effort to build the evaluation suite, scope the permissions, add the approval workflow and produce the cost model, and then re-present the system to the people who need to sign off.

Shujaa.ai is an AWS Advanced Tier Services Partner headquartered in Nairobi, holding the AWS AI Competency for Agentic AI Consulting Services and Generative AI Consulting Services. We help teams across East Africa take agents from demo to production on Amazon Bedrock. Learn more at AI and Agentic AI services.