The AI Gateway Problem Financial Firms Are Solving Wrong

There is a pattern that repeats itself across financial institutions deploying AI at scale. It starts with a specific moment of infrastructure pain.

The data science team is running experiments on Anthropic's Claude. The trading desk wants ChatGPT for research summarization. The quant team has a model sitting on AWS Bedrock. And the infrastructure team has just realized that each of these providers has a completely different API format, a completely different authentication mechanism, a completely different approach to streaming responses, and a completely different error handling contract.

Someone searches for a solution. They find an open-source AI gateway. Within a week it is running in production. The immediate problem is solved.

Six months later, when the compliance team asks for an audit trail of every AI interaction, when the CISO asks how the firm would detect MNPI being sent to an external provider, when legal asks how the firm demonstrates EU AI Act compliance — the answer is the same: the gateway does not do that.

This is not a criticism of any specific tool. It is a structural gap between what AI gateway infrastructure is designed to do and what regulated financial institutions are now required to prove. Understanding that gap — and why so many firms have it — starts with understanding what they actually deployed.

The Problem AI Gateways Were Built to Solve

The AI gateway category exists because of a genuine and acute infrastructure problem: provider fragmentation.

Every major LLM provider has a different API contract. OpenAI uses one message format. Anthropic uses another. AWS Bedrock requires AWS SigV4 request signing — a complex cryptographic authentication mechanism that requires a different signing process for every single request, tied to IAM credentials, timestamps, and canonical request formatting. Azure OpenAI adds its own endpoint URL structure and API version query parameters. Google Vertex AI has its own authentication flow entirely.

Calling any one of these in production is manageable. Calling all of them — from the same application, with the same codebase, with failover between them — is a significant engineering problem that most firms do not want to solve themselves.

AI gateways address this by sitting between applications and LLM providers, presenting a single unified API surface — almost always OpenAI-compatible format — to everything upstream, while handling the per-provider complexity downstream.

The core capabilities that make these tools valuable are genuine:

Provider normalisation — one API format regardless of which model is behind it. Switching from ChatGPT to Claude without reformatting message arrays. Moving from a direct OpenAI connection to Bedrock without rewriting application code.

Load balancing and failover — if a provider returns a rate limit error, the gateway routes to a backup. For production financial workloads where a research tool going down during market hours has real consequences, this is operational infrastructure.

Virtual keys and cost tracking — instead of distributing real provider API keys to every developer, the firm issues gateway virtual keys. The gateway holds the real credentials centrally. Token-level spend tracking per team, per model, per day, with budget enforcement. This is the feature that finance and operations teams specifically ask for.

SigV4 abstraction — this deserves its own mention because it is the single most common reason financial institutions reach for a gateway. AWS Bedrock is the preferred AI deployment model for many financial firms because it keeps workloads inside existing AWS infrastructure with existing IAM governance. But direct Bedrock integration requires SigV4 signing — a per-request cryptographic operation that requires AWS SDK support to implement correctly and maintain as AWS updates specifications. A gateway that handles this internally means every application that needs Bedrock access just sends a Bearer token and the gateway handles the rest.

These are real problems. Tools that solve them well earn their place in the stack.

Why They Spread Quickly in Financial Services

The self-hosted deployment model was critical to financial services adoption.

Financial institutions are generally reluctant to route internal data through third-party SaaS products they do not control. An AI gateway that can be deployed entirely within the firm's own AWS account, on-premise, or in a private cloud — with no data leaving the firm's perimeter to reach the gateway vendor — is a much easier conversation with a CISO than a managed cloud proxy.

The open-source licensing of the most popular options added to this: if you can read the source code, audit it, and run it yourself, the security review conversation is simpler. The tool works, it is free to deploy, and it solves the immediate problem.

For small teams and development environments, this was sufficient. For larger financial institutions, the governance features they actually needed — SSO integration, role-based access control, team-level budget enforcement — were often gated behind enterprise licensing. But that friction came later. The initial deployment was fast, and fast deployments become entrenched.

By the time organizations had a few hundred developers using AI tools routed through their gateway, it had become infrastructure — not a tool people questioned, but a layer people built on top of.

The Compliance Gap That Was Always There

Here is the structural problem that is now becoming visible as the EU AI Act enforcement deadline approaches.

AI gateways manage requests. They route them, normalise them, track their cost, and return responses. This is infrastructure-level work and they do it well.

They do not govern AI systems. And the EU AI Act is not a regulation about AI infrastructure. It is a regulation about AI system governance.

The distinction matters precisely because the two things look similar from the outside. A gateway logs requests. The EU AI Act requires automatic logging. A gateway has user management. The EU AI Act requires human oversight attribution. A gateway has access controls. The EU AI Act requires tamper-resistant audit records.

In each case, the gateway has something that sounds like what the regulation requires. In each case, what the gateway has is not sufficient.

On logging: Article 12 requires automatic logging capabilities enabling traceability throughout the system lifecycle, capturing event logs identifying risks. The critical phrase is tamper-resistant. A log stored in a writable database — regardless of access controls around it — is not tamper-resistant. A database administrator with sufficient privileges can modify records without detection. The regulation requires an architectural mechanism that makes modification detectable, not merely access-controlled. Cryptographic chaining, write-once storage, or an equivalent — not a database table with an access policy.

Gateway logs are operational data. They tell you what happened at the routing layer. They do not produce the tamper-evident chain of custody that regulators are now asking for.

On identity attribution: Virtual keys issued by a gateway create a useful abstraction — the firm controls credentials centrally rather than distributing raw API keys. But a gateway virtual key is not the same as a verified SSO identity. When a regulator asks who sent a specific AI request and what their role was at the time, a gateway key identifier requires a lookup into the gateway's user management system, which requires a lookup into the firm's identity provider, which may or may not have been maintained accurately. The attribution chain is reconstructable in theory. In practice, under audit pressure, it frequently is not.

What Article 14 requires is named individuals with documented training and specific accountability for AI system oversight — not a key identifier that can be traced backward with effort.

On content classification: Gateways route requests based on routing rules — which model, which provider, which region. They do not inspect the content of those requests for regulatory classification purposes. A developer sending a prompt that contains MNPI — material non-public information — through an AI gateway receives a response from an external LLM. The gateway records the token count and the cost. It has no mechanism for detecting the MNPI, blocking the request, or creating a compliance record of the attempt.

For a financial institution operating under MiFID II, SEC Rule 10b-5, or the EU AI Act's own data governance requirements, this is the core exposure. The question is not whether a prompt contained sensitive information — it is whether the firm can prove it had a system to detect and prevent sensitive information from leaving its perimeter. Gateway logs cannot provide that proof.

Where the Gap Actually Sits

The pattern that emerges from looking across financial institution AI deployments is consistent.

Firms that deployed AI gateways solved the problem they had in 2023: provider fragmentation, credential management, cost visibility, SigV4 complexity. These were real problems and the gateways solved them.

The problem they now have in 2026 is different: provable governance, tamper-evident audit, content classification, SSO identity attribution, and Article 12 compliance. These are not features that get added to a routing proxy. They require a different architectural layer with a different purpose.

Consider what happens when a financial institution using only a gateway is asked by a regulator to produce its EU AI Act audit records for AI-assisted trading research over a six-month period:

The gateway logs show which models were called, when, by which virtual key, at what token cost. This is accurate. It is also not what the regulator asked for. The regulator asked for evidence that every AI action was attributed to a named individual with a defined role, that content was classified before leaving the perimeter, that no MNPI reached an external provider, and that the records were not altered after the fact. None of that is in the gateway logs.

The firm then faces a choice: attempt to reconstruct the missing evidence, or acknowledge the gap and accept the consequences. Neither is a good outcome four months before a deadline that was set two years ago.

The Architecture That Addresses the Full Problem

The gap between what AI gateways provide and what regulated financial institutions need is not a reason to replace gateways. They solve a real problem well and that problem does not go away.

The architecture that addresses the full problem is additive — a governance and compliance layer sitting above the gateway, not instead of it.

Developer tools
(Claude Code, Cursor, internal applications)
↓
Compliance and governance layer
→ SSO identity resolution
→ Content classification (MNPI, PII, PHI)
→ Classification gate — block or permit
→ Tamper-evident audit write (hash-chained)
↓
AI gateway
→ Provider routing
→ SigV4 handling
→ Cost tracking
→ Virtual key management
↓
LLM providers
(AWS Bedrock, Anthropic, OpenAI, Azure)

In this architecture, the gateway continues doing exactly what it does well. The compliance layer above it addresses the obligations the gateway was never designed to meet. Every AI request is classified before it reaches the gateway. Every permitted request is attributed to a verified SSO identity. Every interaction is logged in a tamper-evident hash chain that produces the Article 12 audit trail regulators require. Every blocked request — one that contained MNPI, PII, or other sensitive content — is logged with the full classification reason and the identity of who attempted it.

The gateway never sees the sensitive request. The gateway never needs to be modified. The compliance layer adds its obligation without touching the routing infrastructure the engineering team already depends on.

For financial institutions that have already invested in AI gateway infrastructure, the path to EU AI Act compliance does not require replacing anything they have built. It requires adding the layer that was always missing — the one designed to govern AI systems, not just route AI requests.

The Question to Ask Before August 2, 2026

The EU AI Act deadline for high-risk AI in financial services is 113 days from the date this article was published. The question is not whether your firm has an AI gateway. Most firms do.

The question is what sits above it.

If the answer is nothing — if the gateway is the entire AI governance story — then what the firm has is infrastructure, not compliance. Operational logging, not an audit trail. Key identifiers, not SSO attribution. Routing controls, not content classification.

The difference between those two things is the difference between being able to demonstrate EU AI Act compliance after August 2, 2026 and facing enforcement from regulators who are already operational in several EU member states.

The gateway solved the problem of 2023. The compliance problem of 2026 requires something built specifically for it.