Putting a GenAI Gateway in Front of Bedrock

The situation

A platform team looks after Bedrock access for eleven product services across four models in one region. Every service holds its own IAM role and calls the Bedrock runtime directly with the AWS SDK. That arrangement was fine at three services.

Four things have gone wrong since. Finance wants spend broken down per team, and only two of the eleven adopted tagged application inference profiles, so the rest arrive on the bill as one undifferentiated line. A nightly enrichment job saturated the on-demand pool for ninety minutes last month and dragged every interactive service down with it. No lever existed at the time that would have slowed that one caller and left the rest alone. A central guardrail exists, is referenced by eight services, and was never wired into the other three, which an audit found rather than the team finding it. And a fifth team has asked for a model that Bedrock does not host at all.

The proposal on the table is that no application calls Bedrock directly any more. Each one calls a single internal endpoint the platform team owns, and that endpoint makes the model call on the caller’s behalf. AWS files this shape under GenAI gateway architectures, and what the platform team wants from it is secure and compliant consumption patterns: one door, one policy, one meter.

What actually matters

A layer that changes what the caller sends and a layer that makes the call instead of the caller are different propositions with different bills. Moving the model id out of the code and into configuration, which is a solved problem with no hop in it, leaves the caller’s own credentials on the request and the caller’s own role in the CloudTrail record. Putting compute in the path replaces the caller. The principal that reaches Bedrock becomes the platform team’s, and so does the account quota the call consumes. The audit trail names the gateway unless the design goes out of its way to preserve who was behind it. Everything IAM was doing per team has to be either rebuilt inside the facade or deliberately carried through it, and that rebuild is the largest single piece of work in the whole proposal.

Weigh what only compute in the path can do. It can count tokens on every request, for every caller, without asking eleven teams to configure anything, which turns cost attribution from a convention into a property of the system. It can refuse a request: a hard cap on one caller, applied at the front door, is available nowhere else, because a direct caller with valid credentials is going to get its call through. It can hold a guardrail identifier so the safety policy runs on every request with no opt-out, rather than being a thing eight teams remembered. And it can front a provider Bedrock does not host, because once callers speak the platform team’s protocol the backend is a private matter. Ranked by how much they are worth here, metering comes first, the enforced guardrail second, and swapping models a distant third, since the swap is available without a facade at all.

Against that, four costs. There is a hop of latency on every call, which is small in absolute terms and lands hardest on the first token of a streaming response, where a reader notices it directly. There is a new failure domain: eleven services that previously failed independently now share a dependency, and it needs the availability of the most demanding of them. There is a quota funnel, because throttling that used to spread across eleven roles and eleven request paths now converges on one, so one team’s spike becomes everyone’s throttling event. And streaming does not survive every implementation of the hop. That one is a hard constraint rather than a tuning problem: an integration that buffers the whole response before returning it cannot deliver tokens incrementally, whatever the backend does.

The last consideration is what the facade does to signals. A layer put in the path to centralise observability and control can destroy the observability it was meant to provide. A gateway that catches a throttling response from Bedrock, retries it internally, and eventually returns a generic error has taken a precise piece of backpressure and handed the caller mush. The caller can no longer distinguish “slow down, come back shortly” from “this is broken”, so the backoff logic every client already has stops working. Observability across the hop has to be designed in, through structured access logs and trace propagation, or the facade becomes the place where diagnosis stops.

What we’ll filter on

1. Central metering. Are tokens counted per caller on every request, without each team having to opt in?
2. Per-team rate limiting. Can one runaway caller be capped without touching the other ten?
3. Guardrail with no bypass. Does every request pass the same safety policy, whatever the caller did or forgot?
4. Streaming. Does an incremental response survive the extra hop, or does the integration buffer it?
5. Identity at Bedrock. Does Bedrock still see which team is calling, so that least privilege API access to FMs and per-team cost attribution survive the indirection?
6. Operational weight. Added latency, added failure domain, and who carries the pager for it.

The landscape

Stay direct: IAM roles and tagged application inference profiles

The incumbent is not nothing, and it is the baseline every other option has to beat. Each service assumes its own role, the role is scoped to specific model or profile ARNs, and calls go to Bedrock over an interface VPC endpoint. Per-team cost attribution is available through tagged application inference profiles, and access governance through policy, boundaries and service control policies. Streaming works natively, latency is as low as it gets, and Bedrock sees exactly who is calling.

What it cannot do is compel. Every control here depends on each team configuring itself correctly, which is what produced the three services with no guardrail and the nine with no tags. There is no throttle that applies to one team, because Bedrock’s quotas are per account and region, not per role. And a caller with valid credentials will always reach the model.

An Amazon API Gateway REST API in front of an AWS Lambda proxy

The commodity shape. A REST API terminates the caller’s HTTPS request, and a Lambda function resolves the model, applies the guardrail, calls Bedrock, and returns. Using API Gateway to manage rate limiting is the part that carries most of the value. Usage plans bind an API key to a request rate, a burst allowance, and a quota over a day, week or month. Per-method throttling sets ceilings underneath that. Per-team API keys drop out of the same mechanism. Access logging is a configuration setting rather than code, request validation rejects malformed calls before they reach compute, and a private REST API keeps the endpoint reachable only from inside the VPC.

Streaming is where it stops. A standard API Gateway integration buffers the whole response before returning it, and the integration timeout caps how long a generation may run, so token-by-token delivery does not come out of this shape. There are two escapes. An API Gateway WebSocket API lets the Lambda push chunks back over an established connection, at the cost of a connection-management layer. Lambda response streaming through a function URL sidesteps API Gateway entirely, and takes the usage plans with it. Neither is free, and both are a delivery decision of their own.

A gateway container on Amazon ECS with AWS Fargate behind an Application Load Balancer

Run the facade as a long-lived process. The load balancer proxies rather than buffers, so server-sent events and chunked responses pass straight through to the caller and the first token arrives about when it would have without the hop. A container has no invocation timeout to design around and keeps warm connection pools to Bedrock rather than paying a cold start on a quiet path. It can also run whichever open-source gateway or bespoke service the team prefers, including one that fronts a non-Bedrock provider.

The bill arrives as work. There is no usage plan, so per-team rate limiting is something the team implements, typically as a token-bucket keyed on the caller with the counters in ElastiCache so all tasks agree. Capacity, scaling, patching and deployment are the team’s. This shape trades a pile of managed configuration for control over the response path.

AWS AppSync where the client already speaks GraphQL

If the callers are front ends already talking to a GraphQL API, adding a field backed by a resolver puts the model call inside the graph rather than beside it. AppSync handles authorisation per field, so which callers may invoke which model becomes part of the schema. Incremental output arrives as subscription events the resolver publishes over a WebSocket, which suits a client already holding subscriptions open.

For eleven backend services with no GraphQL between them, this is the wrong shape. It asks every caller to adopt a query language to reach a model, and the incremental delivery is subscription-shaped rather than an HTTP response, which suits a browser and irritates a batch job.

Evaluation

Side by side

Option	Central metering	Per-team rate limiting	Guardrail, no bypass	Streaming survives	Identity reaches Bedrock	No extra tier to run
Direct SDK calls, scoped roles, tagged profiles	✗	✗	✗	✓	✓	✓
API Gateway REST API + Lambda proxy	✓	✓	✓	✗	✗	✗
Gateway container on ECS with Fargate behind an ALB	✓	✗	✓	✓	✗	✗
AWS AppSync resolver	✓	✓	✓	✓	✗	✗

Three columns deserve their footnotes read out. Per-team rate limiting is a cross for the container because it is buildable rather than absent, and the build is a distributed counter plus the operational care that comes with one. Identity is a cross for every facade for the same reason in reverse: the collapse is the default, and preserving it is a design decision taken deliberately. Streaming is the only cross that no amount of effort removes, because a buffering integration is a property of the integration.

The direct row is worth reading as a whole rather than counting its ticks. It wins every column that measures cost and loses every column that measures control, which describes the trade precisely: the platform team is proposing to spend latency, availability and identity to obtain metering, throttling and enforcement.

Which shape fits which caller

The gates run in that order because the first two are properties of the caller that no gateway design changes, and the third is the one that splits the remaining traffic. A service whose audit obligation names its own principal at Bedrock is not a gateway candidate at all, however convenient the metering would be. A front end already holding GraphQL subscriptions open should not be handed a second protocol. What is left is the ordinary case, and it divides on whether the response streams.

The solution

Build the facade as an API Gateway REST API in front of a Lambda proxy, and give the streaming services a second route on the same hostname into a Fargate service behind an internal Application Load Balancer. One team owns both, one deployment pipeline ships both, and callers see one base URL with a path prefix deciding which path they take. Nine of the eleven services take the Lambda route today. The chat service and the summariser take the container route because their users watch text appear.

That split looks like two systems and behaves as one, because the parts that matter are shared: the same request contract, the same guardrail identifier, the same metering record, the same per-caller limits enforced before either backend is reached. What differs is the response path, which is the only thing the two backends actually disagree about.

Carry the caller’s identity through

The default facade collapses eleven principals into one execution role, which loses per-team cost attribution and the fine-grained access control that IAM was already providing. Preserve both by making the gateway act on the caller’s behalf rather than in its own name.

The caller authenticates to the API with SigV4 against its own role, so API Gateway’s IAM authorisation validates who is calling before compute runs. The Lambda then reads that principal from the request context and assumes a per-team role, passing the team identifier as a session tag on the AssumeRole call. Bedrock sees a session that names the team and CloudTrail records it. Role-based access control for model and data access carries on working, because each team’s role still allows only the model and profile ARNs that team is approved for. Invocation goes through that team’s tagged application inference profile ARN, so the cost meters where it belongs rather than against a shared execution role. What the facade adds on top is that a team can no longer skip the profile, since the gateway resolves it rather than trusting the caller to send one.

The cheaper variant, worth naming because it is what teams reach for first, is an API key mapped to a team in a lookup table with a single execution role behind it. It gives metering and throttling and gives up the identity chain, so a bug in the mapping is an authorisation bug. Keep the assumed role.

Per-team limits at the front door

Each team gets an API key bound to a usage plan, and the usage plan carries a steady-state request rate, a burst allowance, and a quota per period. Route-level throttling underneath it sets a ceiling per method, so a single expensive route cannot be driven at the rate a cheap one allows. The nightly job gets a plan sized to what the interactive services can spare overnight, and the interactive services get plans sized to their peaks with headroom.

Request rate is a proxy for the thing actually being protected, which is tokens. A plan that permits sixty requests a minute permits a very different load depending on how long the prompts are. Enforce both: the usage plan caps request rate at the edge, and the Lambda checks a per-team token budget in DynamoDB before invoking, rejecting with a clear error when the budget is spent. The first control is free and coarse, the second costs a read and is the one that matches how Bedrock charges.

One guardrail identifier, held by the gateway

The gateway holds the guardrail identifier and version, and applies the guardrailGuardrailA filter or rule applied to an LLM’s inputs or outputs to keep it inside safe, legal, or on-brand behaviour. on every request. Callers cannot pass their own, cannot disable it, and do not know which one is in force. Where a request needs a policy check separated from the model call, ApplyGuardrail evaluates the same guardrail independently, which lets the facade screen input before spending a model invocation and screen output afterwards, uniformly, whatever the backend.

This is the control the current estate is missing most visibly, since three services were never wired at all. It is also the one that converts a written policy into something enforced, because the enforcement point is now somewhere a product team cannot route around.

Keep the path private

Nothing here needs the public internet. The REST API is deployed as a private API reachable through an interface VPC endpoint, so only callers inside the VPC can reach it, and the resource policy narrows that further to the VPC endpoint id. The Lambda and the Fargate tasks run in private subnets and reach Bedrock through the Bedrock interface VPC endpoints over AWS PrivateLink, with an endpoint policy allowing only the actions and model ARNs the platform is meant to use. That gives two enforcement points on the same traffic: the IAM policy on the assumed role, and the endpoint policy on the door it goes through.

Do not swallow Bedrock’s throttling signal

The failure mode that hurts most is the one the facade introduces by accident. When Bedrock returns a throttling exception, the gateway must not retry indefinitely inside the request and must not flatten the response into a generic server error. Retry inside the Lambda with exponential backoff and jitter for a bounded number of attempts. If it still fails, return a 429 with a Retry-After header, so the caller’s own backoff has something to work with. Distinguish it from the gateway’s own 429 raised by a usage plan, because those two mean different things: one says the platform is full, the other says this team is over its allowance.

Then make the funnel visible. Structured access logs go to CloudWatch Logs carrying the team, the model, the guardrail outcome, the input and output token counts and the upstream latency on every line. That is what turns an access log into the metering record. X-Ray traces span the caller, the gateway and the Bedrock call, so a slow request can be attributed to a segment rather than argued about. Alarm on the gateway’s own error rate and p99 as a tier-one service. Alarm separately on the upstream throttling rate, because a rising one warns that the account quota, not the gateway, is the next constraint.

Worked example

The enrichment job goes wide again

The nightly job is rewritten to loop faster and starts issuing four times its usual rate at 02:00. Under the old arrangement it saturated the on-demand pool, every interactive service started seeing throttling exceptions, and the platform team’s only lever was to ring the team that owned the job.

Through the facade, the job’s usage plan quota is reached about eleven minutes in. API Gateway returns 429 to the job and nothing else changes: the interactive plans are untouched, the account quota is never approached, and Bedrock never sees the excess. The job’s client backs off, finishes late, and files no incident. The platform team sees a quota-exceeded metric on one usage plan and a flat line everywhere else, which localises the problem before anyone has to ask whose traffic it was.

A chat request that streams

A request arrives on the streaming path. The ALB routes it to a Fargate task. That task reads the caller principal from the SigV4 signature, assumes the chat team’s role with the team session tag, applies the guardrail to the input, and opens a streaming invocation against the team’s application inference profile. Chunks come back over the open connection as the model produces them, and the first token reaches the browser roughly one network hop later than it would have without the gateway.

Output screening is the part this path has to think about, because a guardrail cannot inspect text that has not been generated. The task buffers each chunk into a small window, applies the guardrail to the window, and forwards it only once it passes, which costs a fraction of a second of smoothness against a policy that actually runs. When a window fails, the connection closes with a policy error and the partial response already delivered is retracted by the client. Token counts are recorded when the stream completes, or when it aborts, so a cancelled generation still meters what it consumed.

What’s worth remembering

1. A gateway in front of Bedrock is compute in the request path, and the thing it changes first is identity: the principal reaching Bedrock becomes the gateway’s unless the design assumes a per-team role and carries the caller through as a session tag.
2. Per-request metering, a cap on a single runaway caller, and a guardrail nobody can skip are what only a facade delivers; swapping models and standardising the request shape are available from configuration and the Converse API with no hop at all.
3. API Gateway usage plans give per-team API keys, request rates, burst allowances and quotas as configuration, and a token budget checked in the backend is the second control that matches how Bedrock actually bills.
4. A standard API Gateway integration buffers the response, so incremental delivery needs a container behind a load balancer, a WebSocket API, or Lambda response streaming; that constraint is structural and cannot be tuned away.
5. Keep the whole path private with a private REST API and Bedrock interface VPC endpoints over PrivateLink, and narrow it twice with the assumed role’s IAM policy and the endpoint policy.
6. The gateway must pass Bedrock’s throttling signal through as a 429 with a retry hint rather than absorbing it, or every caller’s backoff logic goes blind and the facade becomes the place where diagnosis stops.