Putting a GenAI Gateway in Front of Bedrock

The situation

A platform team looks after Bedrock access for eleven product services across four models in one region. Every service holds its own IAM role and calls the Bedrock runtime directly with the AWS SDK. That arrangement was fine at three services.

Four things have gone wrong since. Finance wants spend broken down per team, and only two of the eleven adopted tagged application inference profiles, so the rest arrive on the bill as one undifferentiated line. A nightly enrichment job saturated the on-demand pool for ninety minutes last month and dragged every interactive service down with it. No lever existed at the time that would have slowed that one caller and left the rest alone. A central guardrail exists, is referenced by eight services, and was never wired into the other three, which an audit found rather than the team finding it. And a fifth team has asked for a model that Bedrock does not host at all.

The proposal on the table is that no application calls Bedrock directly any more. Each one calls a single internal endpoint the platform team owns, and that endpoint makes the model call on the caller’s behalf. AWS Prescriptive Guidance describes this shape as a generative AI gateway, a model abstraction layer that centralises rate limiting for different consumer groups and logs token consumption for chargeback. One door, one policy, one meter.

What actually matters

A layer that changes what the caller sends and a layer that makes the call instead of the caller are different propositions with different consequences. Moving the model id out of the code and into configuration, which is a solved problem with no hop in it, leaves the caller’s own credentials on the request and the caller’s own role in the CloudTrail record. Putting compute in the path replaces the caller. The principal that reaches Bedrock becomes the platform team’s, and so does the account quota the call consumes. The audit trail names the gateway unless the design goes out of its way to preserve who was behind it. Everything IAM was doing per team has to be either rebuilt inside the facade or deliberately carried through it, and that rebuild is the largest single piece of work in the whole proposal.

Weigh what only compute in the path can do. It can count tokens on every request, for every caller, without asking eleven teams to configure anything, which turns cost attribution from a convention into a property of the system. It can reject a request before it reaches the model, which is unavailable anywhere else, because a direct caller with valid credentials will reach Bedrock. It can hold a guardrail identifier so the safety policy runs on every request with no opt-out, rather than being a thing eight teams remembered. And it can front a provider Bedrock does not host, because once callers speak the platform team’s protocol the backend is a private matter.

Against that, four things the facade adds. There is a hop of latency on every call, which is small in absolute terms and lands hardest on the first token of a streaming response, where a reader notices it directly. There is a new failure domain: eleven services that previously failed independently now share a dependency, and it needs the availability of the most demanding of them. There is a quota funnel, because throttling that used to spread across eleven roles and eleven request paths now converges on one, so one team’s spike becomes everyone’s throttling event. And incremental delivery has to be configured deliberately at every layer, because buffering the whole response is the default nearly everywhere, and a layer left on that default erases token-by-token delivery however the backend behaves.

The last consideration is what the facade does to signals. A gateway that catches a throttling response from Bedrock, retries it internally, and returns a generic error has taken a precise piece of backpressure and flattened it. The caller can no longer separate “come back shortly” from “this is broken”, so the backoff logic every client already has stops working.

What we’ll filter on

1. Central metering. Are tokens counted per caller on every request, without each team having to opt in?
2. Per-team rate limiting. Can one runaway caller be capped without touching the other ten?
3. Guardrail with no bypass. Does every request pass the same safety policy, whatever the caller did or forgot?
4. Streaming. Can the hop be configured to forward chunks as they arrive, or does it buffer the whole response?
5. Identity at Bedrock. Does Bedrock still see which team is calling, so that least privilege API access to FMs and per-team cost attribution survive the indirection?
6. Operational weight. Added latency, added failure domain, and who carries the pager for it.

The landscape

Stay direct: IAM roles and tagged application inference profiles

The incumbent is the baseline every other option has to beat. Each service assumes its own role, the role is scoped to specific model or profile ARNs, and calls go to Bedrock over an interface VPC endpoint. Per-team cost attribution is available through tagged application inference profiles, and access governance through policy, boundaries and service control policies. Streaming works natively, latency is as low as it gets, and Bedrock sees exactly who is calling.

What it cannot do is compel. Every control here depends on each team configuring itself correctly, which is what produced the three services with no guardrail and the nine with no tags. There is no throttle that applies to one team, because Bedrock’s inference quotas are scoped per account, per model, per Region, and are mostly token-based rather than per-role. And a caller with valid credentials will always reach the model.

An Amazon API Gateway REST API in front of an AWS Lambda proxy

The commodity shape. A REST API terminates the caller’s HTTPS request, and a Lambda function resolves the model, applies the guardrail, calls Bedrock, and returns. Rate limiting is the part that carries most of the value. Usage plans bind an API key to a request rate, a burst allowance, and a quota over a day, week or month. Per-method throttling sets ceilings underneath that. Per-team API keys drop out of the same mechanism. Access logging is a configuration setting rather than code, and request validation rejects malformed calls before they reach compute.

Streaming used to be where this shape stopped. Since November 2025, a REST API integration carries a response transfer mode, and setting it to STREAM on a Lambda proxy or HTTP proxy integration makes API Gateway forward bytes as they arrive instead of waiting for the whole response. That lifts the 10 MB response cap and the 29-second integration timeout to a 15-minute stream, and it works on every REST API endpoint type, private ones included. The default is still BUFFERED, so a facade built without thinking about it delivers nothing until the generation finishes. Three features go away in STREAM mode, all of them ones that need the whole body in hand: endpoint caching, content encoding, and response transformation with VTL. Idle timeouts still apply, at five minutes for a regional or private endpoint. This is a delivery decision of its own, taken per route rather than once for the API.

A gateway container on Amazon ECS with AWS Fargate behind an Application Load Balancer

Run the facade as a long-lived process. The load balancer proxies rather than buffers, so server-sent events and chunked responses pass straight through to the caller, subject to the ALB’s 60-second idle timeout, which is adjustable. A container has no invocation timeout to design around and keeps warm connection pools to Bedrock rather than absorbing a cold start on a quiet path. It can also run whichever open-source gateway or bespoke service the team prefers, including one that fronts a non-Bedrock provider, and it is not confined to the runtimes where Lambda streams natively.

The work lands on the team. There is no usage plan, so per-team rate limiting is something the team implements, typically as a token bucket keyed on the caller with the counters in ElastiCache so all tasks agree. Capacity, scaling, patching and deployment are the team’s.

AWS AppSync where the client already speaks GraphQL

If the callers are front ends already talking to a GraphQL API, adding a field backed by a resolver puts the model call inside the graph rather than beside it. AppSync has a Bedrock runtime data source, so a resolver calls InvokeModel or Converse directly, with a guardrail identifier in the request. Authorisation is handled per field, so which callers may invoke which model becomes part of the schema.

The constraints are tight. That data source only does synchronous invocations that finish inside ten seconds, and it cannot call Bedrock’s stream APIs at all; AppSync caps request execution at thirty seconds regardless. Progressive output means a different pattern: the resolver calls a Lambda in event mode, returns immediately, and the function publishes mutations that fire subscriptions over a WebSocket. For eleven backend services with no GraphQL between them that is two moving parts too many, and it asks every caller to adopt a query language to reach a model.

Evaluation

Side by side

Option	Central metering	Per-team rate limiting	Guardrail, no bypass	Streaming survives	Identity reaches Bedrock	No extra tier to run
Direct SDK calls, scoped roles, tagged profiles	✗	✗	✗	✓	✓	✓
API Gateway REST API + Lambda proxy, STREAM mode	✓	✓	✓	✓	✗	✗
Gateway container on ECS with Fargate behind an ALB	✓	✗	✓	✓	✗	✗
AWS AppSync resolver, Bedrock runtime data source	✓	✗	✓	✗	✗	✗

Three columns deserve their footnotes read out. Per-team rate limiting is a cross for the container and for AppSync because it is buildable rather than absent, and the build is a distributed counter plus the operational care that comes with one. Identity is a cross for every facade for the same reason in reverse: the collapse is the default, and preserving it is a deliberate design decision. The streaming cross against AppSync cannot be configured away, since its Bedrock data source has no access to the stream APIs; the API Gateway tick depends on setting the transfer mode, which is not the default.

Which shape fits which caller

The first two gates are properties of the caller that no gateway design changes. A service whose audit obligation names its own principal at Bedrock is not a gateway candidate, however convenient the metering would be. A front end already resolving GraphQL fields should not be handed a second protocol, provided its calls stay inside the ten seconds the Bedrock data source allows. What is left divides on what has to sit behind the door.

The solution

Build the facade as one private API Gateway REST API in front of a Lambda proxy, with every integration set to STREAM transfer mode. All eleven services take that route, the chat service and the summariser included, because API Gateway now forwards chunks as the function produces them. Streaming from Lambda is native on the Node.js managed runtimes; other languages need a custom runtime or the Lambda Web Adapter, which is worth settling before the language choice hardens.

The fifth team, the one wanting a model Bedrock does not host, gets a second route on the same API: an HTTP proxy private integration through a VPC link to an internal Application Load Balancer in front of a Fargate service, also in STREAM mode. One team owns both, one deployment pipeline ships both, and callers see one base URL with a path prefix deciding which backend answers. The parts that matter are shared: the same request contract, the same guardrail identifier, the same metering record, the same per-caller limits applied before either backend is reached.

Carry the caller’s identity through

The default facade collapses eleven principals into one execution role, which loses per-team cost attribution and the fine-grained access control that IAM was already providing. Preserve both by making the gateway act on the caller’s behalf rather than in its own name.

The caller authenticates to the API with SigV4 against its own role, so API Gateway’s IAM authorisation validates who is calling before compute runs. The Lambda then reads that principal from the request context and assumes a per-team role, passing the team identifier as a session tag on the AssumeRole call, which needs sts:TagSession in the target role’s trust policy. Bedrock then sees a session that names the team, the Bedrock CloudTrail record carries the assumed-role session, and the AssumeRole event records the principal tags alongside it. Role-based access control for model and data access carries on working, because each team’s role still allows only the model and profile ARNs that team is approved for. Invocation goes through that team’s tagged application inference profile ARN, and those tags reach Cost Explorer as cost allocation tags, so usage meters where it belongs rather than against a shared execution role. What the facade adds is that a team can no longer skip the profile, since the gateway resolves it rather than trusting the caller to send one.

The simpler variant, worth naming because it is what teams reach for first, is an API key mapped to a team in a lookup table with a single execution role behind it. It gives metering and throttling and gives up the identity chain, so a bug in the mapping is an authorisation bug. AWS is explicit that API keys are not an authentication mechanism. Keep the assumed role.

Per-team limits at the front door

Each team gets an API key bound to a usage plan, and the usage plan carries a steady-state request rate, a burst allowance, and a quota over a day, a week or a month. Route-level throttling underneath it sets a ceiling per method, so a single expensive route cannot be driven at the rate a cheaper one allows.

Two limits on that mechanism matter. AWS applies usage plan throttles and quotas on a best-effort basis and says not to rely on them to control costs, so a team can overshoot its plan before the edge catches up. And request rate is a proxy for the thing being protected, which is tokens: Bedrock’s own quotas are largely token-per-minute, and a plan permitting sixty requests a minute permits wildly different token volumes depending on prompt length. Enforce both. The usage plan caps request rate at the edge, and the Lambda checks a per-team token budget in DynamoDB before invoking, returning a clear error when the budget is spent. The second control adds a read per request and is the one that matches how Bedrock meters.

One guardrail identifier, held by the gateway

The gateway holds the guardrail identifier and version, and applies the guardrailGuardrailA filter or rule applied to an LLM’s inputs or outputs to keep it inside safe, legal, or on-brand behaviour. on every request. Callers cannot pass their own, cannot disable it, and do not know which one is in force. Where a request needs a policy check separated from the model call, ApplyGuardrail evaluates the same guardrail against text with no model invocation at all, taking a source of INPUT or OUTPUT, which lets the facade screen input before it commits to a generation and screen output afterwards, uniformly, whatever the backend.

Keep the path private

Nothing here needs the public internet. The REST API is deployed as a private API reachable through an interface VPC endpoint, so only callers inside the VPC can reach it, and an aws:SourceVpce condition in the resource policy narrows that further to the endpoint id. The Lambda and the Fargate tasks run in private subnets and reach Bedrock through the bedrock-runtime interface VPC endpoint over AWS PrivateLink, with an endpoint policy allowing only the actions and model ARNs the platform is meant to use. That gives two enforcement points on the same traffic: the IAM policy on the assumed role, and the endpoint policy on the door it goes through.

Do not swallow Bedrock’s throttling signal

The failure mode that hurts most is the one the facade introduces by accident. Bedrock signals a breached account quota with ThrottlingException, which arrives as an HTTP 429, and the gateway must neither retry indefinitely inside the request nor flatten the response into a generic server error. Retry inside the Lambda with exponential backoff and jitter for a bounded number of attempts. If it still fails, return a 429 with a Retry-After header, so the caller’s own backoff has something to work with. Distinguish it from the gateway’s own 429 raised by a usage plan, because those two mean different things: one says the account quota is exhausted, the other says this team is over its allowance.

Then make the funnel visible. Structured access logs go to CloudWatch Logs carrying the team, the model, the guardrail outcome, the input and output token counts and the upstream latency on every line, which is what turns an access log into the metering record. X-Ray traces span the caller, the gateway and the Bedrock call, so a slow request is attributed to a segment rather than argued about. Alarm on the gateway’s own error rate and p99 as a tier-one service, and separately on the upstream throttling rate, because a rising one warns that the account quota is the next constraint.

Worked example

The enrichment job goes wide again

The nightly job is rewritten to loop faster and starts issuing four times its usual rate at 02:00, exactly as it did the month before. Through the facade, the job’s usage plan quota is reached about eleven minutes in. API Gateway starts returning 429 to the job, the interactive plans are untouched, and the bulk of the excess never reaches Bedrock. Some of it does: usage plan enforcement is best effort, so a little leaks past the plan before the edge catches up, which is why the Lambda’s DynamoDB token budget sits behind it. The job’s client backs off, finishes late, and files no incident. The platform team sees a quota-exceeded metric on one usage plan and a flat line everywhere else, which localises the problem before anyone has to ask whose traffic it was.

A chat request that streams

A request arrives on the chat route, whose integration is in STREAM transfer mode. The Lambda reads the caller principal from the request context, assumes the chat team’s role with the team session tag, and calls ConverseStream against the team’s application inference profile with the guardrail identifier in guardrailConfig. It emits the metadata header and delimiter API Gateway expects, then writes chunks as they arrive, and the first token reaches the browser roughly one network hop later than it would have without the gateway.

Output screening is handled by the guardrail rather than by hand. In its default synchronous stream mode, Bedrock buffers one or more response chunks and applies the policies before releasing them, which adds a little latency to each chunk and scans all of it. Asynchronous mode releases chunks immediately and blocks subsequent ones once a violation is found, so some unscreened text reaches the reader first; it also cannot mask sensitive information. Synchronous is the right default for a shared facade. Token counts are recorded when the stream completes, and when it aborts, so a cancelled generation still meters what it consumed.

What’s worth remembering

1. A gateway in front of Bedrock is compute in the request path, and the thing it changes first is identity: the principal reaching Bedrock becomes the gateway’s unless the design assumes a per-team role and carries the caller through as a session tag.
2. Per-request metering, a cap on a single runaway caller, and a guardrail nobody can skip are what only a facade delivers; swapping models and standardising the request shape are available from configuration and the Converse API with no hop at all.
3. API Gateway usage plans give per-team API keys, request rates, burst allowances and daily, weekly or monthly quotas as configuration, but AWS applies them on a best-effort basis, so pair them with a token budget checked in the backend against how Bedrock actually meters.
4. A REST API integration buffers by default; setting the response transfer mode to STREAM on a Lambda or HTTP proxy integration forwards chunks as they arrive, lifts the 10 MB and 29-second ceilings to a 15-minute stream, and rules out endpoint caching, content encoding and VTL response transformation.
5. Keep the whole path private with a private REST API and Bedrock interface VPC endpoints over PrivateLink, and narrow it twice with the assumed role’s IAM policy and the endpoint policy.
6. The gateway must pass Bedrock’s throttling signal through as a 429 with a retry hint rather than absorbing it, or every caller’s backoff logic goes blind and the facade becomes the place where diagnosis stops.