Streaming Responses to Cut First-Token Latency

The situation

The support assistant has grown up. Responses are often 300-500 tokensTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word. now, more context, more careful reasoning, better answers. The user-facing latency has grown with them: 4.2 seconds median to first visible response, 5.8 seconds at p95. Product shows a dashboard: 11% of sessions now end during the wait, against 7% six weeks ago, and almost all of them are users who closed the tab before the reply landed.

The technical baseline today is one synchronous call per turn: browser → API Gateway → Lambda → Converse (non-streaming) → Lambda response → API Gateway → browser. The entire reply has to generate before anything reaches the user. No progress indicator beyond a spinner.

Product wants the typing-animation pattern, tokens arriving as they’re generated, first token visible within one second. Engineering needs to understand how the plumbing changes and where the sharp edges live: ConverseStream, Lambda response streaming, which front doors can forward bytes instead of collecting them, the WebSocket vs Server-Sent-Events choice on the browser, and how error handling changes when the response is in flight.

What actually matters

Streaming changes the response shape. Instead of one request-reply, the streaming variant of the inference API returns an event stream: a message-start event carrying the role, then one or more content-block deltas per block, a content-block stop, a message stop carrying the reason generation ended, and a final metadata event with token usage and latency. Content-block start events appear for tool use. The SDK presents the whole sequence as an iterator the server-side handler reads.

The next decision is how the service surfaces that stream to the client. Every hop between the model and the browser has to forward bytes as they arrive rather than buffering the whole reply. The compute layer has a switch for that. The front door either forwards chunks or collects the whole body first, and a hop that collects re-buffers the stream back into one late response no matter what the model emitted.

After that comes the wire protocol to the browser. The two real options are a unidirectional text-event stream over plain HTTP and a bidirectional persistent connection. Event streams are simpler: a text protocol over an existing HTTP request, native browser support, no connection-lifecycle code. A bidirectional connection is worth the extra machinery when the browser also pushes structured data mid-stream, which is rare for chat and common for collaborative tools.

Then there is what happens when things go wrong. A streamed response that fails halfway leaves half-delivered state, so the client has to distinguish a stream that broke from one that ended cleanly, and there is no resume-from-token primitive to retry against. A synchronous call has one timeout; a streamed call has three, covering time to first byte, time between bytes, and total connection time. Tool calls add their own shape: when the assistant invokes a tool, the stream ends with a tool-use stop reason, the handler runs the tool, and a second call carries the result back for generation to continue. The UI needs a “thinking” state to cover that gap as well as a typing animation. Tools here means Converse tool use, the client-side function-calling pattern, and not Amazon Bedrock Agents Classic, which went into maintenance mode on 30 July 2026 and is closed to accounts with no prior usage.

What we’ll filter on

1. Time to first token: how fast the first byte reaches the user.
2. Wire-protocol overhead: how many hops sit between model and browser, and what each adds.
3. Error recovery: what the client can show when the stream breaks mid-reply.
4. Tool-call handling: whether the path copes with a gap while a tool runs.
5. Infrastructure cost: whether streaming changes the per-request bill.

The landscape

1. ConverseStream + Lambda response streaming through a function URL + SSE to browser. The canonical serverless path where there’s no gateway to start from. The function URL is created with its invoke mode set to RESPONSE_STREAM, the handler is wrapped in awslambda.streamifyResponse, and each Bedrock event becomes an SSE line on the way out. Browser consumes with EventSource or fetch + a reader. CloudFront in front gives a custom domain, WAF, and origin access control so the URL isn’t reachable directly.

2. ConverseStream + Lambda behind an API Gateway REST API with the integration’s response transfer mode set to STREAM. The shape the stack already has, with one setting changed. The mode defaults to BUFFERED, which collects the whole integration response before answering, and that is why the browser still waits for the last token today. Set it to STREAM and API Gateway invokes the function through InvokeWithResponseStream and forwards bytes as they arrive. The setting only applies to proxy integrations, AWS_PROXY or HTTP_PROXY, and only on REST APIs. HTTP APIs have no equivalent and still buffer, so a chat route on one has to move.

3. ConverseStream + API Gateway WebSocket API. WebSocket connection established per session; Lambda pushes events to the connection via @connections API. Bidirectional but more complex: connection lifecycle management, message routing, and a bill metered per message and per connection minute. Worth it when the browser needs to push structured mid-conversation messages (cancel current generation, switch tool results) or when many-client broadcasts are involved.

4. ConverseStream + a long-running Fargate service with SSE. No Lambda cold starts (relevant when cold is ~300ms and time-to-first-token is ~500ms), no Lambda max duration limits. Higher fixed cost but predictable latency. Correct for high-volume services where cold-start variance matters.

5. Non-streaming Converse with a “chunked delivery” illusion. The naive workaround: generate the full response non-streaming, then dribble it to the browser one word at a time to simulate typing. Looks like streaming, isn’t. Still has the multi-second wait for the full generation before the first visible byte; abandonment metric doesn’t improve. Not a real option.

6. ConverseStream + AppSync Events. An Event API carries model output to connected clients over channel subscriptions on a managed WebSocket, with no GraphQL schema to write. Adds a second real-time surface but integrates cleanly with AppSync-backed front-ends. Correct for AppSync shops; overkill otherwise.

Evaluation

Side by side

For a chat interface on a Lambda-centric stack with one-way streaming (server → browser) and no broadcast requirements, SSE over the REST API already in the path is the correct shape, because it is a setting on the integration the chat route already uses. Usage plans, API keys, per-caller throttling, WAF and the custom domain all stay where they are. A function URL reaches the same place with one hop fewer and is the right answer for a greenfield route with no gateway, but it means a second front door and moving per-caller throttling into the function or onto WAF rate rules at CloudFront. An HTTP API has neither option, so a chat route on one moves to a REST API, a function URL, or a WebSocket API with its connection lifecycle to manage.

The streaming path, end to end

The solution

Lambda response streaming, and the runtime choice comes first. Lambda supports response streaming natively on the Node.js managed runtimes; every other language, Python included, needs a custom runtime with the streaming Runtime API integration or the Lambda Web Adapter in front of an ASGI app. On Node the handler is wrapped in awslambda.streamifyResponse, receives a writable stream, and passes it through awslambda.HttpResponseStream.from with the status code and headers. That helper emits the metadata JSON and the delimiter the front door expects, so the handler only writes payload bytes after it. Each SSE event is a data: line terminated by a blank line, which is what tells the browser’s parser the event is complete.

import { BedrockRuntimeClient, ConverseStreamCommand }
  from "@aws-sdk/client-bedrock-runtime";

const bedrock = new BedrockRuntimeClient({});

export const handler = awslambda.streamifyResponse(async (event, stream) => {
  stream = awslambda.HttpResponseStream.from(stream, {
    statusCode: 200,
    headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" },
  });
  const send = (obj) => stream.write("data: " + JSON.stringify(obj) + "\n\n");

  try {
    const res = await bedrock.send(new ConverseStreamCommand({
      modelId: MODEL_ID,
      messages: buildMessages(event),
      system: SYSTEM_PROMPT,
      inferenceConfig: { maxTokens: 800, temperature: 0.2 },
    }));

    for await (const item of res.stream) {
      if (item.contentBlockDelta?.delta?.text) {
        send({ type: "text_delta", text: item.contentBlockDelta.delta.text });
      } else if (item.messageStop) {
        send({ type: "stop", reason: item.messageStop.stopReason });
      } else if (item.throttlingException || item.modelStreamErrorException) {
        send({ type: "error", name: "stream_fault" });
        break;
      }
    }
    stream.write("data: [DONE]\n\n");
  } catch (err) {
    send({ type: "error", name: err.name });
  } finally {
    stream.end();
    await stream.finished();
  }
});

The front door. The chat route stays on the API Gateway REST API it already has, with the integration’s response transfer mode changed from BUFFERED to STREAM. API Gateway then invokes the function through InvokeWithResponseStream against the /response-streaming-invocations form of the invoke API rather than plain Invoke, which is why the function’s output is framed as metadata JSON, then an eight-null-byte delimiter, then the payload bytes. The delimiter has to appear within the first 16 KB of stream data, and output that does not match the format gets a 500 back to the client. Streaming also lifts the 10 MB response ceiling and the 29-second integration timeout, neither of which a chat reply reaches. What stops working on that route is anything that needs the whole body in hand: endpoint caching, content encoding, VTL response transformation. None of them belong on a per-user generation.

The same mechanism without the gateway is a Lambda function URL with its InvokeMode set to RESPONSE_STREAM (the default is BUFFERED), which is the shape to reach for when there is no REST API to start from. CORS is then configured on the function URL’s own CORS block rather than on a method response, and CloudFront in front gives the custom domain, WAF, and origin access control so nobody reaches the URL directly. Origin access control on a function URL adds two requirements: the URL has to use AWS_IAM auth, and because Lambda rejects unsigned payloads, every POST from the client has to carry a SHA256 of the body in an x-amz-content-sha256 header. A browser chat client is doing that hashing on every turn.

Timeouts belong to the function more than the gateway: the stream lives as long as the function runs, up to a maximum of fifteen minutes, which is also as long as API Gateway will hold a stream open. The one to check before shipping is the idle timeout on a quiet stream, which is five minutes on a regional or private endpoint and 30 seconds on an edge-optimized one, and 30 seconds again at CloudFront’s default origin response timeout when a distribution is in the path. That figure, whichever applies, sets the real budget for a pause between bytes, tool dispatch included. The CloudFront one is adjustable; the edge-optimized one is not.

Browser consumption. EventSource is the easiest path when a GET works, and it only does GET, so a chat turn that posts a message body uses fetch with response.body.getReader() instead. The client assembles the streamed tokens into the visible message as they arrive, shows a typing indicator between bytes, and handles the [DONE] sentinel and error events. The current assistant message stays marked partial until the stop event; on error, show what arrived plus a “(generation interrupted)” note.

Tool-call handling. ConverseStream emits a contentBlockStart carrying a toolUse block, then deltas with the partial input JSON, and the stream then ends with a message stop whose reason is tool use. The Bedrock stream is finished at that point. The handler runs the tool (another API call, possibly seconds), appends the assistant message and a user message carrying the toolResult block, and calls ConverseStream again to continue generation. The SSE connection to the browser stays open across both Bedrock calls, so from the client’s side one stream pauses and resumes. It sees a tool_start event (“Looking up your subscription…”), then a gap, then text again, with a “thinking” placeholder covering the gap.

Error handling. Four failure classes. First-byte timeout: the client aborts after 3 seconds of nothing and shows “The assistant is thinking…”; CloudWatch gets a metric. Mid-stream fault: Bedrock defines throttling, service-unavailable, validation and model-stream errors as members of the response stream itself, so the loop checks for them alongside the text deltas, while the surrounding try/catch covers a fault raised before the first event. Either way the handler emits an error event over the SSE connection, closes it cleanly, and the client shows the partial response with a reason. Clean but abbreviated: the stop reason says the token cap was hit or a guardrail intervened, and the client shows a “(response truncated)” affordance. Client disconnect: Lambda does not stop when the client goes away, so the function runs to completion or its timeout and bills for the full duration, and the Bedrock call is charged for tokens produced.

Cost shape. Bedrock charges the same per token either way. Lambda duration is the line that moves, because the function is alive for the whole generation rather than returning as soon as the reply lands. API Gateway meters a streamed response in 10 MB increments rounded up, so a chat reply stays a single billable request and only data transfer is charged on top. A function URL carries no charge of its own and moves that line to CloudFront requests and data transfer instead. Net neutral to slightly higher, and the increase is Lambda duration.

Worked example

Same support-assistant query, 500-token response. Measurements from before and after the streaming rollout:

Baseline (non-streaming)
  Time to first byte:   4,200 ms (full generation)
  Total response time:  4,200 ms
  User-perceived wait:  4,200 ms

Streaming (ConverseStream + SSE)
  Time to first byte:     800 ms (model producing)
  Total response time:  4,400 ms (slightly slower total)
  User-perceived wait:    800 ms

Total time is slightly worse with streaming, since the function stays alive for the whole generation and the stream takes a moment to set up. The wait before anything appears falls from 4.2s to 0.8s. Tokens keep arriving at roughly 140 a second after the first byte, so the typing animation runs at the pace the model generates.

Abandonment during the wait drops from 11% to 2.3% over the two weeks after rollout. Generation is no faster; the user just stops staring at a spinner.

What’s worth remembering

1. Streaming shortens the wait before anything appears, not the generation. Elapsed time is slightly longer; what the user watches is entirely different.
2. ConverseStream is the Bedrock half; the rest is transport. Lambda response streaming plus SSE to the browser is the AWS-native path, reached either by setting a REST API integration’s response transfer mode to STREAM or by a function URL with InvokeMode: RESPONSE_STREAM; anything left on the default buffers the whole response no matter what the model emits. Response streaming is a REST API feature, so an HTTP API route has to move.
3. SSE suits one-way streaming better than WebSocket: simpler wire protocol, native browser support, no connection-lifecycle code. WebSocket is the right choice on bidirectional traffic.
4. Error handling changes shape in streams. Three timeouts (first-byte, between-byte, total), Bedrock faults arriving as events inside the stream rather than as raised errors, and a partial response to display. None of this exists in request-reply.
5. Time to first byte is the metric to instrument, because it is the number the user experiences as responsiveness.

The same assistant, the same model, the same promptPromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot., the same tokens. The user sees typing instead of waiting, and the abandonment number moves.