EventBridge Pipes vs Glue Lambdas

The situation

The platform team looks after twenty Lambdas of a shape that is embarrassingly repetitive. A representative one reads from an order-events SQS queue, checks whether event.detail.type == "OrderCancelled", pulls the customer ID and cancellation reason, reshapes the payload into the schema the cancellation-workflow Step Functions state machine expects, and calls StartExecution. Another matches OrderShipped, rewrites into a DynamoDB PutItem, writes to a shipment-ledger table. Another forwards high-value payment events above a threshold to an SNS topic. Another filters GuardDuty-style findings by severity, enriches each with an asset-owner lookup, and writes to a soar-input Kinesis stream.

They are all the same shape: source, filter, optional enrichment, target. The differences are the filter predicate, the shape of the transform, and the downstream. Multiply the forty-line template by twenty, add an IAM role per function, a pipeline stage per function, an error alarm per function, and a runtime version bump when Python 3.12 goes unsupported.

Three things bother the team: boilerplate volume (forty lines around three lines of actual logic), paying for non-events (a queue with a 2% match rate still invokes the Lambda on the other 98% of messages), and runtime lifecycle (twenty functions means twenty upgrades when a runtime hits end-of-life).

They want source-native connection to SQS, Kinesis, DynamoDB Streams and other queue-and-stream shapes; a content filter without code; optional enrichment without writing a full Lambda for the glue; target-native delivery to SNS, SQS, Step Functions, DynamoDB, Kinesis, EventBridge; pay-per-flowing-event billing; and minimal operational surface.

What actually matters

Before picking a service, it’s worth being explicit about what “glue” actually is, because the mental model decides what counts as overkill.

The core observation is that most of these Lambdas aren’t programs; they’re wiring diagrams expressed as code. Read from here, drop these, reshape those three fields, send to there. The handler is a faithful translation of a wiring diagram into Python, but the wiring diagram is the real artefact, and it would be simpler and cheaper if AWS operated the wiring directly. The interesting choice is where that wiring layer lives.

The next thing is cost shape under filtering. Per-invocation billing charges for every message the source delivers, whether the filter accepts it or not. For queues with low match rates, you’re paying almost entirely for traffic the filter throws away. What we want is a layer where filtering happens before the billable work begins, either in the service’s own declarative filter, or through a producer-side filter that never emits the discardable events at all. Either of those beats “filter in handler code.”

Operational surface is the third thing. A function runtime has a lifecycle: AWS deprecates runtimes on a schedule, every function has to move to a supported runtime by the deadline, and twenty functions is twenty upgrades. A managed service that owns the runtime itself takes that chore off the list entirely. Similarly: no partial-batch-response plumbing to write, no batch-handler scaffolding to maintain, no IAM role per function whose trust policy drifts from the template.

What an enrichment stage actually buys is worth sitting with. The shape “filter, then maybe look something up, then reshape, then send” is fundamentally a pipeline of steps. When we need a synchronous lookup (asset owner for a finding, customer tier for a transaction), the enrichment slot exists; when we don’t, we skip it. Having the pipeline shape be first-class means it’s visible in the console and the template, not buried in a handler.

Blast radius of a glue bug. When a handler has a bug, every message that reaches it is affected. When a managed wiring layer’s filter or template has a bug, the same is true, but the surface is smaller: no batch-handler logic, no error-retry branches, no custom backoff. The glue you don’t write is the glue that doesn’t break.

Finally, the twenty functions aren’t all the same problem. Some are genuinely code, parsing a free-text field, computing a hash, looking up idempotency keys before deciding which downstream to call. For those, a constrained wiring layer is too constrained and the function stays. The migration’s value depends on how many of the twenty are genuinely wiring; in our experience most of them are, and the handful that aren’t are the handful we want keeping a runtime.

What we’ll filter on

1. Source coverage, native reads from SQS, Kinesis, DynamoDB Streams, and similar queue/stream sources.
2. Target coverage, native delivery to SNS, SQS, Step Functions, DynamoDB, Kinesis, EventBridge, and the usual AWS destinations.
3. Declarative filter, pattern-matching predicate, no handler code.
4. Declarative enrichment, named integration point when a lookup is needed, not a full custom consumer.
5. Pay-per-flowing-event cost, billed scoped to events that pass the filter.
6. Low operational overhead, no runtimes, no version pinning, no batch plumbing.

The event-glue landscape

AWS gives four answers for “move events from one queue or stream into another AWS service, with a filter and maybe a small transform.”

Lambda as event glue. The status quo. Lambda polls the source, invokes a handler per batch; the handler does the filter, transform, and send. Source coverage is the widest. Target coverage is unlimited (any SDK). Filter-without-code exists in a limited form via event source mapping filter criteria, up to five pattern-based filters per mapping for SQS, Kinesis, DynamoDB Streams, MSK, and self-managed Kafka, and filtered-out records are not billed as invocations. Enrichment is whatever the handler does. Cost is per-invocation plus per GB-ms: if the filter lives in handler code, every batch is billed; if it lives in the event source mapping, filtered-out records are free but matching records still pay per invocation.

EventBridge Pipes. The managed point-to-point primitive for exactly this shape. Four stages: Source → Filter → Enrichment → Target. Source is one of SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MSK, Amazon MQ (ActiveMQ or RabbitMQ), or self-managed Apache Kafka. Filter is a content pattern identical to EventBridge rule patterns. Enrichment, when needed, invokes a Lambda function, a Step Functions Express state machine, an API destination, or an API Gateway. Target is one of around fifteen destinations. Billing is per event that passes the filter. No runtime, no handler.

Step Functions Express workflows. A workflow engine for short, high-volume executions, sub-second, at-least-once, billed per execution and per GB-ms. Triggerable from SQS, API Gateway, EventBridge. Powerful for logic that really is a workflow: parallel branches, choice states, error handling. Overqualified for “filter then reshape then send.”

Direct SDK integration in the producer. Skip the middle entirely: the producer writes directly to the destination from within its own code path. Two services, no middleware. Right when the producer owns the dispatch decision; wrong when the producer doesn’t know all the consumers, when fan-out is one-to-many across teams, or when the filter is a consumer-side concern.

Side by side

Matching the twenty Lambdas to four answers

The four-stage shape

A pipe is four stages. The middle two are optional; the outer two are mandatory.

Source stage. Pipes owns the polling loop. Batch size and batch window are configurable. For Kinesis and DynamoDB Streams, a shard iterator is managed for you. For MSK, Amazon MQ, and self-managed Kafka, consumer-group and authentication are part of the pipe definition. The source list is exactly six families, not an EventBridge event bus.

Filter stage. The pattern syntax is the EventBridge content-filter dialect, equality, prefix, suffix, numeric comparisons, anything-but, exists. Up to five filter patterns per pipe; any match lets the event through. Events that fail every pattern are dropped at the service level, never delivered to enrichment or target, and not counted as processed events for billing. This is the single biggest cost lever: a filter that drops 98% turns a 50,000-message-a-day queue into a 1,000-event-a-day bill.

Enrichment stage. Optional. Four options: Lambda, Step Functions Express (standard workflows are not valid because pipes invoke enrichment synchronously), API destination, API Gateway (REST or HTTP). The enrichment’s response replaces the event payload going into the target. Input and output templates (a subset of EventBridge’s input transformer syntax) can do static reshaping without enrichment at all.

Target stage. One target per pipe. An input template applies before delivery so the target receives exactly the JSON shape it expects. If the target is itself a fan-out service. EventBridge bus, SNS topic, Kinesis stream, the pipe becomes a well-shaped feeder into a downstream routing layer.

A worked trace: the OrderCancelled Lambda becomes a pipe

Take the first example Lambda. Source: order-events SQS. Filter: body.detail.type == "OrderCancelled". Transform: keep customerId, reason, cancelledAt. Target: cancellation-workflow Step Functions.

The pipe definition is roughly:

{
  "Name": "OrderCancelledToWorkflow",
  "RoleArn": "arn:aws:iam::111111111111:role/PipeRole",
  "Source": "arn:aws:sqs:eu-west-1:111111111111:order-events",
  "SourceParameters": {
    "SqsQueueParameters": {
      "BatchSize": 10,
      "MaximumBatchingWindowInSeconds": 1
    },
    "FilterCriteria": {
      "Filters": [
        { "Pattern": "{ \"body\": { \"detail\": { \"type\": [\"OrderCancelled\"] } } }" }
      ]
    }
  },
  "Target": "arn:aws:states:eu-west-1:111111111111:stateMachine:cancellation-workflow",
  "TargetParameters": {
    "InputTemplate": "{ \"customerId\": <$.body.detail.customerId>, \"reason\": <$.body.detail.reason>, \"cancelledAt\": <$.body.detail.timestamp> }",
    "StepFunctionsStateMachineParameters": {
      "InvocationType": "FIRE_AND_FORGET"
    }
  }
}

The forty-line Lambda, the batch handler, the partial-batch response, the retry, the deployment pipeline stage, the CloudWatch alarm, the IAM role, all replaced by a JSON document and one role that allows sqs:ReceiveMessage on the source and states:StartExecution on the target. Pipes handles the polling, the batching, the filtering, the transform, and the delivery.

The InputTemplate uses the Pipes input transformer syntax – <$.path.to.field> drops a JSON value from the source event into the output. It is a strict subset of EventBridge’s input transformer and intentionally limited: anything more expressive than path substitution and literals needs an enrichment step.

When the Lambda still earns its keep

Not all twenty will collapse to pipes. The triage rule is “is there code that makes the transform decision, or is the transform path-substitution plus a filter?”

Keep as Lambda when the transform is genuinely logic. Parsing a free-text field, computing a hash, looking up an idempotency key in DynamoDB before deciding which target to call, aggregating across batch items, retrying with custom backoff on a specific downstream error code, these are code, not a template.

Keep as Lambda when the target does not exist in Pipes. Most of AWS is covered, but if a function bridges into a non-AWS system with a bespoke authentication flow that API destinations cannot model, or into an on-premises service via PrivateLink with request signing only the SDK implements, the Lambda stays.

Keep as Lambda when the source filter must combine multiple queues. A pipe has one source. A Lambda with two event source mappings pulling from two queues and correlating across them is one function; expressing it as pipes would need two pipes feeding a third thing that correlates, which is no longer simpler.

Keep as Lambda when you already have heavy tooling built around Lambda. Replacing twenty well-tested functions with twenty pipes is a migration; it only pays back when the ongoing savings outweigh the one-off cost.

Of the twenty, the shape of the migration looks like: roughly thirteen drop straight into pipes, five keep a small enrichment Lambda inside a pipe, two stay as full Lambdas. The boilerplate disappears in eighteen cases out of twenty.

Step Functions Express and direct SDK, for completeness

Step Functions Express is correct when the shape is a workflow. A pipe targets one downstream per event; an Express workflow can branch, retry with step-specific backoff, run parallel calls, aggregate results. When a pipe with enrichment invokes a Step Functions Express state machine that then calls the real target, the pipe has become a trigger for a workflow, which is legitimate, but invoking the state machine directly from the queue via an event source mapping is often more direct.

Direct SDK in the producer wins where the producer owns the dispatch decision. If one producer writes to one downstream with a filter the producer itself applies, putting a pipe between them is extra indirection. Pipes earns its place when the producer emits to a shared queue that several consumers read independently.

What’s worth remembering

1. Pipes has four stages: Source, Filter, Enrichment, Target. Middle two are optional; outer two are mandatory. One source, one target per pipe.
2. Pipes sources are six families: SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MSK, Amazon MQ (ActiveMQ or RabbitMQ), and self-managed Apache Kafka. EventBridge event buses are not a pipe source.
3. Pipes targets cover about fifteen AWS destinations, including Lambda, Step Functions, EventBridge bus, SNS, SQS, Kinesis, Firehose, CloudWatch Logs, ECS task, Batch, API destination, API Gateway, SageMaker Pipeline, Redshift Data, and Inspector classic.
4. Filter syntax is the EventBridge event-pattern dialect, up to five patterns per pipe, any-match semantics. Filtered-out events are not billed.
5. Billing is per event that passes the filter, at a published per-million rate. Enrichment invocations cost on top.
6. Enrichment options are Lambda, Step Functions Express, API destination, and API Gateway. Standard Step Functions workflows are not a valid enrichment.
7. Input transformers at the enrichment and target stages do no-code reshaping via <$.path> substitution.
8. Lambda event source mappings also have filter criteria for SQS, Kinesis, DynamoDB Streams and Kafka, up to five patterns, filtered records not billed as invocations.
9. A pipe is simpler than a state machine; a state machine is simpler than a bespoke consumer. Use the lightest thing that fits.
10. Not every glue Lambda should become a pipe. Real transform logic, unusual targets, multi-source correlation, and existing well-tested Lambdas are legitimate reasons to leave a function alone.