How to Warm Up and Drain Instances With Lifecycle Hooks

The situation

A SaaS platform runs a fleet of EC2 instances behind an Application Load Balancer. An Auto Scaling group watches CPUUtilization; above 70% for three minutes a scale-out fires, below 30% for fifteen minutes a scale-in fires. The instances run a Java application with a 400 MB heap, a local LRU cache populated via read-through to DynamoDB, and a Consul sidecar for service discovery among a constellation of internal workers.

On the way in: a new instance boots, user-data completes, the ASG moves it to InService, and the ALB’s TCP health check on port 8080 declares it healthy within twenty seconds of the port opening. The ALB starts routing immediately, and for two to three minutes afterwards requests landing on this instance see elevated latency and a scattering of HTTP 502 and HTTP 504 errors. The JVM is still doing its first major GC; the local cache is cold; Consul hasn’t finished the registration handshake. The instance is technically up but functionally not ready.

On the way out: scale-in picks a victim and EC2 sends the shutdown signal within seconds. Whatever was in-flight, half-written rows, a CSV export streaming to an S3 multipart upload, a sixty-second webhook fanout, is cut off. The in-memory SQS batch loses whatever it had not yet processed; those messages return after the visibility timeout, but partial work already done gets re-done and its downstream effects duplicated.

What actually matters

Before reaching for a feature, it’s worth asking what shape of mechanism could possibly fix both symptoms without inventing a parallel control plane.

The first thing worth thinking about is who owns readiness. The ALB’s target-group health check is the default gate on incoming traffic, but a TCP success on port 8080 is a very weak proxy for “this instance can serve a user well”. The application is the only thing that actually knows when its caches are warm, its first GC is done, and its service catalogue is populated. Any mechanism that wants to prevent the cold-start error window has to let the application tell the platform when it’s ready, rather than the platform guessing from the outside.

The second is the shape of the blast radius when things go wrong on the way out. A dropped HTTP request is recoverable, the client retries. A dropped SQS batch that was half-processed is not recoverable without work: the messages come back after the visibility timeout, but the side effects of the rows that did get written are now candidates for duplication. The drain mechanism needs to give the application enough time to either finish what it started or explicitly return unfinished work to the queue, and it needs to stop new work arriving while that’s happening.

The third is how long is long enough. Ninety seconds of warm-up is not an upper bound on a bad day, and a long export can take ten minutes to finish cleanly. A fixed timeout picked for the typical case will guillotine the atypical case. Whatever mechanism we land on has to accept “I need more time” from a running task without the task having to fail and restart. Otherwise the mechanism becomes the new failure mode.

The fourth is cold-start amplification. Even with a perfect pause on the way in, every scale-out still takes sixty to ninety seconds before the new capacity is live. If the scale-out was triggered by a genuine surge, those are precisely the seconds when the existing fleet is thinnest. Correctness is solved by the pause; latency during a surge is a separate concern that may want a complementary mechanism, pre-initialised capacity sitting to one side, ready to skip most of the boot path.

The fifth is operational overhead. The team already runs EC2 Auto Scaling. A mechanism that bolts onto the ASG declaratively, fires through standard AWS APIs, and doesn’t require a Lambda plus an SQS queue plus a DynamoDB table of in-flight instance state is strictly preferable to one that does. “No new control plane” is worth a lot when the team is small.

And finally, a softer one: symmetry. The two symptoms are mirror images. Cold-start is “the platform thinks I’m ready before I am”; drain is “the platform stops me before I’m done”. A single primitive that handles both halves is easier to understand, document, and on-call for than two unrelated mechanisms glued together.

What we’ll filter on

Distilling that exploration into filters we can score each Auto Scaling feature against:

1. Pre-traffic warmup, can it hold a new instance out of traffic until the application itself declares ready?
2. Graceful drain, can it hold a doomed instance long enough for the application to finish in-flight work?
3. Heartbeat extension, can a running task ask for more time without failing?
4. Cold-start complement, can it be paired with a mechanism that shortens the warm-up itself?
5. Low overhead, is it a native feature of EC2 Auto Scaling, configured declaratively, no new services?

The Auto Scaling lifecycle landscape

1. No lifecycle hooks (the status quo). The default ASG behaviour. An instance goes from Pending straight to InService the moment EC2 reports it running, and from Terminating to Terminated the moment the ASG decides to remove it. There is no intermediate state, no way for the application to signal readiness, no way to run final cleanup. The ALB target-group health check is the only gate, and a TCP success on port 8080 is a weak proxy for application readiness.

2. EC2 Auto Scaling lifecycle hooks. Named autoscaling:EC2_INSTANCE_LAUNCHING and autoscaling:EC2_INSTANCE_TERMINATING. A launching hook inserts a Pending:Wait state between Pending and InService; a terminating hook inserts a Terminating:Wait state between Terminating and Terminated. In the Wait state the instance is held (up to a configured timeout) while the application, or anything listening on the hook’s EventBridge or SNS notification, does its work. When done, the caller issues CompleteLifecycleAction with LifecycleActionResult=CONTINUE to move the instance along, or ABANDON to terminate and re-launch (launch side) or skip remaining hooks (terminate side). Per-heartbeat timeout is 30-7200 seconds (default 3600). The application can call RecordLifecycleActionHeartbeat to reset the countdown; total hold extends up to 48 hours. This is the primitive the symmetry argument points at.

3. Warm pools. A pool of pre-initialised instances sitting next to the ASG in Stopped (EBS only, cheapest), Running (instant but most expensive), or Hibernated (memory preserved, EBS-billed, sub-30-second wake on supported instance types). Scale-out pulls from the warm pool first, skipping most of the boot path. The launching lifecycle hook still fires, but its work is smaller because the JVM is already hot. Complementary to hooks, not a replacement.

4. Instance protection. A flag (ProtectedFromScaleIn=true) that marks individual instances ineligible for scale-in termination. Useful when one instance is hosting a long-running job, it just redirects termination elsewhere. Narrow tool; works with hooks, not a replacement.

5. ALB target-group deregistration delay. A target-group setting (deregistration_delay.timeout_seconds, default 300, maximum 3600) governing how long the ALB keeps sending existing connections to a draining target before forcibly closing them. New connections stop immediately on deregistration. This is the ALB half of graceful drain, it does not, by itself, give the application time to flush its SQS batch or finish a long export; it only governs the load balancer’s side. Pair it with a terminating lifecycle hook and the two mechanisms cover different halves: ALB stops sending new work, hook gives the app time to finish what it has.

Side by side

Reading the table by symptom rather than by feature: launching and terminating lifecycle hooks are the central mechanism for attributes 1, 2, and 3; ALB deregistration delay is the complement on the drain side, it stops the ALB sending new connections while the hook gives the app time to finish what it has; warm pools are the complement on the warm-up side for fleets where the ninety-second cold start is still too visible even behind a hook. No single feature ticks everything; the three together are the shape of the solution.

Mapping symptoms to mechanisms

Three things the diagram glosses worth spelling out:

• ABANDON has different meanings on each side. From Pending:Wait it terminates the instance and the ASG launches a replacement, useful if the readiness check decides this instance is permanently broken. From Terminating:Wait it short-circuits any remaining hooks and proceeds to termination, the instance was going there anyway.
• The Wait state is held by a timeout, not indefinitely. Default heartbeat timeout is 3600 seconds; configurable from 30 seconds up to 7200 seconds. Total hold time can be extended to 48 hours by calling RecordLifecycleActionHeartbeat before the current window elapses. If the window expires without a completion call, the hook’s DefaultResult fires – ABANDON or CONTINUE, set when the hook was defined.
• Only one launching and one terminating hook per ASG is typical. You can attach up to 50 per transition, but they fire in parallel, and the transition only proceeds when all of them complete (or their timeouts expire). Most teams use one of each.

Lifecycle hooks, in depth

The launch side solves the cold-start symptom. The launching hook is configured when the ASG is created (or added later via PutLifecycleHook). The typical shape:

• LifecycleHookName: warmup-ready-check
• LifecycleTransition: autoscaling:EC2_INSTANCE_LAUNCHING
• HeartbeatTimeout: 900 (fifteen minutes)
• DefaultResult: ABANDON (if the instance hasn’t reported ready in fifteen minutes, something is wrong, terminate and re-launch, don’t send traffic)
• NotificationTargetARN: optional SNS topic or EventBridge rule if you want external notification; not required if the instance itself handles the hook

When EC2 reports the instance running, the ASG transitions it to Pending:Wait. The user-data script (or a systemd unit, or an application startup hook) does the readiness work: wait for the JVM to finish its first major GC cycle; populate the LRU cache by issuing a bounded set of background reads against DynamoDB; register in Consul and wait for the health check to flip green; pull the current service catalogue so the first outbound RPC has a populated endpoint list.

Once the application’s /ready endpoint returns 200, the startup script signals completion:

aws autoscaling complete-lifecycle-action \
  --lifecycle-hook-name warmup-ready-check \
  --auto-scaling-group-name services-prod \
  --lifecycle-action-result CONTINUE \
  --instance-id $(ec2-metadata --instance-id | cut -d' ' -f2)

The instance moves from Pending:Wait through Pending:Proceed to InService. The ALB’s target-group health check observes it, declares it healthy, and traffic starts flowing, now against a warm JVM with a populated cache, a registered Consul node, and a current service catalogue. The 2-3 minute error window disappears. If the readiness work takes longer than expected, the startup script calls RecordLifecycleActionHeartbeat every few minutes to reset the 900-second countdown. If the instance is genuinely broken, fails to open port 8080, never returns 200 from /ready, the timeout expires and DefaultResult=ABANDON terminates the instance. The ASG launches a replacement. Traffic never lands on the failing instance.

The terminate side solves the drain symptom. The terminating hook is symmetric:

• LifecycleHookName: graceful-drain
• LifecycleTransition: autoscaling:EC2_INSTANCE_TERMINATING
• HeartbeatTimeout: 600 (ten minutes, long enough for the worst-case long-running export)
• DefaultResult: CONTINUE (if the hook somehow didn’t run, let termination proceed, better to lose some work than to pile up stuck instances costing money)

The ALB target group also has its deregistration delay set to a sensible value, 60 seconds for a fast-request service, longer for something with long-lived connections. The two mechanisms work together: scale-in fires; the ALB deregisters the target so new connections stop arriving while in-flight ones have deregistration_delay.timeout_seconds to finish; the ASG moves the instance to Terminating:Wait; a signal handler in the application (listening for SIGTERM, or for a hook notification delivered via EventBridge) stops accepting new work from the SQS consumer, lets in-flight HTTP requests complete, flushes the local SQS batch (either by finishing processing or by returning messages with a short visibility timeout), deregisters from Consul, and closes database connections cleanly. Once drain is complete, the wind-down script calls CompleteLifecycleAction with CONTINUE and the instance proceeds to Terminating:Proceed and termination.

Warm pools earn their cost when the cold start itself is too visible. Lifecycle hooks solve the correctness problem: no traffic hits a cold instance, no work is dropped by a doomed one. But every scale-out still takes 60-120 seconds of warm-up before the new capacity is available, and if the scale-out trigger was a genuine surge those are the seconds when the fleet is most thinly spread. Warm pools keep a reserve of pre-initialised instances alongside the ASG in a stopped (or hibernated, or running) state. When scale-out fires the ASG pulls from the warm pool first. The launching lifecycle hook still runs but the work inside it is smaller, the JVM is already hot, the cache may already be warm if the shutdown hook snapshotted it to EBS, the Consul sidecar is already on disk ready to start. Warm pools shrink that window from 90 seconds to perhaps 15. Reach for them when scale-out events are frequent enough and user-facing enough that the shortened warm-up justifies the EBS bill for the idle instances.

Gotchas worth naming. The heartbeat timeout is per-hook, not per-instance, if the same instance is held in a Wait state by two launching hooks, the transition doesn’t proceed until both complete. DefaultResult is asymmetric in practice: ABANDON on the launch side is usually right (a broken instance shouldn’t serve users), CONTINUE on the terminate side is usually right (a stuck drain shouldn’t pile up). Instances in Pending:Wait still count against the ASG’s MaxSize and still bill as running. And the EC2 StatusCheckFailed alarm will fire on instances that never open port 8080, which is something a broken user-data script will do just as readily as a broken application, wire the alarm to the same SNS topic the hook reports to, so both paths to “this instance is never coming online” show up in the same notification stream.

A worked example: the state machine for one instance

Every instance in the ASG moves through a fixed set of states. With no hooks configured the path is short; with both hooks configured, two extra Wait states appear, one on each side of InService, and those are where the application does its pre-traffic and pre-termination work.

What’s worth remembering

1. Lifecycle hooks insert Wait states. Pending:Wait on the launch side, Terminating:Wait on the terminate side. The instance is held in the Wait state until the hook completes (or its timeout fires).
2. Transitions are named autoscaling:EC2_INSTANCE_LAUNCHING and autoscaling:EC2_INSTANCE_TERMINATING. Up to 50 of each per ASG; all fire in parallel; the transition proceeds only when all complete.
3. CompleteLifecycleAction has two results. CONTINUE moves the instance forward. ABANDON terminates the instance on the launch side (triggering a re-launch) or skips remaining hooks on the terminate side.
4. Heartbeat timeout is per-hook. Default 3600 seconds; configurable 30-7200 seconds. RecordLifecycleActionHeartbeat resets the countdown; total hold time can extend to 48 hours.
5. DefaultResult is the fallback. ABANDON on the launch side is usually right; CONTINUE on the terminate side is usually right. Pick based on whether a stuck instance should be replaced or allowed through.
6. ALB deregistration delay is a separate mechanism. It governs how long the ALB lets existing connections drain. Pair it with the terminating hook; they solve different halves of drain.
7. Warm pools complement rather than replace hooks. They shorten the cold-start window; they don’t eliminate the need for a readiness gate.
8. Instance protection is orthogonal. It keeps a specific instance out of scale-in’s target list; it does not provide graceful termination. Use it for long-running jobs; use hooks for routine drain.