Testing Infrastructure Code Before It Reaches an Account

The situation

An insurance platform deploys infrastructure through CloudFormation from a pipeline: commit, build, deploy to staging, deploy to production. The only validation is whether the deployment succeeds.

Over the last quarter that has produced four incidents. A security group opened to 0.0.0.0/0 by a copy-paste, deployed and live for six hours. A change that replaced an RDS instance rather than modifying it, discovered when the stack update took forty minutes and the database endpoint changed. A template that deployed cleanly to staging and failed in production because production has a resource limit staging does not. And an IAM policy with a wildcard that passed every check because there were no checks.

The team wants validation in the pipeline. The pushback from the same team is that they will not accept a pipeline that takes an hour, and they have seen enough tools that reject valid templates to be suspicious of the whole idea.

What actually matters

The first thing that matters is that the four incidents are four different failure classes and no single check catches them all. A linter catches malformed templates and some bad patterns; a policy engine catches the wildcard and the open security group; a change set catches the replacement; and only a real deployment catches the resource-limit difference. A pipeline that adds one check and declares the problem solved will keep having three of the four.

The second is that checks have wildly different costs and should sit where their cost is affordable. Syntax and policy checks take seconds and belong before the merge, where feedback is immediate and a failure costs nothing. Deploying into a real account takes minutes and belongs after the merge, on a smaller set of changes.

The third is the difference between predicting a change and observing one. A change set predicts what CloudFormation will do, including which resources get replaced, which is exactly what the RDS incident needed. It predicts from the template and the resource types, so it cannot tell you the deployment will hit a service quota.

The fourth is that a check that produces false positives will be disabled. The team’s suspicion is earned, and the way through is starting with rules that are unambiguously right, in a reporting mode, and promoting them to blocking once the noise is understood.

Underneath it, the environments differ, which is its own defect. Staging that does not represent production will keep producing the fourth incident regardless of what the pipeline checks.

What we’ll filter on

1. Which failure class does this check catch?
2. How long does it take, and where can that cost be afforded?
3. Does it predict, or does it observe?
4. What is its false-positive behaviour, and can it run in report-only mode?
5. Does it run before the merge, after it, or at deployment?
6. Can it be bypassed, and by whom?

The landscape

cfn-lint. Validates template syntax, resource properties, intrinsic function usage, and region-specific resource availability against the CloudFormation resource specification. It runs in seconds, locally and in a pre-commit hook, and it catches the class of error where the template is simply wrong. It says nothing about whether a valid template is a good idea.

CloudFormation Guard. A policy-as-code language for validating templates against rules: no security group with 0.0.0.0/0 on port 22, no IAM policy with Action: "*", every S3 bucket encrypted. Rules are written once and applied across every template. It runs in seconds and it is the check that catches two of the four incidents.

CloudFormation hooks. The same idea enforced by the service rather than the pipeline. A hook evaluates a stack operation before it proceeds and can fail it, which means it applies to deployments that did not come through the pipeline. Guard rules can be packaged as a hook, so the same policy runs in both places, and the hook is the one a determined engineer cannot skip.

Change sets. A prediction of what an update will do, with a Replacement flag of True, False or Conditional per resource. This is the check that catches an update that would destroy a database, and it is the natural approval gate between build and deploy.

taskcat. Deploys a template into real accounts and regions, in parallel, reports pass or fail, and tears down. It is the only check on this list that observes rather than predicts, so it is the one that finds a service quota, a region where a resource type is unavailable, or an IAM permission the deploying role lacks. It costs minutes and real resources.

CDK assertions and snapshot tests. For teams working in CDK, unit tests over the synthesised template: assert a resource exists with given properties, or compare against a stored snapshot so an unintended change fails the test. They run in seconds and catch regressions in the construct’s own logic.

AWS Config and conformance packs. After deployment rather than before. Necessary as a backstop, and it is detection: the security group was open for however long it took the rule to evaluate and remediate.

Ephemeral test accounts. Vending an account per pipeline run, deploying into it, testing, and deleting it. It gives an accurate environment with none of the drift a long-lived staging account accumulates, at the cost of account vending machinery and slower runs.

Evaluation

Side by side

Check	Catches	Predicts or observes	Cost	Where it fits
cfn-lint	Malformed templates, bad properties	Predicts	Seconds	Pre-commit and PR
Guard rules	Policy violations (open SG, wildcard IAM)	Predicts	Seconds	PR
CDK assertions	Regressions in construct logic	Predicts	Seconds	PR
Change set review	Resource replacement	Predicts	~a minute	Before deploy
taskcat	Quotas, region gaps, permission gaps	Observes	Minutes	Post-merge
CloudFormation hooks	Policy violations, unbypassable	Predicts	Seconds	At deployment
Config + remediation	Anything that got through	Observes	Continuous	After deployment

Mapping the four incidents onto the table settles the design. The open security group and the wildcard IAM policy are Guard, in the pull request. The RDS replacement is a change set. The production-only resource limit is taskcat or an ephemeral account, and nothing cheaper would have found it. Four incidents, three different checks, which is why the “add a linter” answer would have caught one.

The solution

Fast checks before the merge, a change set as the approval gate, a real deployment after the merge, and hooks as the layer nobody can skip.

Start with cfn-lint and Guard in the pull request, because they cost seconds and cover two of the four incidents. Write the Guard rules for the things that are unambiguously wrong: no security group ingress from 0.0.0.0/0 except on 443 to resources tagged as public, no IAM policy with a wildcard action, encryption required on storage. Keep the initial rule set small and defensible, because the team’s suspicion about false positives is the main risk to adoption.

Run those rules in report-only mode for two sprints. The output is a list of violations in existing templates, which does two things: it fixes the backlog before the rules start blocking, and it demonstrates the false-positive rate rather than arguing about it. Promote to blocking once the list is empty and the team has seen the noise level.

Add the change set as an explicit approval gate between build and deploy. The pipeline creates the change set, renders the replacement column into the approval request, and waits. A human approves a change that replaces nothing in about ten seconds; a change showing Replacement: True on an RDS instance gets read properly. This is the cheapest possible fix for the second incident and it costs one minute of pipeline time.

Then add taskcat after the merge, deploying into a test account across the regions in use, running for a few minutes and tearing down. This is the only check that would have caught the resource-limit failure, and putting it after the merge rather than in the pull request is what keeps the fast feedback fast. Where the difference between staging and production is severe enough, upgrade this to a vended ephemeral account per run, which removes the drift a long-lived test account accumulates.

Package the Guard rules as CloudFormation hooks as well, so the same policy is enforced by the service on every stack operation regardless of origin. The pipeline check is for feedback; the hook is for enforcement, and it is the one that covers the deployment somebody makes from a laptop during an incident.

Keep Config with remediation underneath everything as the backstop, and treat a finding there as a signal that a preventive check is missing rather than as the working control.

Then fix the environment difference, because no pipeline check compensates for staging not representing production. The resource limit that caused the fourth incident should either exist in staging or be raised in production deliberately, and the general question of which differences are intentional is worth an afternoon.

Why not put everything in the pull request. taskcat deploying real resources across regions takes minutes and costs money on every push, which is how a pipeline becomes something people work around. Predictive checks are cheap enough for every commit; observational ones are not.

Why not rely on hooks alone and skip the pipeline checks. Hooks cannot be bypassed, which is their value, and they fail at deployment time, which is late feedback. The same rules in both places give fast feedback and an unbypassable floor.

Worked example

The report-only fortnight finds 61 Guard violations across 40 templates. Fifty-four are genuine, mostly wildcard IAM actions in roles written years ago. Seven are false positives from a rule about encryption that does not account for a resource type where encryption is implicit; the rule is narrowed, which is the process working.

Promoting to blocking is uneventful because the backlog was fixed first. The team’s suspicion, which was the main obstacle, is answered by data rather than by argument.

The change set gate catches its first replacement in week three: a change to an RDS parameter group that would have replaced the instance. The engineer had not known that property forced replacement, and the change set said Replacement: True in the approval request. Under the old pipeline that would have been the second incident again.

taskcat finds two things in its first month. A template using a resource type unavailable in one of the four regions, which staging never caught because staging is single-region. And a deployment role missing a permission that production’s role happened to have, which is a drift between two roles nobody had compared.

The hooks go in last and immediately block a deployment made from an engineer’s laptop during an incident, opening a security group to debug something. The engineer’s reaction is annoyance, then agreement, and the break-glass path added the following week is a role that pages when assumed rather than a hole in the hook.

Pipeline time goes from 11 minutes to 14, which is inside what the team said they would accept, because the expensive check runs after the merge rather than on every push.

What’s worth remembering

1. Different checks catch different failure classes: a linter catches malformed templates, Guard catches policy violations, a change set catches resource replacement, and only a real deployment catches a quota or a region gap.
2. Put the cheap predictive checks before the merge and the expensive observational ones after it, because a pipeline that takes an hour on every push is one people route around.
3. A change set’s Replacement flag is the specific answer to an update that would destroy a database, and rendering it into the approval request costs about a minute of pipeline time.
4. Run new policy rules in report-only mode first: it fixes the existing backlog before the rules block anything and it answers the false-positive objection with data.
5. CloudFormation hooks enforce the same rules at the service rather than in the pipeline, so they cover deployments that never went through it; pair them with the pipeline checks rather than choosing between them.
6. No pipeline check compensates for a staging environment that does not represent production, so the difference that caused the incident is itself a defect to fix.