Making a Pipeline Fast Enough to Use

The situation

A retail company’s main service has a pipeline taking 47 minutes from commit to production-ready. Twelve engineers work on it, and the effects are visible in the working practice: pull requests accumulate several commits before anyone pushes, the branch pipeline is routinely skipped by pushing directly to a shared integration branch, and deployments happen twice a week rather than on merge.

The stages, measured: source and dependency install 9 minutes, unit tests 6, container build 11, security scan 5, deploy to an ephemeral environment 7, integration tests 8, and a final packaging step 1.

Two constraints. The security scan is mandated and cannot be removed. And the integration tests genuinely need a deployed environment, because they exercise the service through its load balancer against a real database.

What actually matters

The first thing that matters is which duration people actually experience. The 47 minutes is the full path, and an engineer waiting on a pull request may only need the first 15 of it. Splitting feedback into what gates the merge and what gates the deployment is often the largest perceived improvement, and it changes no stage’s duration at all.

The second is that stages run in sequence because somebody wrote them in sequence, not because they depend on each other. The security scan and the unit tests have no dependency on one another, and running them in parallel removes the shorter one from the total.

The third is that most of the install and build time is work that has been done before. A dependency set that has not changed does not need resolving again, and a container layer that has not changed does not need rebuilding. Caching is the largest single lever in most pipelines and it is frequently absent because it needs configuration rather than code.

The fourth is that the environment setup is often the longest stage and the least examined. Seven minutes to deploy an ephemeral environment on every commit, when most commits do not change infrastructure, is a candidate for reuse rather than recreation.

Underneath it, the behaviour change is the real cost. A pipeline people route around does not test anything, so the 47 minutes has already reduced safety rather than increased it.

What we’ll filter on

1. Does this stage gate the merge, or the deployment?
2. Does it depend on a previous stage’s output, or is it sequential by habit?
3. Is it repeating work whose inputs have not changed?
4. Does it need a fresh environment, or would a reused one do?
5. Can it run on a smaller or larger machine to better effect?
6. What does it cost to run it, against what it costs to wait for it?

The landscape

Splitting the pipeline by what it gates. Fast checks that gate the merge (lint, unit tests, security scan) and slower ones that gate the deployment (integration tests against a deployed environment). This changes the experienced wait without changing any stage, and it is usually the first move.

Parallel stages and actions. CodePipeline runs actions in the same stage in parallel by default when they have no shared input ordering. Moving independent stages into one parallel stage removes the shorter durations from the total rather than adding them.

CodeBuild caching. Local caching for Docker layers, source and custom paths, and S3 caching for shared artefacts across builds. Docker layer caching is what takes an 11-minute container build to two when only the application layer changed, and it needs the build spec to declare it.

CodeArtifact. A managed package repository, so dependency resolution pulls from a warm regional cache rather than from the public internet on every build. It also removes the failure mode where a public registry is slow or unavailable during a build.

CodeBuild compute choices. Larger instance types for build-bound stages, and Lambda compute for short stages where start-up dominates. Reserved capacity fleets remove provisioning time for pipelines that run constantly. The right answer differs per stage, and using one compute type for the whole pipeline leaves time on the table in both directions.

Test splitting and parallelisation. Sharding a test suite across parallel build jobs, with results aggregated. It turns a linear duration into a divided one at the cost of running more compute concurrently.

Reusing an environment rather than recreating it. A long-lived integration environment that is updated rather than rebuilt, or a pool of pre-warmed environments claimed per run. It removes the provisioning time and reintroduces the state-contamination problem that ephemeral environments solve, so it is a trade rather than a free win.

Selective execution. Running only the stages affected by a change: a documentation-only commit does not need integration tests, and a change to one service in a monorepo does not need every service’s suite. Determined by path filters or a dependency graph.

Scan placement. Running the security scan against the built image once rather than at multiple points, and running it in parallel with tests rather than after them.

Evaluation

Side by side

Change	Saves	Risk	Effort	Changes the total or the wait
Split merge gate from deploy gate	~25 min of perceived wait	Later feedback on integration	Low	The wait
Parallelise scan with unit tests	5 min	None	Low	The total
Docker layer caching	~9 min	Stale layers if misconfigured	Low	The total
CodeArtifact for dependencies	~6 min	None	Medium	The total
Right-size compute per stage	~3 min	Cost	Low	The total
Shard the integration tests	~4 min	Flaky-test amplification	Medium	The total
Reuse the integration environment	~7 min	State contamination	Medium	The total
Selective execution by path	Varies, large on doc changes	Missing a real dependency	Medium	The total

The first row is the largest number on the page and it makes nothing faster. That is worth sitting with: most of the complaint is about the wait for feedback, and restructuring what gates what addresses it before any stage is optimised.

The solution

Split the pipeline by what it gates, parallelise the independent stages, cache the repeated work, and only then argue about environments.

Start by splitting. A merge-gating pipeline runs source, dependency install, unit tests and the security scan; a deployment pipeline runs the container build, environment deploy and integration tests after the merge. An engineer waiting on a pull request now waits about 15 minutes rather than 47, and the integration coverage still runs before anything reaches production. This is the change that stops people routing around the pipeline.

Parallelise within each half. The security scan does not depend on the unit tests, so both run in one stage and the five minutes disappears from the total rather than adding to it. Check the rest for the same shape; sequential pipelines usually contain two or three stages that are sequential only because of the order somebody wrote them in.

Then cache. Docker layer caching in CodeBuild takes the container build from eleven minutes to about two when only the application layer changed, which is most commits. CodeArtifact turns dependency install from a public-internet fetch into a regional cache read, and removes a class of flaky build failure at the same time. Between them these are the largest genuine reductions available.

Right-size the compute per stage rather than for the pipeline. The container build is CPU-bound and benefits from a larger instance; the one-minute packaging step is dominated by start-up and belongs on Lambda compute. Where the pipeline runs frequently enough, a reserved fleet removes provisioning time entirely.

Only then look at the environment. Seven minutes of provisioning on every run is the largest remaining stage, and the ephemeral-versus-reused argument is a genuine trade: reuse removes the time and reintroduces contamination between runs. A middle path is a pool of pre-warmed environments claimed and reset per run, which keeps isolation and removes the provisioning wait, at the cost of running the pool.

Add selective execution last, because it is the change most likely to be subtly wrong. Path filters that skip integration tests on documentation-only changes are safe; a dependency graph deciding which services a change affects is powerful and is exactly where a missed edge produces an untested deployment.

Then measure it as a metric rather than an anecdote. Pipeline duration, queue time and change failure rate on a dashboard is what tells you whether the next change helped, and it is what makes the case for the compute spend.

Why not run the biggest compute for everything. It helps the CPU-bound stages and does nothing for the ones dominated by network or provisioning time, and it costs on every run including the ones that were already fast.

Why not reuse a single long-lived integration environment. It is the fastest option and it is where flaky tests come from, because one run’s leftover state becomes the next run’s mystery failure. A pool with a reset is the version of this that survives contact with a real test suite.

Worked example

The split ships first and takes two days. The perceived wait drops from 47 minutes to 14, and within a fortnight the direct pushes to the integration branch stop, because the branch pipeline is no longer the expensive path. Nobody asked for that outcome and it was the actual problem.

Parallelising the scan with unit tests removes five minutes and takes an hour to implement.

Docker layer caching is the largest technical win and takes two attempts. The first configuration caches nothing useful, because the Dockerfile copies the whole source tree before installing dependencies, so every commit invalidates the dependency layer. Reordering the Dockerfile to copy the manifest, install, then copy the source is what makes the cache effective, and it is a change to the application repository rather than to the pipeline.

CodeArtifact takes dependency install from nine minutes to three, and removes two build failures a week that had been caused by a public registry timing out.

Right-sizing produces less than hoped: three minutes across the pipeline, mostly from moving the packaging step to Lambda compute.

The environment pool is deferred. With the split in place, the seven minutes sits in the post-merge pipeline where nobody is watching it, and the team decides the complexity is not worth it yet. That is a reasonable outcome and worth recording as a decision rather than an omission.

Six weeks on: merge feedback in 9 minutes, full path in 24, deployments daily rather than twice a week, and change failure rate down because smaller changes ship more often.

What’s worth remembering

1. A slow pipeline changes behaviour: people batch commits and route around it, so the cost is a reduction in safety rather than only in convenience.
2. Splitting what gates the merge from what gates the deployment is usually the largest improvement in experienced wait, and it makes no stage faster.
3. Stages are often sequential by habit rather than by dependency; moving independent ones into a parallel stage removes the shorter duration from the total.
4. Caching is the biggest genuine reduction in most pipelines, and Docker layer caching only works if the Dockerfile is ordered so dependency layers are not invalidated by every source change.
5. Right-size compute per stage rather than per pipeline, because CPU-bound builds and start-up-dominated short steps want opposite answers.
6. Reusing an integration environment removes provisioning time and reintroduces state contamination; a pool with a reset keeps isolation without the wait, and deferring the decision is legitimate once the wait has moved off the critical path.