CodeArtifact and Build Caching for CI

The situation

A platform team operates a monorepo with forty Node services, a handful of Python utilities, and a Go sidecar. Each service builds in CodeBuild, and the build logs read the same way every time:

• npm ci runs ~5 minutes per service, pulling ~300 packages from registry.npmjs.org.
• Python services add another ~1 minute for pip install from PyPI.
• The Go sidecar is fast; it’s the Node and Python tail that dominates.
• Total CI time per service: ~8 minutes, most of it waiting for tarballs.

Two problems surface in the same review:

• Security: internal packages (@acme/auth, @acme/ui, etc.) publish to the public npm registry, visible to anyone. One of them (@acme/auth@2.0) had its version number spoofed on npm for 48 hours before it was taken down. There is no private registry boundary today.
• Cost: ~3,000 builds per week × 8 minutes × ~$0.005/minute on a Linux Small fleet = about $600 a week of wall clock that is mostly npm install.

The team wants:

• A private package registry that internal packages publish to and that CI resolves from.
• A transparent upstream to the public registry so builds don’t maintain two separate resolve-paths.
• A cache that makes repeated dependency installs fast, without mixing build outputs between projects.
• Credential management that doesn’t park an npm token in an environment variable for a year.
• A path to pause pulls from the public registry in an incident without stopping builds entirely.

What actually matters

The two problems, private packages and slow installs, look separate but share a shape. Both want traffic to dependencies to flow through something the platform owns, rather than out to the internet.

For the package side, the question is: where does npm install @acme/auth resolve? Today, it resolves to the public registry, which means anyone can see the package exists and anyone who gets hold of the publish token can push a new version. A private registry ties publication to AWS IAM. The interesting part is how the private registry handles the 95% of packages that aren’t internal: does it mirror the public registry, proxy it, or make builds configure two registries? The pattern that scales is the upstream chain, one logical endpoint the build resolves against, which falls through to a proxy of the public registry when the package isn’t internal. The build sees one URL; the chain handles the rest.

For the cache side, the question is: what parts of a build are idempotent enough to reuse? Downloaded package tarballs are an obvious candidate, a versioned tarball is byte-identical every time. Node’s ~/.npm directory, Python’s ~/.cache/pip, Go’s module cache: all of these are downloaded-and-verified content that would be wasteful to fetch again. The cache options trade host persistence for restore latency, with another mode for Docker layers in containerised builds. Picking the wrong mode is the difference between a 30-second install and an 8-minute install that “has a cache configured.”

The third thing worth thinking about is how caching and the private registry compose. If the private registry is the package source and the builder caches ~/.npm, the first build of a project downloads tarballs from the registry (which in turn fetched them from the public mirror); subsequent builds pull from the local cache, never leaving the build host. The cache plus the proxy is where the wall clock comes out of. A cache with no private registry still leaks internal package names to the outside; a private registry with no cache still pays the network round-trip on every build.

What we’ll filter on

The filters the picks have to clear:

1. Private publication, internal packages never touch the public registry.
2. Transparent public fallback, builds don’t have to configure two registry URLs.
3. Cache scope, caching doesn’t leak build outputs between projects.
4. Credential lifetime, no long-lived tokens sitting in environment variables.
5. Operational control, the platform team can pause or audit pulls from the public registry.

The package-and-cache landscape

1. Public registry + CodeBuild local caches. Status quo plus a cache mode. Fixes some of the install time; does nothing for the private-package problem. Rejected.

2. Self-hosted Verdaccio on EC2 + S3 cache. A self-hosted npm proxy that can publish private packages and mirror the public registry. Works; introduces a piece of long-running infrastructure the team doesn’t otherwise run, plus certificate management, plus scaling. Reasonable for teams with existing Verdaccio expertise; reaches for less-managed territory than the scenario rewards.

3. CodeArtifact single repository. A CodeArtifact repository that holds internal packages and proxies the public registry in one object. Works for small setups; doesn’t scale cleanly to multiple teams sharing a central proxy because every repository then duplicates the proxy configuration.

4. CodeArtifact with upstream chain (domain → proxy → upstream). Two repositories in a CodeArtifact domain: internal holds @acme/* packages, with upstream = public-proxy; public-proxy has external-connection = public:npmjs and fetches upstream packages from npmjs on demand, caching them in the domain. Builds configure one registry URL (internal); misses fall through automatically. This is the canonical CodeArtifact pattern.

5. CodeBuild caches, local (source, Docker layer, custom). LOCAL cache stores artefacts on the build-host disk between consecutive runs. Free; best-effort. CodeBuild reuses hosts when it can but evicts them under pressure. CUSTOM sub-mode lets the build enumerate directories to cache; pairs well with ~/.npm, ~/.cache/pip, /go/pkg/mod.

6. CodeBuild caches. S3. S3 mode uploads and downloads a specified cache path to an S3 bucket between runs. Persistent across build hosts; adds round-trip latency versus local disk but survives host turnover. Useful for large caches that local disk would evict, and for reserved-capacity fleets where caches otherwise get destroyed with the environment.

7. Fleet-level ephemeral caches. CodeBuild reserved capacity fleets (FleetType=RESERVED) can hold caches in the fleet’s own storage, bridging local-disk speed with S3-like persistence. Useful once the build volume is high enough to justify reserved capacity.

Side by side

The combined pick is CodeArtifact with an upstream chain and a CodeBuild cache. CUSTOM local for single-service builds, S3 for the monorepo’s shared node_modules layers.

The upstream chain and the cache

The picks in depth

CodeArtifact: domain, internal repository, proxy repository. One domain (acme) in one account. Two repositories under it: internal for published @acme/* packages, and public-proxy for everything else. internal.upstream = [public-proxy]; public-proxy.externalConnections = [public:npmjs, public:pypi]. The domain is the unit that holds the actual package storage (tarballs are stored once per domain, even if multiple repositories reference the same version) and the KMS key that encrypts them.

Builds authenticate with aws codeartifact login --tool npm --repository internal --domain acme. The CLI generates a 12-hour authorisation token, writes an .npmrc that points registry=https://acme-111122223333.d.codeartifact.<region>.amazonaws.com/npm/internal/, and puts the token in the //…:_authToken line. No long-lived secret sits in the environment; the CLI re-runs at the start of each build.

When npm ci fetches @acme/auth, the request hits internal, which has the package, and returns it, no external call. When it fetches lodash, internal doesn’t have it, falls through to public-proxy, which looks in its cache; on a miss, public-proxy fetches from npmjs, stores the tarball in the domain, and returns it. Next build (any project, in the domain): public-proxy has it; no external call.

Package-origin controls. CodeArtifact lets the team set per-package-name origin controls: for packages inside @acme/*, set publish=ALLOW, upstream=BLOCK. The effect is that even if someone on the public registry tries to register @acme/auth@99.0.0, the internal repository refuses to pull it through the upstream chain. The team publishes @acme/auth; no version that did not come from that publish path ever resolves locally. This is the answer to the @acme/auth@2.0 spoof from last year, origin controls make the attack impossible, not just slower.

CodeBuild CUSTOM local cache. The first line of defence for per-project builds. In the project’s cache block: type: LOCAL, modes: [LOCAL_CUSTOM_CACHE]. In the buildspec: cache.paths: [~/.npm/**/*, ~/.cache/pip/**/*, /go/pkg/mod/**/*]. CodeBuild tries to reuse the same host for the next run of the same project; if it does, those paths are already on disk. No S3 round-trip, no cache-miss penalty beyond the first build.

CodeBuild S3 cache. The durable cache for the monorepo. type: S3, location: s3://ci-cache/monorepo/. Same paths listed in cache.paths. Between runs CodeBuild tars the paths, encrypts them with the project’s KMS key, and uploads to the bucket. The next run downloads and extracts before the build phase. Slower than local-disk reuse on a cache hit but survives host turnover; useful for the monorepo’s shared layers because the local-disk cache gets evicted under load.

Docker layer caching. For projects that build container images, modes: [LOCAL_DOCKER_LAYER_CACHE] lets CodeBuild reuse Docker layer storage between runs. The FROM node:20 and npm ci layers land once and then reuse. Requires privileged mode on the build environment. Combines with the CUSTOM cache when the Docker build itself pulls private packages from CodeArtifact.

A worked build with both in play

Service web-app builds. First run with the new setup:

version: 0.2
phases:
  pre_build:
    commands:
      - aws codeartifact login --tool npm --repository internal --domain acme
      - aws codeartifact login --tool pip --repository internal --domain acme
  build:
    commands:
      - npm ci
      - npm test
      - npm run build
cache:
  paths:
    - '/root/.npm/**/*'
    - '/root/.cache/pip/**/*'

• Build role has codeartifact:GetAuthorizationToken on the domain, codeartifact:ReadFromRepository on internal, and sts:GetServiceBearerToken for the token-generation flow.
• npm ci reads .npmrc written by codeartifact login, resolves every package against internal. The first build’s tarballs flow: request → internal (miss) → public-proxy (miss) → npmjs → cached in public-proxy → returned to build. @acme/auth resolves directly from internal and never touches the upstream chain. Total time ~5 minutes.
• On completion, CodeBuild uploads ~/.npm and ~/.cache/pip (compressed, KMS-encrypted) to S3.

Second run, one commit later:

• npm ci starts, S3 cache restored before pre_build. ~/.npm is pre-warmed; npm resolves packages from the local cache instead of network. The 5-minute install drops to ~40 seconds. Auth still happens via codeartifact login, fast, because the token is generated fresh per build.
• Total time ~2 minutes instead of 8.

Third run, on a different CodeBuild host: same story. S3 cache is host-independent.

Across 3,000 builds/week, the cost shape shifts: builds that averaged 8 minutes now average 2 minutes. The registry traffic to npmjs drops to near-zero because public-proxy serves cached copies of every package that’s been pulled in the domain’s history.

Pausing the public registry

An @acme/auth takeover attempt in year two. Security wants the public registry disconnected for an hour while the incident is investigated. With the upstream chain, this is a single API call:

aws codeartifact disassociate-external-connection \
  --domain acme --repository public-proxy \
  --external-connection public:npmjs

Effect: any package already cached in public-proxy continues to resolve. Any package not in the cache fails the build. Internal packages continue to resolve unaffected. The team investigates, re-associates the connection, and the chain resumes. Without the upstream abstraction, pausing would mean rewriting every project’s registry configuration, twice.

What’s worth remembering

1. CodeArtifact’s upstream chain is the point. An internal repository with upstream = public-proxy and a public-proxy with an external connection to npmjs gives one registry URL to builds, caches public packages in the domain, and separates internal from external origin.
2. Package-origin controls stop name-squatting. Setting upstream=BLOCK on @acme/* packages means the upstream chain never pulls that scope, regardless of what appears on the public registry. This is the durable fix for dependency-confusion attacks.
3. Authentication is a 12-hour IAM token, not a stored secret. aws codeartifact login generates it at build start; no long-lived npm publish token sits in an environment variable.
4. CodeBuild has three cache shapes. LOCAL (best-effort host reuse, sub-modes for source/Docker/custom), S3 (durable, host-independent), and reserved fleet caches. Pick based on cache size, run frequency, and whether reserved capacity is in play.
5. LOCAL_CUSTOM_CACHE + cache.paths is the common answer for dependency caching. Enumerate ~/.npm, ~/.cache/pip, /go/pkg/mod, ~/.m2; CodeBuild preserves those paths between runs on the same host.
6. S3 cache adds a minute of round-trip on miss but survives host turnover. For projects that run hundreds of builds a day the miss rate is low; the durability is the reason.
7. Domain-level storage dedupes tarballs. A package version stored in the domain is shared across every repository that references it; the same react@18.2.0 doesn’t get stored twice.
8. DisassociateExternalConnection is the incident lever. One API call pauses pulls from the public registry without breaking internal resolution or already-cached public packages; re-associate to resume.

The monorepo’s CI gets faster because the cache stops re-downloading, and more private because CodeArtifact stops leaking package names to the public registry. Both are the same change, traffic to dependencies flows through something the platform owns, and the tools that deliver them compose cleanly. Private packages and hot caches are the same story once the upstream chain is in place.