How to Run a Two-Tier Offline CA

Two Raspberry Pis, two SD cards, and a safe give you a private CA with a worst case you can survive. Why this shape rather than a managed CA or a single-tier one comes down to two numbers: a managed private CA is USD$400 a month, and a single-tier CA turns any compromise into replacing the trust anchor in every client you own. Two tiers plus your own hardware avoids both, and costs discipline instead. That trade is the thing to be honest with yourself about before starting; the rest of this is how to hold up your end of it.

This is the first of three. Here we build the CA, which stands alone and is useful whether or not AWS is anywhere near it. Then turning its certificates into AWS credentials, and finally automating the parts you would otherwise have to remember.

What you need

Two Raspberry Pis (any model with an SD slot; a Zero is plenty), two SD cards, two safes or two separate locked drawers, and two people who will both be present at the ceremony. Total hardware cost is less than one month of a managed private CA.

The division of labour: the ceremony card creates the root key, signs exactly one thing, and lives in a safe. The operations card runs the issuing CA that does the daily signing. They never swap roles, and the ceremony card never touches a network: everything it needs is baked into the image before the card is written.

Burning the cards

Verify the image before you write it. This is the root of everything you are about to trust, so a checksum you did not check is a gap in the chain:

sha256sum -c raspios-lite-arm64.img.xz.sha256

The operations card is an ordinary small server, so set it up the ordinary way. Write the verified image, boot it, and on that first boot set the user, install what you need, and turn off what you do not:

xzcat raspios-lite-arm64.img.xz | sudo dd of=/dev/sdX bs=4M conv=fsync status=progress
# then, on the card:
sudo apt-get update && sudo apt-get install -y openssl
sudo systemctl disable --now ssh    # re-enable it key-only once the jump host exists

Treat it as production from there: unattended upgrades on, key-only SSH from one jump host, a dedicated pki user owning /pki and nothing else, and shell access logged. Outbound HTTPS only; nothing inbound, which on this card means a default-deny nftables ruleset rather than a firewall you configured on the network.

If you want the issuing key to be unstealable rather than merely well-guarded, put it on a hardware token. A YubiKey or Nitrokey in the Pi’s USB port, driven through pkcs11-provider, holds a key that signs on request and never leaves the device, so an attacker with root on this card can mint certificates while they have it and takes nothing with them when you notice. That is a real gain for about fifty dollars, and it is the one upgrade here worth making before the ceremony rather than after, because the issuing key has to be generated on the token.

Building the ceremony card

The ceremony card gets a different treatment, because a card that has booted with an address is a card you have to reason about. Install its software before it ever starts: build the image on your Mac, take the networking out while you are in there, and write a card whose first boot is the ceremony itself.

Do this in a Linux VM on the Mac (Lima, UTM, whatever you already run). On Apple Silicon that VM is arm64, so the Pi’s filesystem runs natively in a chroot and no emulation is involved; on an Intel Mac, install qemu-user-static and its binfmt handlers first and the same steps work.

xzcat raspios-lite-arm64.img.xz > ceremony.img
sudo losetup -Pf --show ceremony.img          # -> /dev/loop0
sudo mount /dev/loop0p2 /mnt
sudo mount /dev/loop0p1 /mnt/boot/firmware
for d in proc sys dev dev/pts; do sudo mount --bind /$d /mnt/$d; done
sudo cp /etc/resolv.conf /mnt/etc/resolv.conf
sudo chroot /mnt /bin/bash

Inside the chroot you have the VM’s network, which is the last one this filesystem will see. Install what the ceremony needs, then take out everything that could ever bring an interface up:

apt-get update && apt-get install -y openssl
apt-get purge -y wpasupplicant network-manager isc-dhcp-client openssh-server avahi-daemon
apt-get autoremove -y && apt-get clean
rm -rf /var/lib/apt/lists/* /etc/resolv.conf
systemctl mask systemd-networkd systemd-resolved
: > /etc/machine-id
exit

Then take away the drivers, so there is no interface to bring up even if something tries. Blacklisting covers the wired and wireless chips across the models you might use:

printf 'blacklist brcmfmac\nblacklist bcmgenet\nblacklist smsc95xx\nblacklist lan78xx\nblacklist btbcm\n' \
  | sudo tee /mnt/etc/modprobe.d/no-network.conf

Set the operator account here too, so the card does not run a first-boot wizard and never carries a default login. openssl passwd -6 prompts for the password and prints the hash:

echo "ceremony:$(openssl passwd -6)" | sudo tee /mnt/boot/firmware/userconf.txt

Then turn the radios off in firmware, which is a stronger thing than masking a service, and stronger still on a Pi Zero with no radios to begin with:

printf 'dtoverlay=disable-wifi\ndtoverlay=disable-bt\n' | sudo tee -a /mnt/boot/firmware/config.txt

Unmount, record what you built, and write it. The image checksum goes in the ceremony record alongside the key fingerprints, so the thing you hardened is provably the thing that signed:

sudo umount /mnt/dev/pts /mnt/dev /mnt/sys /mnt/proc /mnt/boot/firmware /mnt
sudo losetup -d /dev/loop0
sha256sum ceremony.img | tee ceremony-image.sha256
sudo dd if=ceremony.img of=/dev/sdX bs=4M conv=fsync status=progress
sudo dd if=/dev/sdX bs=4M count=$(( $(stat -c%s ceremony.img) / 4194304 )) | sha256sum

That last line reads the card back and hashes what actually landed on it. A bad write to an SD card often reports success, and a card that failed is better found now than in a room with two people and a safe key.

You are trusting the Mac that built the image, which you were already trusting to write the card. What you are no longer trusting is a boot with an address on it.

The ceremony script checks for a route before it will run, but do not let a script be your air gap. The gap is a card with nothing on it that can dial out, and a cable you never plug in.

The ceremony

Order matters: the operations card must produce its request first, because the ceremony signs it.

On the operations Pi:

export PKI_ORG=Ironworks
export PKI_CRL_BASE=https://pki.factory.internal
mkdir -p /pki/issuing/{certs,crl,newcerts,private}
chmod 700 /pki/issuing/private
touch /pki/issuing/index.txt
openssl rand -hex 8 > /pki/issuing/serial
echo 1000 > /pki/issuing/crlnumber

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
  -aes-256-cbc -out /pki/issuing/private/issuing.key
chmod 400 /pki/issuing/private/issuing.key
openssl req -config issuing.cnf -new -key /pki/issuing/private/issuing.key \
  -out /pki/issuing.csr

Copy only issuing.csr to a USB stick. The key stays on this card for its whole life.

On the ceremony Pi, air-gapped, both of you present, working from a script you wrote beforehand. Two checks come first, because either one ruins the certificates you are about to make with nothing appearing to be wrong at the time. A Pi has no battery-backed clock and this one has no NTP, so it boots believing whatever fake-hwclock last wrote down, and a wrong clock means a root certificate valid from the wrong day. And a key is only as good as the randomness under it, so confirm the hardware generator is there before generating anything:

sudo date -u -s '2026-09-17 09:00:00'        # from a phone, out loud, agreed by both of you
cat /sys/class/misc/hw_random/rng_current    # bcm2835-rng
mount -o ro,noexec,nosuid,nodev /dev/sda1 /media/usb

Then the keys:

export PKI_ORG=Ironworks PKI_DNS_SUFFIX=.factory.internal \
       PKI_CRL_BASE=https://pki.factory.internal

# Root key. The passphrase is split: you each type half, neither of you knows both.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
  -aes-256-cbc -out /pki/root/private/root.key
chmod 400 /pki/root/private/root.key

# Root certificate, 20 years.
openssl req -config root.cnf -key /pki/root/private/root.key \
  -new -x509 -days 7300 -sha384 -extensions v3_root_ca \
  -out /pki/root/certs/root.crt

# The one thing the root ever signs, plus its own revocation list.
openssl ca -config root.cnf -extensions v3_issuing_ca -days 1825 -notext \
  -in /media/usb/issuing.csr -out /pki/root/certs/issuing.crt
openssl ca -config root.cnf -gencrl -out /pki/root/crl/root.crl

The extensions on that issuing certificate are where the real constraints live:

[ v3_issuing_ca ]
basicConstraints = critical, CA:true, pathlen:0
keyUsage         = critical, keyCertSign, cRLSign
nameConstraints  = critical, permitted;DNS:.factory.internal

pathlen:0 means it cannot create further CAs. nameConstraints means it cannot issue for anything outside your namespace. Together they cap what a stolen issuing key is worth: an attacker holding it still cannot mint a sub-CA or a plausible certificate for someone else’s domain. Add copy_extensions = none in the issuing config so a request asking for CA:true gets an ordinary leaf certificate anyway.

Write the ceremony record on the card, with both fingerprints and both signatures. Then power off, label the card (ROOT CA G1 / ceremony / <date> / never connect to a network), seal it in a tamper-evident bag, and put it in the safe. Repeat onto a second card, verify it reads, and store that one in a different safe.

Carry back root.crt, issuing.crt, and root.crl. On the operations Pi, install and verify before trusting:

install -m 444 /media/usb/issuing.crt /pki/issuing/certs/issuing.crt
install -m 444 /media/usb/root.crt    /pki/issuing/certs/root.crt
openssl verify -CAfile /pki/issuing/certs/root.crt /pki/issuing/certs/issuing.crt
openssl x509 -in /pki/issuing/certs/issuing.crt -noout -text | grep -A2 "Name Constraints"
openssl x509 -in /pki/issuing/certs/root.crt -noout -dates

If the name constraints print nothing, stop: they are missing and the ceremony needs redoing. Check the dates on this card, which knows what day it is: a notBefore in 1970 or last February means the ceremony Pi’s clock was wrong, and every certificate under it inherits the mistake.

When the safe opens

On a calendar, and in four emergencies. Nothing else.

The 180-day opening is the important one, and it is not really about the CRL. It is the rehearsal: it proves the cards still read, the passphrase halves still exist, and the procedure still works, at a time when none of that is urgent. A recovery path nobody has walked is a hypothesis.

Issuing certificates

One profile, short lifetimes, and a roster the signer consults:

[ workload_cert ]
basicConstraints       = critical, CA:FALSE
keyUsage               = critical, digitalSignature
extendedKeyUsage       = clientAuth
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always
crlDistributionPoints  = URI:https://pki.factory.internal/issuing.crl

openssl ca -config issuing.cnf -extensions workload_cert -days 14 -notext \
  -in worker.csr -out worker.crt

Fourteen days sounds relentless until Part 3 automates renewal, at which point lifetime stops costing anyone labour and short becomes free. Short lifetimes are the fallback revocation: whatever your revocation pipeline misses, expiry eventually catches.

Enrolling machines

The rule everywhere: the private key is generated where it will be used and never travels. The CA sees a request, never a key.

Linux. Use the TPM if there is one; otherwise a file key readable only by its owner.

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
  -out /etc/pki-leaf/leaf.key
chmod 400 /etc/pki-leaf/leaf.key
openssl req -new -key /etc/pki-leaf/leaf.key \
  -subj "/O=Ironworks/CN=docs-worker-01.factory.internal" \
  -out /etc/pki-leaf/leaf.csr

Windows. Better than Linux by default, because the key can be born non-exportable inside the TPM and no key file ever exists:

; worker.inf
[NewRequest]
Subject = "CN=docs-worker-01.factory.internal, O=Ironworks"
KeyAlgorithm = ECDSA_P256
Exportable = FALSE
MachineKeySet = TRUE
ProviderName = "Microsoft Platform Crypto Provider"
KeyUsage = 0x80
RequestType = PKCS10

certreq -new worker.inf worker.csr
certreq -accept worker.crt      # binds the returned cert to the TPM-held key

Drop back to Microsoft Software Key Storage Provider on machines with no TPM; keep Exportable = FALSE either way.

macOS. On Apple silicon, create the key in the Secure Enclave (via MDM or a small SecKeyCreateRandomKey helper with kSecAttrTokenIDSecureEnclave) so it is non-extractable by construction, and export only the request. Where that is not available, the Linux flow works unchanged with the key in an admin-only directory.

Revoking

openssl ca -config issuing.cnf -revoke /pki/issuing/newcerts/1042.pem \
  -crl_reason keyCompromise
openssl ca -config issuing.cnf -gencrl -out /pki/issuing/crl/issuing.crl
# publish to the distribution point in the certificates

Two things to internalise. Revocation only works where consumers actually read the list, which is why the distribution point in the certificate has to be somewhere reachable. And revoking is not the same as stopping: anything already authenticated stays authenticated until it next checks. Whether you can shorten that window depends on the consumer, and for AWS you can, which the next post comes back to.

When things go wrong

The question that sorts most of these: did it leak, or did it merely die? A stolen card is a compromise and starts a clock. A card that failed in the safe is an inconvenience, because the second copy exists for exactly this and because short leaf lifetimes bound the exposure to a fortnight either way.

Two rotations are planned rather than reactive. Renewing the issuing CA works by overlap: new key, root signs G2, both run side by side while G1’s outstanding leaves age out, and no trust store changes because clients trust the root. Replacing the root uses a cross-certificate: the old root signs the new root’s key so chains to G2 validate for clients that only trust G1, then G2 spreads through configuration management over months, verified by scanning trust stores rather than assumed, before the cross-certificate is withdrawn. Never cross-sign from a root you suspect is compromised; that carries the compromise forward into the replacement.