deploytype
Type the commands you would actually run. 10 good deploys, 10 firefights. Pick a drill and run the sequence verbatim, or take a timed mode against that tool's corpus.
20 drills
- good deploy★★★★★
Ship a feature branch
Branch, commit, push and open the pull request the way you were taught.
git · linux · 6 lines
- good deploy★★★★★
Build, tag and push an image
A reproducible image build with a commit-sha tag and a floating env tag.
docker · linux · 6 lines
- good deploy★★★★★
Apply a manifest and watch the rollout
Server-side apply, wait for the rollout, confirm the pods are actually ready.
kubectl · linux · 6 lines
- good deploy★★★★★
Plan then apply, never the other way round
Save the plan, read the plan, apply exactly the plan you read.
terraform · linux · 7 lines
- good deploy★★★★★
Preflight before you touch production
Migrations pending, disk headroom, error rate, on-call awake. Then deploy.
bash · linux · 6 lines
- good deploy★★★★★
Deploy a site to IIS
Stop the pool, swap the payload, warm it back up — with PowerShell doing the work.
powershell · windows · 6 lines
- good deploy★★★★★
Swap a Windows service binary
Old-school cmd.exe: stop the service, copy the build, start it, verify.
cmd · windows · 5 lines
- good deploy★★★★★
Promote a canary
Hop to the bastion, scale the canary up, watch the metrics, then promote it to stable.
kubectl · linux · 8 lines
- good deploy★★★★★
Cut a release tag
Annotated tag, signed, changelog generated from the range, pushed with the tag.
git · linux · 7 lines
- good deploy★★★★★
Blue/green with compose
Bring green up beside blue, health-check it, flip nginx, retire blue.
docker · linux · 8 lines
- firefight★★★★★
Undo the rollout
The new pods are crash-looping. Check history, roll back, confirm it settled.
kubectl · linux · 5 lines
- firefight★★★★★
Revert the commit that did it
Find the bad commit, revert it cleanly, push the hotfix, tag it so nobody re-lands it.
git · linux · 6 lines
- firefight★★★★★
Roll back a swarm service
The rolling update ate the service. Inspect it, roll back, watch the tasks come home.
docker · linux · 6 lines
- firefight★★★★★
Restore the database
The migration dropped a column with data in it. Hop to the DB box and restore.
bash · linux · 9 lines
- firefight★★★★★
The certificate expired at midnight
TLS is dead and the pager is screaming. Renew, reload, verify the chain.
bash · linux · 7 lines
- firefight★★★★★
Rescue the terraform state
An apply died halfway and left a stale lock plus a drifted resource. Untangle it.
terraform · linux · 8 lines
- firefight★★★★★
Roll back the Windows web tier
Hop from the Linux bastion onto the IIS box and put the previous build back.
powershell · windows · 7 lines
- firefight★★★★★
Emergency service restart
The worker wedged and the queue is backing up. Kill it, clear the lock, restart.
cmd · windows · 5 lines
- firefight★★★★★
OOMKilled at 03:00
Pods keep getting OOMKilled. Diagnose on the node, raise the limit, verify the node survives.
kubectl · linux · 9 lines
- firefight★★★★★
Disk full on the primary
No space left on device. Find the offender, reclaim safely, rotate the logs.
bash · linux · 7 lines
incidents
24 narrative firefights. A scene sets, the pager goes off, and each stage is a real typing run under escalating pressure — type your way from the first deploy to whatever it becomes.
24 incidents
- 5 stages★★★★★
the friday deploy
it is friday, 16:47.
git · gh · kubectl · psql · linux
- 4 stages★★★★★
the certificate expired at midnight, and so did its friends
the wildcard cert was set to auto-renew. "set" is doing a lot of work in that sentence.
openssl · certbot · sed · nginx · linux
- 4 stages★★★★★
the rollback that would not
the 14:00 deploy is bad. not subtle bad. checkout-returns-500 bad.
kubectl · docker · git · linux
- 4 stages★★★★★
the secret is out
mid-incident, under pressure, a teammate committed a "temporary" debug change.
git · psql · kubectl · ssh · linux
- 4 stages★★★★★
the retry storm
the payment provider is slow. not down. slow. down would be easier.
kubectl · redis-cli · linux
- 4 stages★★★★★
the clocks fell back, the cron did not
daylight saving ended at 02:00. the hour between 01:00 and 02:00 happened twice, as a treat.
ssh · powershell · psql · mixed
- 4 stages★★★★★
the debug flag nobody turned off
last tuesday someone set LOG_LEVEL=debug to catch a heisenbug. the bug was caught. the flag was not.
ssh · du · journalctl · sed · linux
- 4 stages★★★★★
the quick fix, applied to everything
one timeout value is wrong in prod-us. a thirty-second fix, via terraform, because process matters.
terraform · linux
- 4 stages★★★★★
the ttl was 86400
the migration to the new load balancer went perfectly. the runbook got a round of applause in slack.
dig · aws · ssh · iptables · linux
- 3 stages★★★★★
the flag that never died
new order-router ships tonight. eight servers, one deploy script, no rollback plan because nobody thinks they'll need one.
based on: based on Knight Capital Group, Aug 1 2012 — a dead test flag reactivated live-trading code across 7 of 8 servers, $440M lost in 45 minutes.
bash · ssh · linux
- 4 stages★★★★★
wrong terminal, right command
replication lag between primary and secondary has been climbing for hours. spam traffic is hammering the db, connections are piling up.
based on: based on GitLab.com, Jan 31 2017 — an engineer removed a postgres data directory on the wrong host while fighting replication lag, and five of five backup strategies had silently stopped working.
ssh · bash · postgresql · linux
- 3 stages★★★★★
the flag that took the internet with it
the s3 billing subsystem is running slow. a small number of servers need to come out of the index-cache pool to get diagnosed.
based on: based on AWS S3 us-east-1, Feb 28 2017 — a mistyped parameter during a routine debugging playbook removed far more capacity than intended, taking the index and placement subsystems down with it.
bash · aws-cli · linux
- 3 stages★★★★★
the regex that ate the edge
a new waf rule ships to catch a pattern in xss payloads. it looks fine in review.
based on: based on Cloudflare, Jul 2 2019 — a single WAF rule with a catastrophically backtracking regex was pushed globally in one step and drove edge CPU to 100%, causing a 27-minute worldwide outage.
bash · git · linux
- 3 stages★★★★★
the network that couldn't find itself
routine command, run a hundred times before: audit backbone capacity across regions, confirm nothing's oversubscribed.
based on: based on Facebook/Meta, Oct 4 2021 — a routine backbone capacity-audit command disconnected all datacenters from the internet, which caused Facebook's own DNS servers to withdraw their BGP routes, making facebook.com unresolvable — and locked staff out of the badge systems needed to reach the hardware.
bash · dig · linux
- 3 stages★★★★★
the config change that woke a sleeping bug
a customer pushes a routine config change to their service. nothing unusual about it on its own.
based on: based on Fastly, Jun 8 2021 — a valid customer configuration change triggered a latent software bug from an undeployed code path shipped weeks earlier, causing 85% of the CDN network to return 503s within about a minute.
bash · linux
- 3 stages★★★★★
the uploader nobody re-read
every ci run on this pipeline pulls a script straight from codecov's servers and executes it, no pinned version, no hash check. it's been like that for years.
based on: based on the Codecov Bash Uploader compromise, discovered Apr 1 2021 — attackers altered the publicly hosted uploader script used by thousands of CI pipelines, silently exfiltrating environment variables and secrets for roughly two months before detection.
bash · git · linux
- 3 stages★★★★★
the second that broke the clock
tonight's leap second has been on the calendar for months. everyone said the ntp patches were fine.
based on: based on the Jun 30 2012 leap second — a Linux kernel timer bug triggered by the inserted 23:59:60 UTC second drove Java processes into 100% CPU spins across many companies' fleets, including Reddit, Mozilla, and Qantas.
bash · linux
- 6 stages★★★★★
Ship on Friday, Pay All Night
friday, 16:52. the sprint's done, qa signed off two hours ago.
git · kubectl · gh · psql · linux
- 4 stages★★★★★
Forty Minutes to Expiry
cert-manager was supposed to renew this three days ago.
kubectl · cert-manager · curl · openssl · linux
- 4 stages★★★★★
One Slow Service, Three Casualties
checkout's p99 is climbing.
kubectl · curl · psql · linux
- 4 stages★★★★★
The Revert That Wouldn't
the deploy broke prod. clean fix: revert it.
git · psql · kubectl · linux
- 4 stages★★★★★
A Live Password in the Channel
mid-triage, someone pastes a stack trace into the incident channel.
psql · kubectl · gh · linux
- 4 stages★★★★★
The Job That Ran Twice
clocks moved over the weekend.
systemctl · psql · linux
- 4 stages★★★★★
The Quick Fix That Wasn't
one customer reports a bug in the new checkout flow.
curl · kubectl · linux