Anatomy of a cascading incident: a PPA removal, an emergency AMI, and a four-year-old race condition

Part 2 of a series on ten years of deployment evolution for one legacy app on AWS — no Kubernetes required. Series index.

Most outages I have run have a single root cause you can point at and a fix you can ship the same day. This one did not. Over three days in May 2026, a workforce-management product I look after — a ~10-year-old PHP/Node monolith on EC2, three environments, a DR region, and a small platform team — served roughly two minutes of 502s to real users during what should have been a routine blue/green release. The two minutes were the visible part. Behind them sat four upstream failures, one emergency decision I would make again, a workaround from weeks earlier that nobody had scheduled for removal, and a race condition latent in the deploy path since I built the platform four years before.

None of the individual links was fatal. Each was survivable on its own. They only killed the deploy because they lined up.

I described the four deployment generations of this app in Part 1. This is the incident that ended Generation 2 and forced Generation 3. It is worth reading slowly, because the failure mode — infrastructure health signals masquerading as application readiness — is not specific to my stack. If you run blue/green behind an ALB, you probably have a version of this waiting.

Act 1: the week the package repositories died

Here is the thesis, up front, before the story earns it: upstream package repository availability is part of your deploy pipeline's contract, whether you wrote it down or not. If your instances pull nginx or PHP or an Ansible collection from a public repo at boot, then that repo's uptime is your uptime. Mirror fallbacks and retry wrappers are not a nice-to-have you get to backlog. They are load-bearing. I learned this the way you usually learn these things, all at once.

The first symptom was apt update failing across every fresh instance:

The ondrej/nginx PPA had been withdrawn upstream. Not degraded — gone. That PPA was where our base image pulled its nginx build, so every new EC2 spinup in the deploy chain now failed at package installation. A deploy that cannot install nginx cannot produce a healthy instance, and blue/green with no healthy green instances is just an expensive way to keep serving blue.

I switched us to an AWS-hosted nginx mirror cached inside the VPC and wrapped apt update in a retry. That should have been a two-day patch. Hold that thought; it is Act 3.

While I was doing that, the second fire started. archive.ubuntu.com was being DDoS'd upstream, so ordinary apt steps against the Ubuntu archive began hanging or returning 503s. Mirror swap, apt-retry wrapper in the Ansible roles, and a note-to-self to build a known-good mirror fallback list that rotates on failure.

Third: ppa.launchpadcontent.net, Launchpad's content host where PPA fetches resolve, went unreachable. Same class of failure, same mitigation. I burned an hour auditing our own VPC routing and NAT egress before accepting the problem was genuinely upstream. Egress was healthy. The internet was just having a bad week.

Fourth: galaxy.ansible.com started throwing intermittent 502 Bad Gateway during collection installs. This one had the cleanest fix (five attempts with backoff), and after it shipped, the logs read the way you want them to:

Four independent public dependencies, four different operators, all failing inside the same week. The instinct is to treat each as a freak event, patch it, and move on. That instinct is how you end up here. The correct read is structural: my pipeline had four hard, unbuffered dependencies on public infrastructure I do not control, and I had gotten away with it for four years because those four things had never all wobbled at once. The week they did was not bad luck. It was the bill coming due.

Act 2: the emergency decision

By day two the Packer pipeline was effectively dead. Packer builds a base AMI by launching an instance, running apt and Ansible against it, and snapshotting the result. When apt cannot reach a working mirror and Galaxy is returning 502s, Packer cannot build. No base AMI means no new instances, and I had a release that needed to go out.

So I did the thing you do. I took a running Stage instance — already up, nginx and PHP-FPM installed and working, a known-good package set on disk — and snapshotted it into an AMI directly. Not Packer-built. A photograph of a live, running box, taken on 2026-05-05.

I want to defend this decision before dismantling its consequences, because the two are not in tension. Under the constraints I had, snapshotting the running Stage instance was the right call. The alternative was no deploys at all until four separate upstream operators fixed their infrastructure on their own schedule. A snapshot of a working box gave me a deployable AMI in minutes using packages I had already validated. If you are ever in that seat, you make the same move. I would make it again.

What matters is what that AMI inherited: a snapshot of a running instance is not a Packer build, even if it boots the same. It carries state.

Two pieces of inherited state did the damage.

First, the snapshot captured nginx and PHP-FPM in the running and enabled state. On a Packer-built base image, those services are installed but not started; the deploy is what starts them, at the end, deliberately. On my snapshot, systemd would bring them up automatically at T+1s of boot, because that was their state on the live box when I photographed it. The AMI booted into "serving traffic" instead of booting into "ready to be configured."

Second, and more subtly, the snapshot baked in PHP's opcache: compiled bytecode from the previous release. When a new instance booted from this AMI and Ansible later extracted the real, current release from S3 onto disk, PHP-FPM kept serving the stale bytecode it had cached at snapshot time. New code on disk, old code in memory, no error anywhere. I fixed that symptom with an explicit PHP-FPM restart and an opcache flush at the end of the deploy, and left a note about opcache.validate_timestamps=1 with revalidate_freq=2 as a longer-term auto-invalidation option we keep disabled for performance.

The opcache issue was annoying. The running-and-enabled services issue nearly took the site down, because a loaded gun was already sitting in the nginx config, waiting for exactly this.

Act 3: the seed, planted weeks earlier

Go back to Act 1, to the ondrej PPA removal and the two-day patch.

When that PPA vanished, a side effect surfaced that I have not mentioned yet: on the affected instances, the running nginx config was missing the /validate endpoint. /validate is our ALB health check path. If the ALB cannot get a 2xx from it, it marks the target unhealthy and pulls it from rotation. With the endpoint missing, perfectly fine instances were being marked unhealthy, and the deploy chain, already fighting four upstream fires, was now fighting its own load balancer.

So I added a workaround to keep the ALB happy:

Read that carefully. It does not check anything. It returns 200 unconditionally, the instant nginx is running, regardless of whether the application behind nginx is deployed, migrated, connected to its database, or even present. It says "nginx is up" and calls that health.

At the time this was fine. It was a targeted patch for a specific breakage, meant to live two days until the real config came back. Here is the problem: nobody gave it a removal date. It was not filed as tech debt with an SLA. It was not tracked. It just sat in the config, returning 200, while the incident moved on to the next fire, and the two-day workaround quietly lived for weeks.

This is the lesson I most want you to take from this article, and it is boring, which is why people skip it: workarounds applied under incident pressure need expiry dates baked in at the moment you write them. Not "we'll clean it up later." A ticket, an owner, a removal SLA, written down before you close the incident. The reason is not tidiness. A workaround is a deliberate suspension of a safety property, and a suspended safety property that outlives its emergency becomes a latent defect nobody remembers is load-bearing. When the next incident arrives, it does not arrive into the system you designed. It arrives into the system you patched.

In Act 4, this line meets the emergency AMI, and together they teach my pipeline to lie.

Act 4: detonation

Now the pieces are all on the table. Let me put them in order.

A new instance boots from the emergency AMI. Because that AMI inherited running-and-enabled nginx and PHP-FPM, systemd starts both almost immediately, serving the stale baked code. Because of the /validate { return 200; } workaround, nginx answers the ALB health check with 200 the moment it is up. The ALB check runs at an 8-second interval and needs 2 consecutive passes, so roughly 16 seconds after boot the target flips to healthy and the weighted target group starts forwarding real traffic to it.

At this point Ansible has not started. The instance is "healthy" on the strength of a hardcoded 200 from a photograph of last week's Stage box.

Walk the failure the way the pipeline experienced it. The playbook runs for minutes, and somewhere in the middle a PHP task fails. The entrypoint's failure path does the correct thing: it calls aws autoscaling set-instance-health --health-status Unhealthy to take the broken instance out of service. And the ASG rejects the call, because the instance is still inside its health-check grace period, the window where the ASG deliberately ignores health signals to let new instances warm up. The instance that most needs to be pulled cannot be pulled.

Meanwhile the pipeline's own health poll is checking /validate. It has been getting 200 since T+16s and it keeps getting 200, because /validate reports "nginx is running," which tells you nothing about whether the deploy that just failed actually succeeded. The pipeline reads 200, reports SUCCESS, and switches the blue/green weights to promote the freshly "deployed" instances to primary. A pipeline reporting success on a failed deploy is worse than one that crashes, because it does not stop. It confidently routes production traffic onto broken code. Two minutes of 502s.

The root cause is not one bug. It is three, compounding, each individually survivable:

A — the workaround. /validate returns 200 the instant nginx runs (the Incident 1 patch that never expired).

B — the snapshot. The emergency AMI inherited running nginx/PHP-FPM, so the ALB saw health ~16 s after boot, before Ansible had started (the consequence of snapshotting a live instance).

C — weak failure propagation. An Ansible failure could not reliably mark the instance unhealthy, because set-instance-health got rejected during the grace period and the pipeline trusted the ALB over the deploy's own exit code (a pre-existing gap I had never been forced to notice).

A and B together are the race: the ALB was declaring health from an infrastructure-level signal (nginx is up) instead of an application-level one (the deploy finished and succeeded). C is what turned the race from a cosmetic flicker into a customer-facing outage, by letting the pipeline believe the ALB.

Remove any one of them and there is no incident. Kill the workaround, and /validate reports real readiness, so the ALB never goes healthy on baked code. Use a Packer AMI, and services do not start until the deploy starts them, so there is no 16-second head start. Fix failure propagation, and the failed Ansible run aborts the pipeline before the weight switch. I had all three defects for years and shipped hundreds of clean deploys, because it took the specific alignment of a dead Packer pipeline forcing an emergency snapshot into a config still carrying an un-expired workaround to line them up. That is what "latent" means: not that the bug was hard to find, but that nothing ever pulled the pins on all three grenades in the same deploy.

A second fire, in parallel: MongoDB connections

One thread ran alongside all of this, because the same conditions triggered a different failure class. Elevated traffic plus repeated back-to-back blue/green deploys meant each color of each ASG brought up its own fleet, each host with its own pool of PHP-FPM workers, each worker holding MongoDB connections. The combined fleet blew past the 500-connection hard cap on our managed MongoDB tier. Connection exhaustion under load is its own outage, independent of the AMI race.

The immediate fix was a tier upgrade from the 500-connection tier to the 1,500-connection tier, plus a Slack alert at 70% of cap (~1,050 connections) so we see the wall coming instead of hitting it. Headroom is not a fix, though. The growth rate pointed at an application-level leak in the Mongo client lifecycle, so the real work, still open as of this writing, is per-route connection-acquisition instrumentation. The lesson generalizes past MongoDB: your connection cap is a shared resource every color of every ASG draws from at once, and blue/green doubles your peak draw during the overlap window by design. Size for the overlap, not the steady state, and alert well before the cap.

Act 5: the five-layer fix

When I sat down to fix Act 4 properly, I resisted the urge to patch the three defects independently and call it done. Independent patches are how you get three more latent defects. Instead I wrote down one design principle and made every layer serve it:

The only path to nginx and PHP-FPM running on a fresh instance is the end-of-deploy "Start services" task, which runs only after every other Ansible task has succeeded.

If that invariant holds, the ALB physically cannot see a healthy /validate until the deploy is complete and correct, because until then the service answering /validate is not running. Health becomes application readiness by construction, not by convention. There are five layers instead of one because each closes a distinct vector through which a service could come up early. They look redundant; they are defense in depth, and I will name the regression each one catches.

Layer 1 — cloud-init bootcmd, in the Terraform modules. At T+2s of boot, before the ALB's first health check can land, stop and disable nginx and PHP-FPM. This kills vector B directly: whatever running state an AMI inherits, Packer-built or emergency snapshot, gets forced back to "stopped" the instant the box comes up. It protects the fresh-boot window. (One wrinkle worth recording: systemctl stop accepts a glob like php*-fpm.service, verified on systemd 249 / Ubuntu 22.04, but disable does not, so the disable targets are listed by explicit unit name.)

Layer 2 — a stop task in the app-specific nginx role. There was a residual window I would have missed if I had stopped at Layer 1. Mid-deploy, the shared nginx role runs a task named "Reload nginx" that is actually configured state: restarted, so despite the name it starts nginx. That left nginx up for roughly 16 seconds while later composer and cron tasks ran, a window in which /validate would answer 200 mid-deploy. Layer 2 stops nginx immediately after the role finishes its config, collapsing that window from 16 s to about 1 s. It protects the mid-deploy flicker vector. I made this fix in the app-specific role, not the shared one, because the shared role is consumed by other services and carries a commented-out state: line whose history nobody could explain; changing shared behavior without a blast-radius audit is how you turn one incident into two.

Layer 3 — a stop-and-disable task at the top of the app playbook. Pure insurance. It is idempotent, so on a healthy deploy it does nothing visible. Its job is to be a regression canary: if anyone later reintroduces a running service earlier in the boot or base-image path, this task quietly stops it and the deploy stays correct despite the upstream mistake. It protects against future regressions for free.

Layer 4 — the "Start services" task at the very end of the app playbook. The blessed start, the single source of truth for "this instance is ready to serve traffic." It runs only after every prior task succeeded. This is the first moment in the entire lifecycle that /validate returns 200, and by then the deploy is provably complete. It is the positive half of the invariant.

Layer 5 — exit-code and health propagation in the entrypoint and failure-notify script. The entrypoint now exits non-zero on any Ansible failure. Because Layers 1–4 guarantee nginx is not running on a failed deploy, the instance never became ALB-healthy, never entered rotation, and there is nothing for the grace period to protect. The pipeline sees the non-zero exit, aborts, and never reaches the weight switch. This closes vector C: failure now propagates all the way up instead of being overruled by a health check that was never measuring the deploy.

What the fix looks like on a clean deploy

Here is the post-fix boot sequence, read against the Act 4 timeline — same clock, opposite outcome:

And the post-fix verification numbers from a real blue/green deploy after the change:

Socket ASG: 4m53s, ok=109 changed=36 failed=0

Normal ASG: 4m14s, ok=94 changed=28 failed=0

Galaxy retry: attempt 1/5 OK

Entrypoint: exit 0

The flicker timeline on the socket ASG, measured:

Sixteen seconds down to about one.

What the fix looks like when the deploy fails

This is the part that matters: a deploy pipeline is only as good as its behavior on the bad path. Walk the same failure that caused the incident — a PHP task failing mid-Ansible-run:

Ansible fails. The playbook never reaches the Layer 4 "Start services" task, because that task only runs after everything before it succeeded. nginx and PHP-FPM stay stopped (Layer 1 disabled them at boot; nothing blessed re-enabled them). The ALB never sees a 200 on /validate, so the instance never goes healthy, so it never enters the target group's rotation. The entrypoint exits non-zero (Layer 5). The pipeline sees the failure and aborts. The blue/green weight switch never happens. The previous color, the one that was serving fine the whole time, keeps serving. Zero 502s.

That is the entire point of the redesign in one sentence: on the failure path, a broken instance is silent and invisible to the load balancer instead of loud and primary. The worst thing a failed deploy can now do is not deploy.

Lessons

Six things I took out of this, in rough order of how much they will save you:

Health checks are infrastructure signals, not application-readiness signals. A 200 from nginx means nginx is running. It does not mean the app is deployed, migrated, connected, or correct. If your health check can pass before your deploy runs, it is measuring the wrong thing, and the fix is to make the end of the deploy the only thing that can expose a passing check. This is the headline; the rest are corollaries.

Workarounds need expiry dates, written down at the moment you write them. The two-day /validate patch lived weeks and became the linchpin of the outage. File every incident workaround as tech debt with a removal SLA before you close the incident, or it will still be there when the next one arrives.

Emergency AMIs from running instances are dangerous and must be flagged and replaced fast. A snapshot inherits every running, enabled service and every warm cache. It is a photograph of state, not a clean build. Track it and kill it the moment the real pipeline is back.

Failure propagation deserves as much design as success. A pipeline that reports SUCCESS on a failed deploy is more dangerous than one that crashes. Make the deploy's own exit code the authority, not a health check it does not control.

Defense in depth is not redundancy. The five layers each close a different vector — fresh boot, mid-deploy flicker, future regression, blessed start, failure propagation. Remove any one and a class of bug walks straight through.

Upstream repository availability is part of your pipeline's contract. Four public dependencies failed in one week. Mirror fallbacks and retry wrappers are the difference between an upstream operator's bad day being their problem or yours.

How this became Generation 3 — and the irony waiting in Generation 4

This incident did not just get patched. It ended an architecture.

Generation 2 — Packer base AMI, Ansible-at-boot, blue/green — was good for its time, and I said so in Part 1. But the incident exposed a structural weakness I could not un-see: the deploy did real, failure-prone work at boot, on the live instance, in the window between "instance exists" and "instance is trusted." Every second of Ansible-at-boot is a second where a fresh box is half-configured and one hardcoded 200 away from being declared ready. The five-layer fix made that window safe. The window itself was the problem.

So Part 3 is about splitting the AMI into layers — an OS base, a weekly-baked app-base, and a per-release layer — so that almost none of the risky work happens at boot anymore, and PHP-FPM itself becomes the health gate instead of a hardcoded endpoint. Release deploys dropped from ~10 minutes to 3–5. That is the direct architectural descendant of this outage.

And here is the irony I did not appreciate until Generation 4, later in the series. The original sin of the emergency AMI was that it baked running state (services already started, caches already warm) and baked it uncontrolled, as an accidental side effect of photographing a live box. The lesson I took in May was "stop baking running state." But the eventual Gen 4 design does the opposite: it bakes everything, one fully-configured AMI per commit, with zero boot-time configuration. The difference is not what gets baked. It is control. The emergency AMI baked state by accident and let it start itself; the Gen 4 AMI bakes state on purpose and lets nothing start until a deliberate, blessed step says so. Same technique, opposite discipline. It took this incident to teach me which one I actually had.

Next in the series: Part 3 — how I borrowed container image layering to split one AMI into three, inverted the health check so PHP-FPM gates its own readiness, and cut release deploys from ten minutes to three, all without touching Kubernetes.