Four generations of deploying the same legacy app

Part 1 of a series on ten years of deployment evolution for one legacy app on AWS — no Kubernetes required. Series index.

I have deployed the same application in four fundamentally different ways over ten years. It is a workforce-management product — a ~10-year-old PHP/Node monolith on EC2, three environments, a DR region, and a small platform team. Same business logic at the core, same database it has always talked to, but the machinery that gets code from a git tag onto a server has been rebuilt three times.

I inherited the first version. I built the other three. That last part is the part I want to be honest about, because it is the part that makes this interesting rather than triumphant: I replaced my own work twice. The blue/green system I was proud of four years ago, I tore out. The split-AMI system I built to fix that, I collapsed a few weeks later. Neither replacement happened because the previous thing was wrong. Each one was right for the constraints it was born under, and each one created the next set of constraints.

That is the through-line of this whole series. Deployment architecture does not evolve from wrong to right. It evolves as a re-balancing of which pains you are willing to carry. When scale, team size, and circumstances change, the pain you optimized away last time stops being the one that hurts, and a different one moves to the front. This article walks the four generations in order, and each section ends with the one thing that generation taught me.

Generation 1: one host, some scripts, and CodeDeploy

The version I inherited was the version a lot of small AWS shops still run, and there is no shame in that. Three EC2 hosts, each a long-lived pet. Deploys ran through AWS CodeDeploy pulling a revision onto the box and running lifecycle hook scripts. It worked. People had shipped features on it for years.

The problems were the kind you only notice once the fleet or the team grows past a certain size.

Every host was a snowflake. Configuration (the .env files, the tuning, the little manual fixes) lived on the hosts and nowhere else. If a box died, the knowledge died with it. There was no authoritative copy of "what is the correct state of a production server" anywhere except the production server. That is a terrifying thing to realize during an outage.

Runtime upgrades were manual. New PHP, new Node, new anything meant SSHing into each host and doing it by hand, which meant Test, Stage, and Prod drifted apart in ways nobody could fully account for. Library dependency conflicts showed up because two things on the same host wanted two different versions of the same underlying package, and there was no isolation to keep them apart.

And the network layout was a liability. The three hosts sat in the public subnet of the default VPC, each with its own Elastic IP, spread across only two availability zones. Every host was directly addressable from the internet, every host had a distinct public IP that external parties had to whitelist individually, and losing an AZ meant losing half the fleet. HA and security were compromised by the same decision.

None of this was incompetence by the people before me. It was a system that had grown by accretion, one reasonable small step at a time, until the sum of the steps was a machine nobody could confidently reason about.

What this generation taught me: the real cost of pet servers is not the servers. It is that the only correct copy of your system lives on the thing most likely to fail.

Generation 2: the full modernization

I led the migration off Generation 1 about four years ago, and it was a large piece of work. This was not a tweak. It was a new network, a new build system, a new deploy model, and a new source of truth, all at once.

The goals were explicit and I still stand by them: every EC2 machine should be disposable and stateless, the fleet should scale up and down with traffic, and releases should be zero-downtime. To get there:

A new VPC with a conventional network and security design, standing alongside the old one so existing services were untouched, with VPC peering to keep cross-VPC traffic cheap.

EC2 hosts moved into private subnets. A bastion host in the public subnet became the only jumpbox. A NAT Gateway gave the whole fleet a single egress IP, so external parties whitelisted one address instead of one per host.

Packer to bake a base AMI, Ansible to configure it, Terraform for all the infrastructure, and Bitbucket Pipelines for CI and CD.

Blue/green deployment through ALB weighted target groups. Traffic sat at 100% blue / 0% green; a deploy stood up the other color, and a listener-rule weight flip cut traffic over.

Config and secrets centralized in an S3 bucket instead of living on hosts, so a fresh instance could pull its .env on boot and be identical to its siblings.

Prod ran across three availability zones for real HA. Hosts were split into a "normal" tier and a "socket" tier so each could be tuned and scaled independently.

The ASG topology is worth stating plainly because it matters for the rest of the story. Three host types times two colors gave six ASGs per environment. That is the number to hold in your head.

The honest assessment: this was good. A deploy took about 10 minutes. Rollback was re-tagging the last good commit and re-running the pipeline. Instances were genuinely disposable — a new host was ready in about 15 minutes and could be a different instance type entirely just by changing config. Everything was in Terraform, everything was tagged for cost tracking, and the whole thing was reproducible in a way Generation 1 never was.

I am not going to trash Generation 2, and if you are reading this deciding what to build, Generation 2 is a completely defensible place to land and stay. It solved every Generation 1 pain I listed above, and it solved them the way the industry said to solve them at the time. For four years it did its job.

What this generation taught me: "the fleet is now reproducible from code" is worth an enormous amount, and it is worth paying real complexity to get it. Just know that you are choosing which complexity to carry, not whether to carry any.

Generation 2's hidden costs, and then May 2026

Blue/green's complexity was quiet for a long time, which is exactly why it was dangerous. It sat there as latent cost, invisible until the week everything went wrong at once.

The quiet costs first. Generation 2 baked a base AMI and then, at every single deploy, ran the full Ansible playbook again on every fresh instance — OS packages, PHP and nginx config, and the app code, all in one combined role tree at boot. We baked, and then at boot we essentially rebuilt on top of the bake. That made every deploy dependent on upstream package repositories being available at deploy time. Rollback needed CI to be online, because rollback was re-running the pipeline. The two-color state space doubled everything you had to reason about: which color is live, which target group has weight, is the idle color actually empty or just idle. And the driver script that orchestrated the weight flips grew, over the years, into hundreds of lines of bash holding all of that state in its head.

Then came the week of May 2026. Four upstream package sources went down more or less simultaneously — a PPA was removed upstream, the Ubuntu archive was under a DDoS, Launchpad was unreachable, and Ansible Galaxy was returning 502s. Because every deploy rebuilt from those sources at boot, the entire deploy chain was blocked. Under pressure, we built an emergency AMI from a snapshot of a running Stage instance to get unblocked. That AMI carried baked-in running services and a stale PHP opcache, and it combined with a health-check workaround someone had added days earlier to expose a race condition that had been latent in the blue/green ASG design the whole time. The ALB marked instances healthy from baked code within about 16 seconds of boot — before Ansible had even started — so the pipeline switched traffic to half-deployed instances and reported success on Ansible failures. Users got a couple of minutes of 502s from instances the pipeline believed were fine.

I am compressing a multi-day cascade into one paragraph on purpose. The full postmortem is Part 2 of this series: the five-layer defense-in-depth fix, the workaround that should have lived two days and lived weeks, the managed MongoDB connection tier we had to take from 500 to 1,500 connections. What matters here is the architectural lesson, not the firefight.

The incident did not tell me blue/green was bad. It told me that "rebuild the whole thing from upstream at boot, on every deploy" was a dependency I had been carrying for free right up until the day it cost everything. The health-check race was survivable in isolation. The emergency-AMI mistake was survivable in isolation. The upstream outage was survivable in isolation. They were only fatal together, and the thing that let them combine was that a deploy did far too much work, far too late, in far too coupled a way.

What this generation taught me: latent cost is the most expensive kind, because you never priced it until it presented the whole bill at once.

Generation 3: split the AMI into layers

The fix, in May 2026, was to stop rebuilding the world at boot. I split the single AMI into layers, borrowing the mental model directly from container images: slow-changing layers get cached, fast-changing layers get rebuilt cheaply.

An OS-base AMI at the bottom.

An app-base AMI baked weekly (and on demand) with OS packages, language runtimes, system users and directories — the slow-changing layer.

A release AMI on top, built per git tag, carrying the app code, rendered at deploy time.

The operational payoff was immediate and threefold. Release deploys dropped from about 10 minutes to 3–5 minutes per instance, because the OS-level work now happened once a week at bake time instead of on every release. Patch cadence decoupled: OS security patches landed via app-base rebuilds without forcing an app release, and app releases stopped dragging OS updates along with them. And critically, failure isolated. An upstream repo outage — the exact May failure — now only affected bake time. A failed app-base build did not block releases, because releases kept using the last-known-good app-base AMI; Terraform's most_recent data source resolved to it automatically. The class of incident that had just cost us production was structurally removed from the release path.

The other important change was inverting the health signal. In the old combined role, nginx was the gate, and that was the crux of the incident: nginx being up says nothing about whether the app is actually deployed. In the split-AMI release role, php-fpm became the gate. php-fpm ships stopped and disabled in the app-base AMI, nginx starts early on purpose to open the ALB health window while other tasks run, and the only thing that brings php-fpm up is the last task of the deploy. The ALB probes a PHP-rendered health endpoint that returns 502 until php-fpm starts. Deploy-readiness became an application-level signal instead of an infrastructure-level one. Three pipeline gates — publish, propagate, deploy — split ownership cleanly between the bake, the operator's Terraform apply, and the developer's git tag.

What did not change is as important as what did. Same VPC, same ASGs, same ALB and target groups, same blue/green model, same DR mirroring, same pipeline orchestration, same failure semantics, same git-tag triggers, same driver script. Generation 3 was still blue/green. It was a surgical fix to the layer that had failed, not a rearchitecture. The full design story — including a genuinely maddening multi-week bug where a mv into an existing directory nested fresh code one level down and ran the stale baked copy instead — is Part 3.

What this generation taught me: when a system fails, fix the layer that failed and resist the urge to rewrite the layers that didn't. The blue/green model was not the problem, so I left it alone.

Generation 4: notice you're already baking, and stop throwing it away

Generation 3 was live and stable, and I could have stopped. The thing that pushed me to Generation 4 was not an incident. It was a single embarrassing observation.

We already baked the app code into the release AMI at bake time. And then, at every instance boot, the release role wiped the app directory and re-deployed the exact same code from scratch — git tag resolution, S3 zip, unzip, configure — using the baked code only to warm the page cache. We were baking a cake and then scraping the icing off to re-ice it with identical icing, every single boot.

So Generation 4, in July 2026, is mostly the discipline to stop doing that.

Pin each bake to an explicit commit SHA. The artifact is the AMI now — one AMI per commit, env-agnostic by construction, promoted unchanged across test, stage, and prod. What soaked in test is byte-identical in prod.

Shrink boot from 5+ minutes of full Ansible to a sub-60-second "activation" step: fetch secrets, render instance-specific config, start services. Everything env-specific was already boot-time anyway, so nothing of substance moved.

Collapse the blue/green dual ASGs into one ASG per tier (six ASGs per environment become three), deployed with native AWS ASG Instance Refresh instead of ALB weight flips.

That third point is the one that retires the most machinery. Instance Refresh is AWS rolling your servers onto a new launch template in batches, with native alarm-gated auto-rollback and a native concurrency lock. A whole category of hand-rolled logic evaporates. The "is a deployment already running" check that we used to do by inspecting ASG emptiness is now the API rejecting a second refresh. The rollback that used to mean a ~20-minute rebuild from git is now relaunching the previous launch-template version, whose AMI still exists.

The release pointer is an SSM Parameter Store parameter holding the SHA that is the current release per environment. Its parameter history is, for free, the release history. Deliberately not a resolve:ssm: reference inside the launch template — because instance-refresh rollback relaunches the previous LT version, and a live SSM reference would resolve to the new value even during a rollback, which is exactly backwards. Explicit AMI IDs per LT version make rollback trivially correct. That kind of small, load-bearing decision is the subject of Part 5.

The measured before-and-after, from the field after full test cutover and roughly fifteen live drills:

Dimension	Generation 2/3 blue/green	Generation 4 (measured)
Deployable artifact	git tag, re-resolved at every boot	AMI pinned to a commit SHA
Instance boot	~5+ min (git fetch, S3 zip, full Ansible)	<60s activation, ~90s to serving
ASGs per env	6 (b/g × normal/socket/cmd)	3
Deploy, tag → serving	~10 min target	6m59s fresh build; ~6m pre-baked
Rollback	rebuild from git, ~20 min, non-deterministic	first good instance ~2 min, full fleet 5m30s–6m49s
Auto-rollback on error spike	none	native alarm-gated, zero human involvement
Dead-app self-healing	none	ELB health check, replaced in ~2.5 min
Cutover mechanism	ALB weight flip, ~1,400 lines of bash	`start-instance-refresh`, native

The deploy driver in the original design sketch was about 60 lines, down from the roughly 1,400 lines of weight-flip bash it replaced. In practice, field-hardening grew the real driver back to well over a thousand lines — capacity-adaptive batching, an incident-only --rollback mode, a deploy lock, per-tier failure cleanup — and I would be lying if I pretended the 60-line version survived contact with production. But the safety logic that used to be my bash is now carried by native AWS primitives. The lines that remain are orchestration, not reinvented rollback. Part 6 covers rollback across all four generations, and Part 7 covers the capacity-adaptive refresh numbers in detail.

There is one honest trade recorded in the ledger: blue/green's theoretical warm-fleet instant flip is gone. In practice it was already gone — the old pipeline tore the old fleet down with sleep 0, so the warm window was effectively zero. Generation 4's rollback beats the old path's real fallback, which was a 20-minute rebuild under pressure.

What this generation taught me: the biggest wins are often not new capability. They are noticing work you are already doing twice and doing it once. "We bake, then throw the bake away" was sitting in plain sight for years.

When to move, and when to stop

Four generations, three of them mine, two of them replacing my own work. If there is a meta-lesson, it is about reading the signals that tell you a generation has run its course — and, just as importantly, recognizing when it hasn't.

The signals that it is time for the next generation, in my experience, are these. Your deploy does work that could have been done earlier and cached, the way Generation 2 rebuilt from upstream at boot. A single failure only became fatal by combining with two others, which means your blast radius is too coupled (the May incident). You catch yourself doing the same expensive operation twice and shrugging (baking then re-deploying). Or the state you have to hold in your head to reason about a deploy has quietly grown past what one person can (the two-color state space, the hundreds of lines of driver bash). None of those is an outage. They are all latent cost accruing interest.

Now the part that matters more, because the industry is very good at telling you to move and very bad at telling you to stop.

You should stop at Generation 2 — or even Generation 1 — if your circumstances say so. If you run a small fleet, deploy rarely, and have no platform team, then the immutable-artifact, instance-refresh, SSM-pointer apparatus of Generation 4 is not a gift to you. It is a second system to maintain, and the person maintaining it is you, at 2am, without the context I spent months building. Generation 1's pets are a real liability, but if you have three hosts and deploy once a month, the honest fix might be centralizing your config and moving to private subnets, a slice of Generation 2, and stopping there. Generation 2's blue/green served this app well for four years, and the only reason I left it is that this app grew a deploy cadence and an incident history that made its latent costs real. If yours hasn't, its costs are still latent, and latent costs are free until they aren't.

I built Generation 4 because I had a decade of accumulated pressure, a fleet that scaled, a real incident to learn from, and the time to do it right. Take that as the precondition, not the recommendation. The right generation is the one whose pains you can afford and whose complexity you can staff. Everything else is résumé-driven architecture, and I have watched that cost more than any pet server ever did.

The last lesson, across all four: evolution is not progress toward a correct answer. It is a running negotiation with your own constraints. The skill is not knowing the newest pattern. It is knowing which pain you are currently choosing to carry, and being honest with yourself about why.

Part 2 takes the one paragraph I spent on the May 2026 incident and gives it the room it deserves: four upstream repositories failing in the same week, an emergency AMI built from a running instance, a two-day health-check workaround that lived for weeks, and the 16-second race that routed live traffic to half-deployed servers. It is the best worst week I have had in this job, and the anatomy of how three individually survivable failures compounded into one that wasn't.