Self-Healing Deployment
Trigger a fault on purpose, then watch the pipeline spot it and heal itself.
- 01Two slots, one door
- 02Health gate
- 03Retry with backoff
- 04Rollback role
- 05Chaos on demand
- 06Log and dashboard
Architecture Overview
A deployment pipeline that ships a new version of a web service, checks whether the new version actually works, and puts the old one back automatically when it does not. Two container slots sit behind one port: the outgoing version is stopped but kept, the new one takes the port, and a health gate decides which of them keeps it. Every attempt — success, rollback, or a build that never got that far — is written to a log that a small dashboard draws as a timeline. It generalises the rollback and validation automation I built for production microservices into an open demo you can run for yourself.
How it works
Core mechanics, failure recovery paths, and system design decisions.
Two slots, one door
The outgoing version is stopped and renamed `previous` rather than deleted, and the new one starts as `active` on port 8080. A stopped container keeps its filesystem, so restoring it takes a second — which is exactly why nothing here runs with `--rm`.
Health gate
validate.py requires all three of HTTP 200, a body reporting healthy, and a reply inside the latency budget. A 200 from a service that is broken underneath, or so slow it is useless, does not count as healthy.
Retry with backoff
A service that has just started often needs a moment, so a failed check retries on a growing delay — 2s, 3s, 4.5s, capped at 10s — before the gate gives up and hands over to rollback.
Rollback role
It deletes the broken release, renames `previous` back to `active`, starts it and waits for the port. First it checks whether the deploy actually touched anything: a build that failed before the swap must never turn into an outage.
Chaos on demand
Running the workflow with force_fail set makes the deployed container return HTTP 500 from its health endpoint on purpose. Recovery gets proved whenever you like, instead of waiting for a real outage to prove it for you.
Log and dashboard
Every ending appends one atomic entry — success, rollback or failed — to deployments.json. A dependency-free HTML page draws it newest-first as a colour-coded timeline and publishes to GitHub Pages after each run.
Engineering Highlights
- •A rollback deliberately fails the run: recovering from a broken release is not a successful release
- •Health is three checks, not a curl — status code, body and response time
- •Chaos testing is one workflow input away
- •A failed build never becomes an outage, because nothing is torn down before the swap
- •Every tunable lives in one group_vars file, with secrets kept separate