Self-Healing Deployment
Trigger a fault on purpose, then watch the pipeline spot it and heal itself.
- Status:
- Live
- Stack
- 6
- Architecture notes
- 6
Overview
A deployment pipeline that ships a new version of a web service, checks whether the new version actually works, and puts the old one back automatically when it does not. Two container slots sit behind one port: the outgoing version is stopped but kept, the new one takes the port, and a health gate decides which of them keeps it. Every attempt — success, rollback, or a build that never got that far — is written to a log that a small dashboard draws as a timeline. It generalises the rollback and validation automation I built for production microservices into an open demo you can run for yourself.
Architecture Breakdown
How it works
Core mechanics, failure recovery paths, and system design decisions.
- 01
Two slots, one door
The outgoing version is stopped and renamed `previous` rather than deleted, and the new one starts as `active` on port 8080. A stopped container keeps its filesystem, so restoring it takes a second — which is exactly why nothing here runs with `--rm`.
- 02
Health gate
validate.py requires all three of HTTP 200, a body reporting healthy, and a reply inside the latency budget. A 200 from a service that is broken underneath, or so slow it is useless, does not count as healthy.
- 03
Retry with backoff
A service that has just started often needs a moment, so a failed check retries on a growing delay — 2s, 3s, 4.5s, capped at 10s — before the gate gives up and hands over to rollback.
- 04
Rollback role
It deletes the broken release, renames `previous` back to `active`, starts it and waits for the port. First it checks whether the deploy actually touched anything: a build that failed before the swap must never turn into an outage.
- 05
Chaos on demand
Running the workflow with force_fail set makes the deployed container return HTTP 500 from its health endpoint on purpose. Recovery gets proved whenever you like, instead of waiting for a real outage to prove it for you.
- 06
Log and dashboard
Every ending appends one atomic entry — success, rollback or failed — to deployments.json. A dependency-free HTML page draws it newest-first as a colour-coded timeline and publishes to GitHub Pages after each run.
Key Takeaways
Engineering Highlights
- A rollback deliberately fails the run: recovering from a broken release is not a successful release
- Health is three checks, not a curl — status code, body and response time
- Chaos testing is one workflow input away
- A failed build never becomes an outage, because nothing is torn down before the swap
- Every tunable lives in one group_vars file, with secrets kept separate
Stack & Tools
- Ansible
- Docker
- FastAPI
- GitHub Actions
- Python
- GitHub Pages