We deleted our staging environment
Six months in. The best thing we did was delete a database nobody had ever tested against.
Vedant · reviewed 7mo ago
Six months ago we deleted our staging environment. This post is unfinished and will stay that way until I know whether it was a mistake.
The case for deleting it: in four years, staging had been the target of exactly three real deployments. Everything else went to production directly. Meanwhile it cost roughly 400 engineer-hours a year to keep its data vaguely fresh, and the drift was bad enough that a passing staging test told you nothing about production.
The case against, which I still find persuasive: when we cut a release two hours before a Friday deploy, we had nothing. Not a degraded path, nothing. We waited nine hours. I do not think I would make that call again without a real answer.
- Shadow traffic against production with writes faked. Caught two schema incompatibilities in the first month.
- Database restore-from-backup rehearsal on a schedule, measured in minutes rather than described in a wiki page.
- A five-minute deploy freeze on Fridays. Not a technical control — a social one, and the only one that actually worked.
Open question I do not have an answer to: does the shadow-traffic setup actually catch the class of bug that staging would have? I believe it catches schema and serialization bugs. I have no evidence it catches the "this is slow only under real data volume" class, and I suspect it does not.
The number I keep avoiding putting here: the nine-hour wait cost us roughly one business day of engineering time and one customer-visible gap. Against 400 engineer-hours a year for staging, that is not obviously a bad trade — but it is a single incident, and single incidents are exactly the sample size I should not be drawing conclusions from.
How deep do you want this?
First time here? Set your depth once and every post honors it — skim the argument, read it properly, or take the footnotes too.
Full argument, prerequisites explained inline.
Revision history
Claim a handle to fork this post into your own seedling.
Marginalia
No notes yet. Select any sentence in the text to leave the first one — no account needed.
Ask this article
Answers are quotes from this post, not generated text — so they can be wrong by omission but never by invention. What it does not cover, it says so. If you have brought your own model key, a model summarizes those same quotes, and the quotes stay the source of truth.
Answer with a specific model
Your key is used for your provider if you added one, otherwise the site's. Automatic stays on free-tier models only.
Try to break this argument
The claims this post makes most concretely — the ones with numbers, absolutes or prescriptions in them — put to a model that has read only the sentence in question. Every objection quotes the post, and a claim nothing can be said against is reported as standing.
Where this sits
It leans on 1 other post. One post has picked it up since. Oldest at the top, so a claim resting on something written after it would be visible as an arc running upward.
When this changes
Leave an address and you will hear if this post is ever substantially revised — once, and only when it actually changes.