Sooner or later every system stops. A certificate expires, a disk fills up, a release introduces a bug, a provider has a problem. The difference between a mature team and an improvised one lies not in the number of outages, but in how it handles them: how quickly it notices, how orderly its reaction is and how much it learns afterwards.
During the outage: order matters more than speed
In the first minutes of an outage instinct pushes everyone to put their hands everywhere. The result is often chaos: three people change the same configuration, nobody knows what has already been tried, users get no news. A few simple rules change the situation completely.
1. Declare the incident
The first step is to acknowledge there is a problem and say so openly, in a dedicated channel. Waiting until you are sure it is serious only wastes time. Declaring an incident that later turns out to be small costs nothing.
2. Assign roles
Even in a small team it pays to distinguish at least three functions, which can be held by the same person if there are few people:
- the coordinator: doesn't touch the systems, but keeps the overall picture, sets priorities and assigns tasks;
- the responder: investigates and applies fixes, telling the coordinator what they are about to do;
- the communicator: updates users, customers and managers at regular intervals.
Separating coordination from technical work is the change with the greatest impact: someone buried in logs cannot also think about the big picture.
3. Restore before understanding
The immediate goal is to bring the service back, not to find the cause. If the problem appeared after a release, you go back to the previous version even without knowing why the new one doesn't work. If a server is struggling, you move traffic elsewhere. In-depth diagnosis comes later, calmly and with the data collected.
4. Write everything down
A chronological log, even written hastily in the incident channel, is invaluable: the time of each observation, hypotheses, actions and their effects. It helps during the outage, to avoid repeating attempts already made, and afterwards, to reconstruct what happened.
5. Communicate regularly
Users tolerate an outage far better if they know someone is dealing with it. An update every half hour, even just to say "we're still working on it, next update at 3 pm", is worth more than a long silence followed by a perfect explanation. A status page hosted outside your own infrastructure stays reachable even when the main service is down.
Classifying severity
Setting severity levels in advance avoids arguments at the wrong moment.
| Level | Example | Response |
|---|---|---|
| Critical | Main service unreachable for everyone | Immediate action at any hour |
| High | Important feature unavailable or very slow | Immediate action during extended hours |
| Medium | Limited problem with a workaround available | Action during normal working hours |
| Low | Minor defect with no significant impact | Scheduled among ordinary work |
After the outage: the postmortem
When the service is back and tension has eased, the most important part arrives: understanding why it happened and what to change. The document that gathers this analysis is called a postmortem.
Blameless
The fundamental principle is that the postmortem should be blameless. You don't look for who made the mistake, but for why the system allowed an inevitable human error to cause damage. If someone deleted the production database thinking they were on the test one, the right question is not "who did it?", but "why were the two environments so easy to confuse, and why didn't the deletion ask for confirmation?".
This approach is not leniency. It is the only way to get honest information: if reporting a mistake leads to punishment, people stop reporting them, and the same problems recur.
What it contains
A good postmortem is short and concrete:
- Summary: what happened, for how long, who was affected.
- Timeline: the key moments, from the first signal to full recovery.
- Causes: the chain of events and conditions that led to the outage. There is rarely a single cause.
- What worked and what didn't: in detection, in response, in communication.
- Corrective actions: each with an owner and a deadline.
Actions that really change things
The most effective corrective actions change the system, not people's behavior. "Be more careful" is not an action. "Add an alert when a certificate expires within fifteen days" is. The same goes for "automate renewal", "add a confirmation to destructive operations", "separate test and production credentials".
In my work the measure of a postmortem's quality is simple: after a few months, how many of the listed actions have actually been completed? A perfect document whose actions remain pending has improved nothing.
In short
- Declare the incident right away and separate whoever coordinates from whoever responds and whoever communicates.
- Restore the service before looking for the cause, even by going back to the previous version.
- Keep a chronological log and update users at regular intervals.
- Write a blameless postmortem that analyzes why the system allowed the error.
- Define concrete corrective actions, with an owner and a deadline, and check that they get completed.
