A backup you have never restored is a hypothesis
Most businesses find out their backups don't work on the worst day of the year. Here is the drill we run instead, and what it costs to run it.
Every business we take over has backups. Almost none of them have ever restored one.
That gap is the whole problem. A backup job that reports “success” is telling you that a file was written somewhere. It is not telling you that the file can be read, that it contains what you think it contains, or that anyone at your company knows how to turn it back into a working system before your customers notice. Until you have done that on purpose, at a time you chose, your backup is a hypothesis.
What actually goes wrong
We have now run enough first-time restore drills to see the same handful of failures over and over.
- The job has been failing for months. Someone set up email alerts to an employee who left in 2023. The dashboard is green because nobody has looked at it.
- The backup covers the files but not the system. The data is there. The application that reads the data, the database that holds it together, and the server configuration that made it all run are not. You have the contents of the filing cabinet and no filing cabinet.
- The backup lives on the same network as the original. Ransomware that reaches your server reaches your backup drive twenty seconds later. Off-site and offline are not the same word as “second copy.”
- Nobody knows the order of operations. Even with good backups, the restore takes eleven hours because it is the first time anyone has attempted it and half of that time is spent finding credentials.
None of these show up in a status report. All of them show up during an outage.
The drill
Once a quarter we schedule a restore drill on a client system. It is a calendar event, not a surprise, and it runs roughly like this.
- Pick a real failure. Not “restore one file.” We pretend the primary server is gone (destroyed, encrypted, or seized) and rebuild from backup onto separate hardware.
- Start a clock. The number that matters is how long it takes to get staff working again, measured from the moment we start, not from the moment the backup software finishes.
- Have the client verify, not us. Our engineer can make a login screen appear. Only the office manager can tell you whether last Thursday’s orders are in there. They log in and check real records.
- Write down what broke. Something always does: a certificate, a license key tied to hardware, a print queue nobody documented. That list becomes work tickets.
- File the report. One page: what we restored, how long it took, what was missing, what we fixed. It goes to the owner, and it goes into the folder you hand your cyber-insurance carrier.
The first drill at a new client is usually rough. The second one is boring. A boring drill is the entire product.
What “good” looks like in numbers
Two numbers describe your actual exposure, and every owner should know theirs.
How much work can you afford to lose? If backups run nightly at 2am and the server dies at 4pm, you have lost a full business day of order entry, chart notes, or work orders. For some offices that is survivable. For a distributor entering orders continuously, it is not, and the backup schedule needs to change.
How long can you be down? “We’ll be back up in a few days” is a fine answer for an archive server and a business-ending answer for the system your front desk runs on. Sort your systems by that question before you spend a dollar on recovery infrastructure, because the answer determines what you actually need to buy.
Most businesses discover, once they write these two numbers down, that they are spending recovery money on the wrong systems.
What it costs
Less than people expect, and the cost is mostly time rather than hardware. A quarterly drill on a small practice or office runs a few hours of engineering, scheduled after close so nothing in production is touched. It is included in our managed engagements, not because we are generous, but because we are the ones who get called at 2am, and we would rather find the broken license key on a Tuesday evening in advance.
If you are not a client and you take nothing else from this: pick your most important system, book two hours next month, and try to restore it somewhere harmless. Whatever you learn will be cheaper to learn then than during the real thing.