A backup you have not restored is not a backup
Every team we meet has a nightly backup job. Most have never restored from it. Those are not the same thing, and the gap between them is where the bad afternoon lives.
The job runs nightly. It reports success. Everyone believes the problem is handled, and nobody finds out otherwise until the day they need the file.
Success is a weak signal
A backup script that exits zero has told you one thing: the last command in it exited zero. That is not the same as a file you can restore.
Dumps stop halfway because a disk filled, and the pipeline still reports success because the failure happened upstream of the exit code anyone checked. Archives end up forty bytes long - a header and nothing else - and get rotated into the retention window on schedule, night after night, quietly replacing the last good copy.
The script was working exactly as written. It was written to run, not to verify.
Four checks that cost almost nothing
- Decompress it. A gzip integrity test catches truncation immediately, and if the file will not decompress then nothing downstream matters.
- Set a size floor. You know roughly how big last night's dump was. A file dramatically smaller than that is a failure wearing a success message, so refuse it.
- Rotate only on success. This is the one people get wrong: if tonight's backup is broken and rotation runs anyway, you have deleted a good backup to make room for a bad one.
- Record what happened somewhere a person will actually see, so silence becomes suspicious rather than reassuring.
None of that is clever. All of it gets skipped, because the script is written on a day when the deadline is somewhere else.
The only check that counts
Everything above verifies the file. None of it verifies that the file is a database.
The real test is a restore into a scratch instance, followed by counting something. Row counts on your largest tables, and one query a person would actually run. That exercises the whole chain: the dump, the compression, the transfer, the storage, the schema, and your assumption about which database was being backed up in the first place.
Schedule it. Not nightly - it is heavier than the backup itself - but often enough that the answer is never older than your last schema migration.
The first restore should not be the one that matters.
Where it goes
A backup on the same machine as the database is a copy, not a backup. It survives a dropped table. It does not survive the machine.
Offsite does not have to be complicated, but it does have to be somewhere a compromise of the primary host cannot reach - which mostly means credentials that do not live on the box being backed up.
What this looks like in practice
Nightly dumps, compressed, checked for integrity and against a minimum size before anything rotates. Retention advances only on a file that passed. A restore rehearsal on a schedule, into a throwaway database, with row counts compared against the source.
It is unglamorous, and on a small system it is about forty lines of shell. The value was never in the sophistication. It is in the part where somebody actually looked.