Lessons Learned

A few elemental lessons from this exercise will be put into place, since the site did eventually become unrecoverable.

• Make and test backups. It should be obvious, but taking the backups regularly was one thing, actually then moving them to another location successfully was another. Even restoring from the same server isn't enough when you can't actually access the backup media.
• Automate the backups. On systems which aren't actually all that important, it's really easy to not bother to do the backups and the updates. I've an ansible script that runs around doing a lot of this work, but I didn't run it against this test system, as it wasn't important enough.
• Don't skimp on major OS releases. Building on Ubuntu 22.4 is all well and good, but even LTS releases are probably better off being treated as full wipe and restore scenarios.
• Most modern open source systems are full of tightly coupled dependencies, so just trying to change the major component, such the CMS, is often going to make a dependency (such as NGinx) go slightly haywire.