An informal update to our official statement regarding the outage on the 11th February:
We’ve re-written most of our custom utilities, including a backup and restoration program which is critical to the longevity of our server – we rewrote a backup script on the day of the outage which executes at 6am (GMT) every day.
However, we haven’t been able to test our matching restoration script due to the risk of data loss. For this reason, we’re taking the site offline to prevent data loss whilst we create a backup and restore it within the time window, to prevent customer data loss – and apply fixes, if needed.
The way our automatic backups work are that the website files are compressed into an archive. This archive can retain extended meta information, but can’t be encrypted.
That archive, along with a dumpfile from the database, is then nested into another archive that is encrypted, but not compressed. This type of archive can be encrypted, but can’t retain meta information.
Individually, neither of these archives are suitable for backing up a webserver – but combined, they are almost perfect.
As such, the extraction process goes something like this:
- Copy the original backup to a temporary directory
- Decrypt and extract the top-level archive into the temporary directory
- Delete the copy of the full archive
- Stop the webserver temporarily
- Restore the database using the extracted dumpfile
- Update the database, noting the date of the restoration for record-keeping reasons
- Delete the database dumpfile, leaving only the nested archive in the temporary directory
- Extract the nested archive into the temporary directory
- Delete the nested archive, leaving only the website files from the backup
- Replace existing website files with files in the temporary directory
- Delete the temporary directory
- Restart the webserver and reload the database
In short, a lengthy process that will take us a while to do manually, but should only take a few minutes when completed by a script.
Thanks for reading, and we apologise for any inconvenience. See you next time!