Categories
Projects Website

Failed Refit

Today’s server refit failed due to some bad firmware programs & bad luck.

Hey everyone – some unfortunate and bizarre news today I’m afraid – we weren’t able to refit our server as planned. Our server configuration and data was restored to it’s state as of 6 am (BST) this morning – the start of our maintenance period.

To highlight what happened, let me explain what our plans were and why they failed:

Plan A: Use beta bootloader and boot to USB SSD

This was a complete hassle. For anyone who doesn’t know, our server is a raspberry pi 4. On May 15, the Pi Foundation released a beta bootloader that allowed booting to USB without needing an SD card on the Pi 4.

To be frank – this is a complete gimmick. A PR stunt. It’s finicky, unstable and half-measured. We followed almost a dozen online tutorials to get this working, but every time the pi would do it’s infamous triple-orange flash that means it doesn’t recognise any boot media.

The one solution that actually booted for us was cloning the SD card data to the SSD. This booted, but for some reason the Pi had the error: Device write error: Device has physical write protection – which it didn’t 😕

After 6 hours of trying and failing to circumvent this error, we moved on to….

Plan B: Use BerryBoot to install OS to an external drive

BerryBoot is a GitHub project similar to the Pi Foundation’s own NOOBS software – but it allows you to install OS’s to external drives.

However, this project hasn’t been updated (at the time of writing) since October 2019 – so this didn’t work properly either. The server would boot using BerryBoot, but it took 14 minutes and 54.2 seconds according to the built-in analysis tools. That is unreasonably and ridiculously slow. The server was stupendously slow to respond to commands and our websites would timeout before they could be loaded.

We tried a few fixes, but found ourselves making the sluggishness worse instead of better. This plan was a wash, too.

So what’s next?

Well…. We have some ideas. Each have their pro’s and con’s:

Plan C: Boot from SD card, move all server-related data to SSD

This is certainly an option – it would allow the tried, true & stable boot to SD card, while still providing some speed & capacity benefits to the server, however the speed benefits would probably be negligible, and the capacities would be fixed – i.e. each part of the SSD would have to be a set size, it wouldn’t be possible to resize, say, the database storage area if we needed to.

Plan D: Ditch the Raspberry Pi

This is the option that 99% of companies go with in the first place – using a Desktop computer or even a blade-style rackmount server unit instead of a Raspberry Pi. This would allow future upgradability, greater flexibility in choice of OS, higher-power CPU, more and/or faster RAM and much more. However, it is physically larger, making transport & storage more difficult, requires more power, making runtime more expensive, and is generally overkill for what we need right now.

At the moment, our Pi is coping with the computational load just fine aside from the speed being bottlenecked by the slow SD card technology and running out of space sooner than expected. For those two small problems, spending hundreds or even thousands of pounds on equipment upgrades and higher running costs simply isn’t necessary yet.

Inevitably, as both our company and any client companies who we host websites for grows, load is going to increase and there will come a time where a Pi’s 32-bit OS and ARM-based CPU simply won’t be enough anymore. We’ve known this from day one of planning our server technology, but that day simply isn’t even close just yet.

Closing remarks

For the time being, we’ve restored our server to it’s physical & digital state from 6am this morning whilst we investigate what the causes for our problems were & any possible solutions.

If we don’t find anything, we’ll look into Plan C and if we decide to go ahead, we should be able to perform the necessary changes via live patching – no additional downtime should be required.

We’ll keep you updated – you’ll know more when we do. Thanks for reading and we apologise for any inconvenience.

Leave a Reply