Planned Server Maintenance

We’ve found some gremlins in our server that we need to displace. On Monday, we’ll be doing just that.

Notice of Amendment

This post has been edited since it’s publish. Any text that has been stricken has an update at the bottom of the post.

At 4am today, we announced on Twitter that we’d placed our server in Quarantine mode as a purely precautionary measure following a string of bizarre behaviours and discoveries within our server’s command line and our internal services. More detailed information will be available in our official statement of the incident which will be written & published once our server is freed of it’s gremlins.

We’ve investigated thoroughly and found no indication that our customer-facing sites (this primary one and our AKA service) are affected, nor are partner websites which we host.

With regards to the issues with our server – mentioned a short while ago, it’s been determined that a large-scale repair is necessary. On Monday morning our primary website will be temporarily transferred to a substitute system along with the sites of our partner’s, at their request.

This temporary system will continue to serve these limited sites in Quarantine mode until we’re confident we’ve found & rectified any and all issues on the server.

All non-essential sub-sites (like our AKA service, internal email system and file-sharing, plus more) will not be available on this substitute system due to software constraints.

We’re also not following our standard practice this time, we’re not going to constrain ourselves to 1 hour, 3 hours or 24. We’ll continue to keep our substitute system online until such a time as our hardware is tested & proven ready to stay online for an extended period as our website itself nears it’s final “Stable” state.

We’ve had a lot more problems with our server lately than we’d like. As a result, we’ve decided to go overkill this time. In hindsight, there has been very gradual and subtle reductions to our server’s reliability compared to it’s first few months of operation – we intend to find out exactly why and in so doing, rectify any problems and once again have a reliable system to serve our customers online.

Although we’re only facing a small selection of insignificant glitches (gremlins), they’re progressing systematically. Granted, only slowly, but we’re not prepared to “tempt fate”. We plan to completely wipe our server, re-initialise it, run as many tests as are required and proceed based on the results.

At that stage, we should be able to implement any number of fixes/tweaks to get the performance we expect from the system, without risk to our customer data.

Once the original server is cleared of it’s gremlins and deemed secure, we’ll semi-manually rebuild the website and databases to help prevent any residual gremlins re-appearing from one of our backups.

We’ve had a lot of problems with our hardware of late, and for that we can only apologise and ask that you be patient with us. We’re new to self-hosting and we’re still getting a feel for the ropes, but it’s safe to say that our knowledge has progressed significantly from when we first transferred to our own hardware.

We’re always working to better ourselves and our services. We thank all our loyal customers for their patience during this time.

UPDATE 28/06/2020

All websites will now be transferred to our temporary substitute, not select ones as originally planned. The websites should be functionally indistinguishable from the originals, aside from their remaining in Quarantine mode (where applicable).

The decision has now been made to move on from the Pi entirely and instead to transfer most of our server to a desktop tower-type computer that we have in our stock – with the exception of our personal blog sites – those will remain on the Pi and become a personal project entirely outside of S-City Tech. This will allow us a wider choice of Linux distribution, more storage options, upgradability and more.

We’ll restore our main & client sites to their current states after the transfer which should prevent any data loss.

Leave a Reply