Categories
Security Statements

Official Statement: Outage of March 20

The official statement regarding our 6 hour downtime in March 20, 2020.

Background

Here at S-City Tech, we’re proud of our commitment to data security. We began our first website with Wix, as you’re aware. However, Wix were completely unresponsive to any of our queries during our custom with them. This didn’t sit right with us, when we were always hyper-aware of the data that Wix had at it’s fingertips, waiting with the potential to be sold or stolen, and there to be next to nothing we could do about it for our customers if that should occur.

After an extended outage with Wix and zero answers to our three queries, we decided that we were no longer comfortable entrusting other companies, particularly Wix, with the data at the core of our business and subsequently our reputation for years to come. It was time to take responsibility for ourselves and host our own website.

However, as with all theories, it turned out much less simple. We ruled out using our office as the hosting location, due to poor network quality and the building manager not being willing to forward the internet ports to our subnet.

We also ruled out our own homes, as one founder lives in shared accommodation with metered networking and the other lives with family who declined permission, for fear of security as pubic IP addresses are easily obtainable and can be traced to an approximate physical location.

After asking around, a family member of one our founders agreed to provide access to their network and location for a small monthly fee. We weren’t comfortable with this due to some background with said family member that shall not be disclosed here, but our options were limited, to say the least.

After being unable to find another location, we chose to pay the monthly fee and use the space of the said family member, but we decided that additional precautions must be taken.

We chose to compliment our database encryption (all user data is contained in the database and only in the database) with full hardware-based LUKS encryption on the disk, using a keyfile and a recovery password. The keyfile allowed the server to boot up on it’s own, without verification – very important for off-site server management, while the backup password could be used to unlock the data in the absence of the keyfile, but required direct input via a keyboard during the boot process.

This new layer of encryption not only encrypted the database a second time, but also encrypted backup files a second time and included sensitive config info that was not otherwise encrypted, as protection in the event of attempted physical extraction. We also set out a procedure that allowed us to lockdown the server, preventing access to the encrypted data until we manually intervened:

  1. Remotely connect to the server’s Command Line Interface (CLI) securely
  2. Delete the original keyfile from the system
  3. Issue a shutdown command to the server

Our target for this procedure is 90 seconds, but it can often be done in less than 60.

What happened?

Due to a typical family problem plus an unstable family member, it was decided that we needed to monitor the server logs for attempted digital intrusion. This decision was made shortly before 16:15 and the monitoring was ongoing from that time.

At 18:28, the family member who hosted the server was making threats against the founder to which he was related, and it was decided that the server needed to be locked down. The commands were issued at 18:36.

These procedures proved effective. The boot process was halted due to the lack of the keyfile, and the recovery password is guarded so closely that it is only even known to one of us. No attempt appears to have been made to attempt to physically extract the data, however due to one of us being on the way to collect the server immediately after issuing the lockdown commands, it’s likely the family member didn’t want to risk the extraction attempt due to time constraints.

Even if an attempt was made, the only unencrypted files on the server are the ones for the software on the system. All website files, sensitive configuration files, database information and backup files are encrypted. For example, the webserver software that we use, as it comes when first installed, isn’t encrypted, but the actual HTML, CSS and PHP files that make up our specific website, are encrypted.

Effect on website and/or user data

None. All server data is accounted for, as it was at the moment of the lockdown starting. There’s no backup restoration involved, just unlock the encryption and continue (figuratively speaking, see below).

Our plans to prevent a recurrence

Unfortunately, the realities of the world today mean that some day, we will no doubt be forced to lockdown our server again to protect it’s data. We much prefer that our website be down for a few hours over the sensitive data of our customers being compromised.

Due to the fact the the procedure was effective in protecting the data, no changes are strictly necessary. There has been ABSOLUTELY NO compromise to any data on our server. There is no question, we are certain that all data of any importance is secure.

Nonetheless, we’re talking a few measures, just to be on the safe side. Firstly, we’ve rotated the keyfile and recovery password (just to be safe in case of brute-force guessing attacks). Secondly, we’re making it a point to monitor the logs of our server a little more closely, as there were an alarming number of firewall blocked connections from an unknown IP address that we’re currently investigating, but there is no security risk there as all connections from the suspicious IP are being blocked by our firewall which is only our very first line of defense, and there are many more defenses beyond that.

Also, as a matter of course, the server has been moved to a much safer temporary location, pending us finding a permanent location for the server. We’ll also be much more selective with regards to our new permanent location.

Summary

An attempted physical extraction on our server was foiled by our lockdown procedure, which caused an outage between 18:36 and 23:55. We pride ourselves on our commitment to data security, which proved effective today.

We’ve confirmed that no user data or general website data was affected by the lockdown, and normal operations will continue as expected.

As a direct result of this incident, our attention was drawn to failed connection attempts on port 20, twice per hour, every hour, from as far back as our logs go (Saturday 14th March) from a specific IP address.

All of these connections are confirmed to have been blocked by the most basic of our defense measures, but we’re investigating nonetheless.

We’re implementing a few basic measures to help prevent a recurrence, and our server has been moved to a temporary location to protect it’s data.

Closing statement

We deeply apologise for placing our server in an environment where it was at potential risk. No amount of additional measures implemented from the get-go make that decision less erroneous, and we are going to be far more selective of our server’s future locations.

We can, however, rest a little easier in the absolute knowledge that our procedure for securing the server and it’s data at the drop of a hat is so effective, should the need arise again.

We’ll provide updates with regards to the ongoing investigation about the suspicious blocked connections as more info becomes available.

One reply on “Official Statement: Outage of March 20”

Leave a Reply