When cloud-based source code hosting provider Code Spaces announced it was shutting up shop permanently last month following a combination Distributed Denial of Service (DDoS), hacking and ransom attack. The reaction of cloud nay-sayers was all too predictable.
That's what happens when you put data into the cloud, they cried. Actually, they were wrong: that's what happens when management misunderstands the process of business continuity.
Let's take a look at what happened to Code Spaces as best we can tell and figure out from that how you can avoid being a cloud-based combination attack fatality; as specifics regarding the nature of the breach are still in very short supply.
On 17 June, Code Spaces found itself subject to a DDoS attack coupled with a ransom demand involving a 'large fee' in order to make it go away.
Losing control
Upon investigation it soon became apparent that Code Spaces had also lost control of the Amazon EC2 control panel. This meant that not only had the attacker prevented access to services but also had the ability to delete data in the cloud. In a statement at the time, Code Spaces says it managed to get panel access back but "not before he had removed all EBS snapshots, S3 buckets, all AMIs, some EBS instances and several machine instances" or, in other words, wreaked havoc to data and backups such as they were.
That was enough for the company to declare it had "no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us" which, frankly, doesn't sound like it will be much.
Interestingly, just the previous week other online services including Evernote and Feedly were apparently subject to similar DDoS coupled with an extortion demand attacks. Although they suffered prolonged downtime, neither is understood to have paid the ransom and both managed to restore normal service. So where did Code Spaces go wrong and what lessons can you learn?
The main differentiator here would appear to be the attacker had access to that EC2 control panel, which gave them much more leverage. But, as I've said before, appearances can be deceptive and the fatal flaw at play would seem more likely to have been poor decisions made by Code Spaces.
What happened to the business continuity plan or the disaster recovery plan or even the breach response strategy? This includes a DDoS attack plan, which recent research (by Forrester Research) suggests is something 43 per cent of enterprise do not have.
Lesson learned: always have a disaster recovery plan, always rehearse it and always stick to it
What happened to the number one rule of data backups which is not to keep all your eggs in one basket? And that's before we mention the number two rule: not to keep your baskets in the same barn. Cloud Spaces would seem to have misunderstood what backups are.
There's an old adage amongst those of us who have been around the block enough times to have seen many a data disaster that was disastrously handled: it's not a backup if it's in the same building, it's a copy. That applies whether you define 'building' literally or to mean system, network or even cloud.
Separation, and that means real separation not virtual, is key to backup success within a business continuity plan. If Cloud Spaces had ensured that there was proper redundancy and separation built into the strategy, as far as there was one, then the attacker would not have been able to delete the backups and 'offsite' (but still accessible from the single control panel) cloud backups as they did.
Lesson learned: always backup to distinct services and sites: secure isolation of data is the key
Of course, we have no way of knowing for sure at the moment but if, as seems likely, Cloud Spaces tried to fire fight on its own, there's another procedural error right there. The first thing on the breach response strategy list should have been - involve AWS support immediately.
Even if the DDoS attack could not have been stopped, then access to the control panel could have been suspended by locking the admin account. Even if data had already been 'deleted' in the console by the attacker, surely such a call might have meant that Amazon could have recovered some of the incremental EBS snapshots (courtesy of bucket versioning) before they were actually, physically, deleted from S3?
Lesson learned: always work with the provider, and start working from minute one
How did the attacker get access to the control panel in the first place? Again, we cannot be sure but it could be anything from social engineering through to phishing or malware.
There are two important points raised by this question: first is to ensure that endpoint and network protection is enforced to minimise the credential stealing risk,. The second one is to implement multi-factor authentication to minimise the risk of stolen credentials being of any real use.
We don't know what protection Code Spaces had, but you should be seriously looking at a network firewall and Intrusion Prevention System (IPS) coupled with anti-malware software. Amazon, like most cloud providers, will offer some kind of multi-factor authentication to protect access to control panels.
If, as would appear to be the case, multi-factor authentication was not being implemented here then security is weakened to an unacceptable level. This is as much a failure by the provider as the customer; it would be far better to make such access control the default. Let's face it, multi-factor systems are no longer expensive to procure, rollout nor maintain. Their usage by now should be a no-brainer.
Lesson learned: don't make it easy to hand over the keys to your kingdom.
Ultimately, Code Spaces could and should have survived this attack. It may have been wounded for a few days, a week or two even, but not dead and buried. The combination of attack types here made the wounds inflicted all the more dangerous, but the apparent lack of a coherent and considered response proved to be the real fatal blow. Ensuring your business has the proper armour to defend against such attacks is not rocket science and it's not too costly to consider; especially when you consider the true cost of not doing so...
