SANS ISC: Reminder: Beware of the "Cloud" - SANS Internet Storm Center

• SANS Site Network
  • Current Site
  • Internet Storm Center
  • Other SANS Sites Help
  • Graduate Degree Programs
  • Security Training
  • Security Certification
  • Security Awareness Training
  • Penetration Testing
  • Industrial Control Systems
  • Cyber Defense Foundations
  • DFIR
  • Software Security
  • Government OnSite Training

SANS ISC InfoSec Forums

Today, when you buy a product, there are chances that it will be “connected” and use cloud services for, at least, one of its features. I’d like to tell you a bad story that I had this week. Just to raise your awareness... I won’t mention any product or service because the same story could append with many alternative solutions and my goal is not to blame them.

I’ve multiple NAS at home with terabytes of data. You can imagine that the backup process for such amount of data is not easy. My backup plan is:

• a daily backup to a cloud storage provider
• a monthly backup to an external disk (physically stored away from the source)
• a file restore test performed every month (ex: restore file ‘x' backup at time ‘t’)

Last week, our city suffered from a major power outage and my UPS was unable to keep all the devices online. As a result, an unexpected shutdown of one NAS. When the power was restored, you can guess what happened: It did not boot at all: The OS was corrupted. After several attempts, I successfully restored a fresh operating system and, lucky me, the data were not affected. I started a rebuild the RAID5 and, a few hours later, I had access to all the data! Phew!

The next step was to reconfigure my backup configuration and “relink” the existing online backup with the new backup task. The procedure is described in the product documentation and looks very easy. I had all the required information (the most important was the encryption key). Except that it failed with strange error messages saying that some files were not found. After several unsuccessful attempts, I contacted the NAS manufacturer support and asked for some help. Followed the classic exchange of boring emails like “Are you running the latest version?” or “Did you turn it off and on again?”. Yesterday, I received the final reply (anonymised and simplified):

Thank you to try to log on your cloud service console to check if your files are available. If they are not available, please contact your cloud service support to get more help. We already notified them about this issue and we received a lot of complaints from other customers who are facing the same issue. You should try to see with them how to recover your files, if possible...

To read between the lines: "It's not our fault, check with the other party". I’m waiting for more feedback but it looks that my backup is lost (1.5TB of data). Hopefully, I did not lose data but I can’t imagine the disaster if I had to restore my complete backup from the cloud service. The conclusion of this story: Do NOT rely on cloud services only and make multiple backups. Keep in mind that, once you sent your data to the cloud, you completely lose control of them! Stay safe!

Xavier Mertens (@xme)
ISC Handler - Freelance Security Consultant
PGP Key

Can your UPS issue a controlled shutdown?

Never ever rely on a potentially fatal single point of failure for anything you consider to be critical. This goes for your backups, your supply of cough drops, your paper document storage, your eyeglasses/lenses, your supplier for CPU chips, anything. That concept should be deeply embedded in your backup philosophy and design.

{o.o} Just sayin'.

Quoting Urd:Can your UPS issue a controlled shutdown?

Yes and it is configured like this but the UPS decided to shutdown at 20% of the battery capacity. Another lesson learned! :)

Can I say this, I have never had a problem with gdrive, and there is an insanely easy to implement Linux bash cli available. This makes backups via Cron job easy peasy... But yes I have dealt with many a drive mishaps and many a hours digging through photorec output to get the crucial things...

Sorry about that, hope everything worked out