It's Not Our Fault

I follow the Internet Storm Center's diary. Recently one of the entries related a situation with a personal NAS (Network Attached Storage) with terabytes of data. It was configured with RAID5. The NAS vendor offered a cloud backup service that he used.

He had a detailed backup plan consisting of:

• a daily backup to a cloud storage provider
• a monthly backup to an external disk (physically stored away from the source)
• a file restore test performed every month (ex: restore file ‘x' backup at time ‘t’)

While this diary is about a personal situation the lessons apply to enterprise as well.

The power failed in his area and the NAS crashed hard. He reloaded the operating system and rebuild the RAID5.

No data was lost.

But then he had to “relink” the existing cloud backup with the new backup task on the NAS. That  failed with strange error messages saying that some files were not found.

After the normal "Turn it off and back on" with no success the vendor responded:

Thank you to try to log on your cloud service console to check if your files are available. If they are not available, please contact your cloud service support to get more help. We already notified them about this issue and we received a lot of complaints from other customers who are facing the same issue. You should try to see with them how to recover your files, if possible...

Remember that "your cloud service" was arranged by the NAS vendor.

Yep, his cloud backup was lost (1.5TB of data).

What if that had been YOUR enterprise data?

Have a plan. Have a backup to that plan. Test it. Test it again.