Why Data Protection? — Real Life Data Disaster Stories

Nearly 50% of companies with a data protection plan in place execute it in any given year.

— 2008 Continuity Insights and KPMG Advisory Services Business Continuity Management Benchmarking Report

“But, we already back up our data.”

One business I worked with did have a simple backup plan in place that included backing up all of their critical data.  Their data protection procedures, however, did not include backing up their operating system configuration nor specific installs for customized applications.  Making matters worse, all critical applications and data were centralized on one server — including email, phone communications, customer contact and support records, product designs and accounting data.  Any problem with this server would affect all services. 

One day I was called in to help with a disk error that I discovered quickly was actually a failing disk system.  The application drives were deemed unrecoverable.  We had no choice but to rebuild the array from scratch, recover the critical data, and manually rebuild the operating system and application installations.

Even though this business had a basic backup process in place it took two full weeks and well over 200 hours of emergency rate consulting to get all of their key services back up and running!

Of course it doesn’t have to be this way.  With a comprehensive data protection plan in place, this company could have been back up and running within hours of the disk system failure — all for a fraction of the cost of recovery and downtime.

Lessons Learned √     Simply backing up files, while a good start, is not enough to protect your business from data loss and downtime.
√     Well-meaning employees with other expertise and duties cannot be expected to protect critical data properly without outside help.
√     Recovery plans must be validated with live testing to ensure they work when needed.

 

 

“We use RAID, isn’t that enough?”

A small photography/videography studio in the greater Seattle area had only a few photographers but was very modern in that all of their content was digital.  Over the years they had amassed several terabytes of video and photos, which they relied on for residual income from selling additional copies and prints. 

One day, a flashing red light — on the small network storage device the company used to store ALL of their data — indicated there was a problem.  The owner of the company contacted the maker of the device and discovered that one drive in the system had failed.  Fortunately, the system was protected by RAID (Redundant Array of Independent Disks) and thus the failure of one drive did not prevent access to data on the device.  A major caveat for RAID, however, is that when one drive fails, the likelihood of another drive failing is very high; and when a second drive fails, all data is lost. Therefore, it is critical to replace the failed drive immediately.

Unfortunately for this business owner, when Support walked him through replacing the damaged drive a few days later, he unknowingly removed and replaced the wrong drive – and lost all the data in the array!  Support then informed the business owner that they had no way to recover from this type of mistake and, if he had no backup, they suggested he contact a reputable data recovery firm.

The owner did contact a data recovery firm and luckily they were able to recover the array … after several days of work and shipping the device back and forth; the studio being without access to their data for an entire business week; and paying $25,000 in recovery fees! That fee might sound excessive, but the process of recovering a failed RAID array is very labor-intensive work, requiring highly skilled individuals and expensive tools.

This is a compelling example of why RAID is not a backup plan in and of itself.

If this business had proactively set up a data protection plan, their very costly mistake in replacing the failed drive would have been just an inexpensive, minor inconvenience:  a few hours without access to their data while it was recovered.

Lessons Learned √     RAID is not a sufficient form of data protection.
√     The cost of protecting data is much less than the costs of recovery.