Providers at risk for sloppy backup and migration practices

Loss of data during information system backups, upgrades to operating systems and databases, and while migrating data from one storage vendor to another—such as from EMC to NetApp—is common, and human factors are the primary cause.

That’s the view from Dave Logue, senior data recovery engineer at Kroll Ontrack, which offers data recovery, restoration, erasure and e-discovery services. For instance, the challenge with data storage is moving from one storage platform to another, but the task often fails because of poor preparation and taking shortcuts, he contends.

An example: Before migrating data from one source to another target platform, data from the first source should be fully backed up. “Information technology pros know this, but the challenge is knowing what to do and actually doing it,” Logue says.

Backups take a skill set to do and validate, but those doing the job often are not given enough time, so they cut corners. They may just complete a backup and move to another backup without checking the error report on the first backup—they assume that it worked. Looking at the error report “is a 10-second check if it is clean, and if it is not clean, those are the problems you want to address before you get in trouble.”

SQL databases are another source of data loss during backups. The SQL database locks files, so the backup may duplicate the data on the main hard drive, but not the files in the SQL database. This is common across all database backup vendors, who sell plug-in software to handle locked files, Logue notes.

Another major factor leading to data loss during backups and migrations is the continued siloed environments within healthcare organizations, he contends. Database and server personnel often work in different departments, so there isn’t good communication that details which tasks each are doing. “Make sure teams have a plan for what and how they want to back up—get the parties together.”

Inadequate bandwidth is another cause of data loss. Server administrators are given certain time periods to do a backup, such as from 11 p.m. to 2 a.m. But if the backup isn’t completed by 2 a.m., they have to try to finish it the next day.

Another problem that results in data loss is failure to check disks and tapes to make sure there is sufficient space remaining on them to handle all of the backups scheduled to be done. If a disk is full, it’s likely that some of the data assumed to be on it are not, and no one will know unless the error report is checked.

Logue offers the following tips for more successful data backups and migrations:

• Develop back-up and migration plans spelling out who does what. Engage your storage vendors—before starting any backup or migration, call the vendor and ask who can take a call, if necessary.
• Validate and test the backups and make sure applications are working after the backup is complete. Look at the error report.
• Have a rollback plan. If doing a backup on Friday with a four-hour window and it’s not done in time, go back through the migration plan and see why it failed. Have a “failback” plan on how to quickly reverse a data migration that didn’t work.
• Make sure that those doing backups are adequately trained—cross training of server, backup and storage administrators is even better. But focus on training in their individual duties, like checking the error report.
• Provide enough time for the task at hand. If the time window is too short, people start taking shortcuts. If an organization really needs five staff members who do backups or migrations and only has two, outsourcing some of the work to a third party some—such as an expert that just does data migrations, might be an option.
• End a backup or migration that is going poorly. “Don’t panic and stop pushing buttons,” Logue counsels. “Just stop. If you’re beating up the drive too much, you can do more damage than what is already done.”