Recently reported computerized data corruption and potential loss due to hardware failures in the Cayman Islands government IT system is a serious issue that could affect the entire public service, Cayman’s information commissioner said last week.
While he lauded the government Computer Services Department and the Royal Cayman Islands Police Service for ongoing efforts to recover trillions of bytes worth of police data following hard drive crashes in October and March, Acting Information Commission Jan Liebaers said the government needs to take steps to ensure this does not happen again.
“I hope that some positive changes result from this,” said Mr. Liebaers, an archivist by trade. “[The] Computer Services Department should not let this happen again, and government should be very, very concerned about this, since it could happen to any of our data.”
Following an open records request by the Cayman Compass, the government Computer Services Department revealed that the computer file server used to store various data for police experienced at least five major hardware failures since February 2012.
The latest hardware failures, between October 2013 and March 2014, were responsible for a significant number of police records being corrupted – some of which have been unrecoverable, Computer Services Department officials confirmed this week.
Computer services records show that a controller card in the police data server at the Citrus Grove building in downtown George Town failed in February 2012. A hard drive in the server failed in August 2012, and another two hard drives failed in April 2013.
In October 2013, three more hard drives failed at the same time, causing storage of RCIPS files to “crash,” according to computer services officials. At that time, the server operating system was rebuilt and police data was restored from a tape backup.
Computer services information manager Rex Whittaker said that the total data contained on the RCIPS server – roughly 10.3 terabytes – had to be restored once the physical hard drives had been repaired. Ten terabytes of data would hold more than 100 million copies of this story if it were saved to a standard Microsoft Word program.
Initially, officials in the Ministry of Home Affairs, which has oversight responsibility for the police service, reported that 1.2 terabytes of data in the Citrus Grove hard drive had potentially been corrupted.
“However, since then, multiple restores have been completed by Computer Services Department to drop that figure down to the now-estimated 3.79 gigabyte figure [of corrupted files],” Mr. Whittaker said, adding that he was hopeful the remaining corrupted files could be restored.
“My main concern … was that reasonable efforts should be made to recover the data, and they appear to be doing that,” Mr. Liebaers said.
But there were additional managerial concerns about the preservation of computerized data in general that the information commissioner believes should be addressed.
“How is it that a backup is corrupted and nobody knows?” Mr. Liebaers asked. “Aren’t backups tested occasionally to make sure they actually contain real, usable data? Apparently that was not the case.” Mr. Whittaker said the Computer Services Department had found that a portion of the RCIPS server “did not get backed up.”
“The origin of this problem was traced back to the rebuilding of the server after the October 2013 crash, when the storage was divided into four logical data drives to effectively manage the RCIPS’s large volume of storage,” computer services officials reported. “After reviewing the backup logs, it was determined that the backup job for the RCIPS server was only manually updated with three of the four logical drives. So a portion of the server did not get backed up.”
To correct this problem in the future, computer services has implemented “more checks and balances” in its server change management and backup procedures to ensure multiple checks are done on backup systems by different individuals, he said.