Tweet
Hello,
First and foremost, I want to apologize once again for the downtime that has occurred today. As you know very well, this is not normal for us and we strive very hard to avoid situations like this.
This morning's extended downtime was the result of a few very unfortunate failures. At approximately 3:00 AM CDT logs indicate that cpanel78 began experiencing file system errors. At this point we don't know the exact cause of this but it appears to be due to a failed SSD not having been dropped by the RAID controller properly. Because of this, the corruption was able to spread to the RAID 1 mirror. Eventually the RAID controller did drop the failed SSD and rebuild the RAID 1 array with one of the hot spares present in the server but it was too late and the corruption had already occurred.
Our initial plan was to attempt to repair the corrupted files but these turned out to be far more widespread than we expected. This unfortunately left us with only one option; restoring from backup. We proceeded to restore this server's root partition from yesterday's backup. As a result of this, MySQL databases and any other data not residing in a user's home directory will have been reverted to what it was on May 31st at approximately 8 AM CDT. The home partition was not affected (this includes each account's email, public_html data, etc.). We will be following up with a much more thorough investigation of what transpired and will be working with our hardware vendors to hopefully eliminate this from ever occurring again.
If you experience any further problems please don't hesitate to submit a ticket so that we may investigate them. We greatly appreciate your patience and understanding.
Best Regards,
Andrew Thornton
President - Dathorn, Inc.
First and foremost, I want to apologize once again for the downtime that has occurred today. As you know very well, this is not normal for us and we strive very hard to avoid situations like this.
This morning's extended downtime was the result of a few very unfortunate failures. At approximately 3:00 AM CDT logs indicate that cpanel78 began experiencing file system errors. At this point we don't know the exact cause of this but it appears to be due to a failed SSD not having been dropped by the RAID controller properly. Because of this, the corruption was able to spread to the RAID 1 mirror. Eventually the RAID controller did drop the failed SSD and rebuild the RAID 1 array with one of the hot spares present in the server but it was too late and the corruption had already occurred.
Our initial plan was to attempt to repair the corrupted files but these turned out to be far more widespread than we expected. This unfortunately left us with only one option; restoring from backup. We proceeded to restore this server's root partition from yesterday's backup. As a result of this, MySQL databases and any other data not residing in a user's home directory will have been reverted to what it was on May 31st at approximately 8 AM CDT. The home partition was not affected (this includes each account's email, public_html data, etc.). We will be following up with a much more thorough investigation of what transpired and will be working with our hardware vendors to hopefully eliminate this from ever occurring again.
If you experience any further problems please don't hesitate to submit a ticket so that we may investigate them. We greatly appreciate your patience and understanding.
Best Regards,
Andrew Thornton
President - Dathorn, Inc.
Comment