1/26/08 - cpanel61 downtime

[AndrewT]

At approximately 9:45PM CST cpanel61 experienced a kernel panic due to ext3 file system errors. A manual fsck was required on reboot and as it was running we discovered that the home file system was completely destroyed. Here is what we are in the process of doing:

1. Copying backups to an alternative location for additional redundancy on this important data.

2. Although no hardware failure has been indicated, we are going to go ahead and replace the disks in the RAID array just to be safe.

3. We will be reloading the OS and setting up the server once again (cPanel/WHM/etc.).

4. We will be restoring all accounts from system backups that were generated on January 26th starting at approximately 2:00AM CST.

This entire process will take a while to complete. I will continue to update this thread as we progress. Please do not submit tickets asking for updates.

Unfortunately this sort of problem will inevitably arise from time to time (albeit fairly rarely). There is really no way to fully prevent this from occurring as the drives have shown no signs of hardware problems at all. Fortunately we do have the system backups available that were generated just before this incident occurred so the overall impact will be much less than it could have been.

I do apologize for the inconvenience that this has undoubtedly caused. We're working as quickly as we can to get the server back online and fully functional with all accounts restored.

[AndrewT]

We're currently on step 3 of the process that I outlined above. We hope to begin restoring accounts by 8:00AM CST at the latest if everything goes as planned.

[AndrewT]

All WHM users have been restored and you should now be able to access WHM just like before. If you go to "List Accounts" in WHM you will be able to see a list of your accounts that have been restored thus far. It will still take several more hours for us to restore the remaining accounts. I will continue to update this thread with progress information. Please do not recreate accounts for your domains as this will prevent the backups for them from being restored.

Please note that SMTP and incoming e-mail is being blocked until all accounts have been restored. This is to prevent the server from bouncing e-mails that are sent to accounts that have yet to be restored.

Once all restores have been completed we will be re-assigning the dedicated IP addresses to the domains that had them. At this point you may need to re-install your SSL certificates. If you purchased one from us simply submit a ticket and we can get it installed for you.

[AndrewT]

The restores are still in progress and quite a few accounts have already been restored. We hope to have most accounts restored by 6:00PM CST.

[AndrewT]

Approximately 88% of the accounts have been restored at this point. The remaining accounts (some of which are fairly large) should be finished restoring by late this evening.

[AndrewT]

All accounts have now been restored.

Any domains that previously had dedicated IP addresses have been re-assigned the very same dedicated IP as before. The domain may appear to be unavailable until your ISP updates their DNS to the new IP address. If you had an SSL certificate installed you may have to re-install it via WHM. If you purchased one from us simply submit a ticket and we can re-install it for you.

SMTP and incoming e-mail are fully enabled now as well.

For those of you using our Postini service, the Postini backend is currently offline. As a result, the Postini servers will continue to queue your incoming e-mail until their backend comes back online so that we can trigger it to resume normal delivery to cpanel61.

If you have any further problems or questions please submit a ticket with specific details.