9/14/07 - cpanel57 Outage

[AndrewT]

cpanel57 experienced severe corruption of the file system earlier this morning. We attempted to recover it but it is well beyond usable at this point and refuses to boot. We're in the process of adding new drives to the server and will be reloading the OS onto these new drives. We will then be restoring all accounts from system backups that were generated last Saturday (9/8/07). Updates as to the status of this process will be posted to this forum thread as soon as we have them.

I apologize for the inconvenience that this has undoubtedly caused. Unfortunately these things do happen, although rarely, and really cannot be controlled.

[AndrewT]

I apologize for the lack of updates thus far.

cpanel57 is now fully setup with the new drives. All WHM users have already been restored along with a few domain accounts. Everything is fully operational as the accounts are restored with the exception of sending e-mail (SMTP) and receiving new e-mail. These will be disabled until we can get all of the domains added to the mail server so that the server is not rejecting all e-mail for domains that haven't been restored yet.

We're restoring the rest of the domain accounts as quickly as possible. If you have a very urgent need to have a particular domain restored then please submit a ticket with the domain's username and we can see about getting it restored a bit earlier. Otherwise you can login to your WHM and your accounts that have been restored so far will be displayed under "List Accounts".

Note: If you paid for our daily backup service, your daily backups have already been restored.

[AndrewT]

Approximately 45% of the accounts have been restored. All restores should be completed by no later than 11:00AM.

[AndrewT]

All accounts have now been restored and e-mail services are fully enabled. During this downtime, e-mail servers that were sending to ours would have kept the e-mails in their own mail queues and will regularly attempt redelivery whenever possible, generally within 24 hours max.

Domains that previously had a dedicated IP were just re-assigned the same dedicated IP. As a result, the domain may temporarily appear to be unavailable to you until your ISP updates their DNS. You'll know this is the case if your domain has a dedicated IP and is now displaying "cpanel57.gzo.com". You will have to wait for your ISP's DNS to update.

SSL certificates do need to be reinstalled. If you purchased one from us simply submit a ticket and we can take care of it for you.

Custom DNS and MX records need to be re-entered into the system.

If you have any further questions/problems please submit a ticket.

[AndrewT]

Between 4PM and 6PM CDT today we'll be taking this server offline to remove the last of the corrupted disks. Should be no more than 10 minutes of downtime.

[AndrewT]

This was completed approximately 45 minutes ago.