cpanel04 Downtime

[AndrewT]

cpanel04 is currently offline and we're trying to find an easy solution to get it back online quickly but this may end up being a fairly lengthy downtime, at this point we simply can't say for certain. I'll update this thread as the issue progresses.

[AndrewT]

Just to clarify, customer data is perfectly fine. We're just having lots of trouble with the kernels and other boot configurations. This may require the immediate upgrade of the server to one of our newer systems if we are unable to get it online as-is.

[AndrewT]

The server is now back online. I sincerely apologize for the inconvenience that this undoubtedly caused, this will be placed at the top of our upgrade list and will be upgraded within the next week or so to resolve any remaining issues permanently.

[AndrewT]

Once again we are having similar problems with this server and we are trying to get them sorted out as quickly as possible. I will keep this thread updated.

[AndrewT]

Unfortunately, to be 100% honest, this downtime is going to last several more hours. We've decided to just go ahead and upgrade this server to our new system immediatley. This will solve the current issues and make sure that it won't happen again like it has today, in addition to the huge increase in performance that will be had afterwards.

I would estimate this being done sometime this evening by the time we get the new server built, the drive image loaded onto it, and all data copied between the new drive and the old one.

I do apologize for putting everyone through this, but I think this is the best route at this point to ensure that no more problems will occur.

[AndrewT]

We are now beginning the restoration process for all accounts. This is being done from the last know good backups to make sure that any previous problems aren't transfered over here in the process, these backups are from 4/18/04. MySQL databases, however, will be 100% live as they were just before the server went down. Right now I cannot provide any ETA until I get a feel for how fast (or slow) this process is going to go. I would guess at least 3 hours for all accounts to be restored (in alphabetical order by username). Reseller (WHM) access will be restored last.

[AndrewT]

Just for an update, the "O" usernames just began transfering. This process has severely slowed down due to the number of larger accounts being restored now, I would hope that this would be done in the next 4 hours or so. I do apologize for the inconvenience once again.

[AndrewT]

We just began on the "S's". I had to just go ahead and block web/e-mail traffic for now as that was causing severe delays in getting these accounts restored in any sort of timely fashion. These should be done now by about 3PM I would imagine since I have disable the flow of traffic until they are completed.

[AndrewT]

Well unfortunately this is going to take a while longer than 3PM. I'm done giving estimates but we are a fair amount through the "T's" at this point (mind you that T and S are fairly popular).

[AndrewT]

Everything has now been completed except for the allocation of dedicated IP addresses to domains that had them previously, which will be done within the next couple of hours. Other than that if you are having problems please submit a trouble ticket. If you have accounts missing from WHM submit a trouble ticket with the username of each.

[AndrewT]

Dedicated IP's have now been re-assigned to those domains that had them previously. You will need to re-install SSL certificates.

[AndrewT]

Also, it is perfectly normal for you to possibly have domains listed in your account that you terminated long ago. Just terminate them again if they are not needed.

[AndrewT]

Several of you have submitted tickets regarding the slowness of cpanel04 right now, this is because of the huge amount of traffic that it is receiving from people trying to catch back up on e-mail, etc. It will likely take another day or two before it all calms down especially as we continue to restore a handful of missing accounts.

[AndrewT]

There's currently a hardware problem with the server that we're trying to pinpoint exactly which is causing the server crashes every 6-12 hours which require a reboot. There may be a time in which we will take it offline to run some further tests but it will not last long and I will update this thread as I know more.

[AndrewT]

We will be swapping out the server chasis on 4/24 at 1AM CST. There may be up to an hour of downtime but hopefully it will not last that long. This will eliminate virtually any possibility of hardware problems with the exception of the hard drives.