Datacenter-related question

[ZYV]

Hi everybody!

That might be much of a personal question, but anyways I would like to hear from Andrew on this matter if that's even possible. Ages ago, if I remember correctly, Dathorn was renting some servers at ServerMatrix. Later on, they completely migrated to ThePlanet. Recently, I've been investigating into the reliable DC providers and figured out that ThePlanet merged with EV1 which was pretty much criticized by the community. Then I found SoftLayer which is, as far as I understand, driven by the former TP executives and aims to bring this business to the new level.

What’s your attitude against SoftLayer and are there any plans on trying them out? The reason I ask is the recent power outage. I find it hilarious that super-equipped DC with multiple diesel generators, UPS batteries etc. can’t handle such problems and that leads to the unclean server shutdowns. What are all those things for, if it doesn’t work when needed?

Z.

[AndrewT]

I regret that we have not fully explained our situation. I kept putting this off and was going to incorporate it more into our new website but since we're on the topic, I might as well bring it up.

The Planet (EV1) is a huge mess. Completely horrible now in my opinion. There are many reasons behind all of this but I'd rather not get into them. Simply put, their customer service, even for larger customers such as us, is awful. As a result of this, we did venture a bit into Softlayer (also located in Dallas for those that don't know). I personally worked with a lot of people that are now at Softlayer when The Planet was in its younger years. These are all great people that made The Planet what it once was.

After a bunch of testing with some local customers' dedicated servers at SL we decided to place our first shared server over there, which is cpanel54. Since then, our shared servers have been put up at SL (cpanel54-cpanel58). SL is far better than The Planet in more ways than I care to list. It is a complete night and day difference.

Having clarified this, the power outage was at SL, and not The Planet. Let me give you a brief run down of what happened:

On Friday Dallas had a lot of thunderstorms roll through including at least one tornado near by. Utility power in parts of Dallas is notorious for having glitches and outages during strong storms. As a result, SL switched to generator power to help ensure that no problems would arise. (This was also a great real world scenario to see that everything was working over an extended period.) Finally on Saturday night they were going to switch back to utility power and in doing so a 2500amp main breaker failed. This breaker controls generator and utility power into parts of their datacenter. When it failed, all they had to run on was battery power which lasted roughly 30 minutes. Afterwards the inevitable occurred and there were many racks left without power. Replacing a breaker like this is far from an easy and simple task. They had a spare onsite but it's still a replacement process that can easily take a few hours. Once it was replaced, power was restored without any problems. However, additional time was taken to power on various sections and racks a few at a time to prevent a power surge from causing even further problems.

I hope that makes sense. It was certainly much more than just a simple power outage because you are exactly right in that this is why datacenters have all of this redundancy. They've yet to clarify how they plan on preventing this from happening again which I am interested in hearing. Regardless, power transfers occur all the time and you would never notice or realize it. But it just takes one failure like this to create a large problem.

[ZYV]

Wow! Thanks a lot, Andrew, for an exhaustive explanation. Of course that makes sense to me and I just didn’t know that you are actually working with SL, since my accounts are on the servers below cpanel53. And it makes another good reason to choose them for a dedicated server if they serve companies like Dathorn.

Concerning the power outage in question, you reply left no blank spots on the situation which was much more serious then I supposed it to be. I hope they would learn for this “real world” redundancy testing experience and we would have even less outages in future.

[AndrewT]

Another thing that most don't know about yet is that we are now offering dedicated servers to existing customers on a per quote basis. We have several customers that have already taken us up on this offer. Server management is also available in various forms. The most popular option is to have us manage and monitor your server just like we do our own and you still get WHM reseller access to manage your accounts. Previously we were putting these up at The Planet but we're now using SL for these as well.

In the 8+ months that we've been using them, this latest outage is the only real problem that we've had. Through my working with Lance (the SL CEO) while he was at The Planet, I have no doubt that SL will only improve upon something that is already great. If the current trends continue, we'll continue to use SL when we can.

[powvex]

the planet + ev1 merge was disaster in my opinion. they are now far beyond industry now, from cost to customers services. we also are moving servers from them to other major dedicated providers like dalas colo.

but another disaster is SL, as we had a very bad experience with them, specially their network outages over Asia networks. their connection to Asia and Europe is not well enough to rely on.

so I hereby ask Andrew; If you are going to put more of your servers in SL, please please inform us at least a month before you start transfer, so we can have time to move elsewhere.

[Grunfeld]

I find this thread very interesting having kicked around here for I think 4 years now I have been so pleased with the service and support. There has been minimal downtime connection issues etc....

I have to support Andrew on his decisions,when he presents something to his clients here it's usally bang dead on and well thought out ....

I suspect the changeover/upgrade for the older servers ( cpanel 17) will take a bit longer however I suspect it will go smoothly as it has in the past as shown with other upgrades there might onf been some issues however in the big scheme of things pretty good ....

The merger with The Planet/EV1 has certainly caused many ripples in the industry , however based on connectivity with the sesrver I am on, the merger hasn't nor should it affect service to end clients

[AndrewT]

but another disaster is SL, as we had a very bad experience with them, specially their network outages over Asia networks. their connection to Asia and Europe is not well enough to rely on.

Everyone has a bad experience at some point or another with any given company because as hard as we all try, no one is perfect. A very large percentage of our customers are outside of the U.S. and SL has not had any such network problems since we've been with them nor have any international customers reported any problems. Whatever you experienced previously, I have no reason to believe that it has been occurring any time recently. We simply wouldn't be putting servers there if it wasn't top notch.

Upgrading our older servers (cpanel13-cpanel36) is still the main priority; it's just taking a bit longer than expected.