Emergency cpanel04 Upgrade on 9/6/2006

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • AndrewT
    Administrator
    • Mar 2004
    • 3653

    Emergency cpanel04 Upgrade on 9/6/2006

    ATTN ALL customers located on cpanel04, please read and understand this entire thread thoroughly.

    At approximately noon (CST) on September 6th, cpanel04 will begin the upgrade to our new platform. More information on this can be found here: http://forums.dathorn.com/showthread.php?t=2094

    Instead of being on cpanel04, users will now be on cpanel52.

    Server: cpanel52.gzo.com - 67.19.41.146
    DNS #1: dns103.gzo.com - 67.19.41.148
    DNS #2: dns104.gzo.com - 67.19.40.72

    IMPORTANT
    Anyone that has custom nameservers or that has used the cpanel04 DNS IP addresses (64.5.36.128 and 64.5.36.129) for anything will need to update their entries to the new nameservers and IPs listed above.

    To avoid all possible downtime on your domains, do not make any DNS changes until we have confirmed here in the forums that the cpanel04 account transfers have been completed.

    Those that are using the cpanel04 gzo.com nameservers (dns7/dns8) without IPs, do not need to change anything. They will be updated to the new IPs automatically.

    Beginning around noon on September 6th, we also advise all users to disable anything involving MySQL updates (order forms, forums, CMS, etc.) until it is running on the new servers. This will avoid updates being lost during the transfer and ending up with a database that is a few hours old. If this is not an option, you can leave everything active, cpanel04 will still be online for a few days following the upgrade. But you will be responsible for transfering any database updates that might be needed.

    Once again, if you have any questions or comments, please post in the forum thread that is linked above.
  • AndrewT
    Administrator
    • Mar 2004
    • 3653

    #2
    Note that during this upgrade any domain usernames that have dashes or underscores in them will be renamed to exclude these symbols. Customer usernames with dashes in them will remain the same.

    Comment

    • AndrewT
      Administrator
      • Mar 2004
      • 3653

      #3
      At approximately 4:08PM CST cpanel04 stopped responding completely. By this time we had only transfered the actual reseller logins and a handful of domains.

      We've been investigating this incident and so far it appears that the RAID card in the machine experienced a severe failure which corrupted both hard drives that were in the RAID 1 array. Neither of them will boot. At this time we're unable to determine if data can be retreived from these drives. We're in the process of hooking them up to another machine as slave drives to see if we can even read data off of them. Once we get a chance to look into this further I will be able to provide additional information. Unfortuantely, if you are not paying for our backup service, your data may be lost. We should know for sure by later this evening.

      Currently all cpanel04 customers should be able to login to WHM on cpanel52. I recommend that you change the DNS for your domains to cpanel52 and recreate your accounts on cpanel52 with the exact same usernames as they were on cpanel04. You should then begin recreating e-mail accounts and restoring any data that you might have. Even if data is retreivable from cpanel04, the accounts will have to be recreated in some manner on cpanel52.

      If you need a list of the accounts that you had setup in WHM on cpanel04, we can provide you with this, simply submit a trouble ticket requesting it.

      For those of you that are paying for our backup service, please recreate your accounts as outlined above and submit a trouble ticket requesting that we restore the data on the accounts. Please be sure to specify the usernames in this request so that we may restore them as quickly as possible.

      Please submit a trouble ticket with any questions/concerns/comments and we will address them as quickly as possible.

      I would like to apologize to everyone that this might involve. Certainly there is no one that wants something like this to occur. Hardware failure is simply a fact of life. On this very note, we no longer use these RAID cards in our new servers (which includes cpanel52). However, this should stress the importance of maintaing backups of some kind at all times.

      Comment

      • AndrewT
        Administrator
        • Mar 2004
        • 3653

        #4
        This has been asked a couple of times already so I thought I'd mention it here. If you have a full cPanel backup of a domain you can simply upload it to the server and provide us with the location of it, we can then restore it for you.

        Comment

        • AndrewT
          Administrator
          • Mar 2004
          • 3653

          #5
          Just to clarify, dns7.gzo.com and dns8.gzo.com have already been pointed to cpanel52. If you are using those nameservers you do not need to make any changes.

          Comment

          • AndrewT
            Administrator
            • Mar 2004
            • 3653

            #6
            We've been able to retreive all MySQL databases from the cpanel04 server and they have now all been restored on cpanel52. If you notice any problems with this please submit a trouble ticket and let us know. So far they all look okay but with so many databases its hard for us to know for certain.

            Comment

            • AndrewT
              Administrator
              • Mar 2004
              • 3653

              #7
              We're now attempting to retreive all user directories from the cpanel04 drives. I cannot make any guarantees as to how this will turn out. This process will take several hours to complete given that there is over 60GB of data to transfer from a hard drive that wants to be put out of its misery. I'll continue to update this thread as the situation develops.

              Comment

              • AndrewT
                Administrator
                • Mar 2004
                • 3653

                #8
                So far we've recovered some of the data from the home directories (13GB) and we're still working on retreiving everything that remains.

                Comment

                • AndrewT
                  Administrator
                  • Mar 2004
                  • 3653

                  #9
                  We're still continuing to transfer data off of the old cpanel04 drives. So far this is working properly although it is a very slow process given the current state of the drives. Assuming that no further problems arise, we're hoping to have the home user directories transfered off of the drives by tonight.

                  Comment

                  • AndrewT
                    Administrator
                    • Mar 2004
                    • 3653

                    #10
                    Within the next couple of hours we should have everyone's data from their public_html directory off of the old cpanel04 drives. The way it is setup now is that these drives are in an extra server mounted as slave while we are trying to copy the data onto a working drive. Once this is complete (in a couple of hours or so), we'll need to transfer all of the data to cpanel52 so that it can be restored. There is a possibility that we may not be able to begin restoring public_html data until tomorrow morning. However, I do want to stress that at this point everything looks good and all of this data should be useable as it was on cpanel04.

                    To ensure that we can restore your public_html data as quickly as possible, please make sure that you have recreated your domains in WHM with the same usernames as they were on cpanel04. This is very important otherwise your databases will not work properly.

                    I will update this thread again once we are able to begin restoring the public_html data and at that point you will need to submit a trouble ticket with a list of usernames that you would like data restored to. We're doing this to ensure that no other existing data is overwritten without your approval.

                    Comment

                    • AndrewT
                      Administrator
                      • Mar 2004
                      • 3653

                      #11
                      Please do not submit requests to restore public_html data at this time, I will update this thread once we are prepared to proceed with these restores.

                      Comment

                      • AndrewT
                        Administrator
                        • Mar 2004
                        • 3653

                        #12
                        Just as a quick update, this is taking us much longer than expected to transfer the data to cpanel52, restores might not be possible until this afternoon.

                        Comment

                        • AndrewT
                          Administrator
                          • Mar 2004
                          • 3653

                          #13
                          We're now ready to begin restoring public_html data. To request this please make sure that the domain has been created with the exact same username that it had before. Then submit a trouble ticket with only a list of usernames (not domains) that you would like the public_html data restored on, place each username on a separate line. Failure to follow this format will result in your ticket receiving lowest priority. We will be processing these tickets as quickly as possible but please realize that it may take a little bit.

                          Comment

                          • AndrewT
                            Administrator
                            • Mar 2004
                            • 3653

                            #14
                            Note that these restores WILL overwrite any data that might already be there if the filenames conflict with the existing cpanel04 data.

                            Comment

                            Working...