3/3/12 - Scheduled Maintenance

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • AndrewT
    Administrator
    • Mar 2004
    • 3656

    #1

    3/3/12 - Scheduled Maintenance

    On Saturday, March 3rd we will be performing various maintenance tasks on our hardware. While we cannot provide an exact time schedule for these I wanted to go ahead and post a notification concerning them along with estimated downtime.

    Power Maintenance
    A redundant power drop has been installed in our cabinet. We will be re-cabling all power connectivity to accommodate this. Approximately 5-10 minutes of network downtime will occur while single corded switches are powered down, plugged in to a new automatic transfer switch, and powered back on. Individual hosting servers are dual corded so these will not need to be powered off. After this maintenance all of our hardware will be powered by two entirely separate power sources on different grids, UPSs, and generators.

    Kernel Upgrades
    Kernel upgrades will be performed on cpanel75, cpanel77, and cpanel78. This will require a reboot with approximately 5-10 minutes of downtime for each server.

    cpanel75 RAM Maintenance
    After a recent reboot only 40GB of RAM is being detected in this server instead of 48GB. We will have to take the server offline to make sure all RAM is properly seated. If this does not resolve the issue we will identify the failed 8GB module and remove it temporarily while we await a matched replacement. This could result in anywhere between 15 and 60 minutes of downtime depending on what we find.

    cpanel75 RAID Maintenance
    One of this server's RAID 10 disks has begun showing a couple of errors. This isn't anything to be worried about since we do run RAID 10 and have hot spares installed in all of our servers. We'll simply be hot swapping the drive with a new one. No downtime is expected.
  • AndrewT
    Administrator
    • Mar 2004
    • 3656

    #2
    All maintenance has been completed and I've included some brief details below. Please let us know if you have any questions or concerns.

    Power Maintenance
    All of our hardware is now connected to fully redundant A and B power. We successfully tested fail over of both. Total public network downtime was about 1 minute for the switches to power back on.

    Kernel Upgrades
    These were completed as scheduled. cpanel77 was offline for approximately 4 minutes and and cpanel78 for approximately 10 minutes. cpanel75 was rebooted as part of the RAM maintenance detailed below so a separate reboot was unnecessary.

    cpanel75 RAM Maintenenace
    We identified the failed module and replaced it. All 48GBs are available once again. Total downtime was approximately 29 minutes.

    cpanel75 RAID Maintenance
    The drive with errors was hot swapped after the RAM maintenance and the array is currently rebuilding.

    Comment

    Working...