More Power Too Us Part 3: Rack Redo Recap
August 9th, 2006 at 5:59 pm by zack
Continuing our drive to make our co-located services more reliable and easier to work on, we tackled the very large problem of the mess of wires and wasted rack space.
The problem:
1. There was nearly 7U of wasted space. (For the normal people out there, a ‘U’ is 1.75″ of vertical space in a rack)
2. Wires were everywhere.
3. We had outgrown our power outlets into the rack and needed more to plug in our servers.
4. Without physically being at the rack, we couldn’t turn on or off a device.
The solution:
1. Remove every last component from the rack and rebuild it from scratch. By doing this, we were able to group our different servers into logical units, as well as put all the dells at the top, making it look cool. Also, we re-mounted our switches in a way that freed up another U.
2. While the rack is apart, properly install and use cable management to clean up the nest of wires.
3 & 4. Install higher amperage power distribution units that have more outlets. As well, we chose two Switched PDUs from APC that have both serial and IP based management. Now, if a device needs to be rebooted, all we have to do is visit a web page.
10pm, The Night of the Swap
What in the world have we gotten ourselves into? Percy, Jason and I arrived at the facility and took a second to look at the task ahead of us. We took all of our external monitoring applications offline for the time being (as to not get paged over and over that the site was down) and then turned off all of the servers in order.
We started by removing each server one at a time and marking all of the important pieces.
It was looking nice.
11PM, Making Progress
Slowly but surely we took each item out.
11:30 PM, Everything Is Out
38U of stuff later and we had a real mess on our hands.
We left the console installed, but finally had everything else out. Not too bad.
Now To Put It Back Together
We started by installing the wiring (network and kvm) and putting in the cable management.
4am, A few Snags Later
At this point, I’m only bleeding a little (dang sharp metal), but we’ve made a lot of progress. We found out why the bottom U was never used (it had a ledge that stuck up) but still managed to use it by moving a non-dell server to that spot.
7am, Pretty much back together
At this point, we had most of the servers back in and wired right. Each box took about 45 minutes to totally wire, notwithstanding the fact that we were tired by this point.
Once the boxes were all in and powered up, we started testing the sites. We did have one casualty. A backend data loading box decided it didn’t want to boot anymore. It couldn’t decide if it was 650mhz or 1ghz and kept hanging. We had to take it in to work on it.
In The End
After it was all said and done, we had a much more manageable space on our hands. We all learned a lot of things on proper rack installation and will draw on that knowledge as we do more work. Since these pictures were taken, we’ve actually put in 3 more servers at the top of the rack. It really fills out the space nicely.