From Our Host in Houston:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.
OK, then!
I don't think we can be very angry at them.
Frankly, I'm just very glad that no one was hurt and that we'll be back up and whole soon.
We'll advise as we learn more. If you need us, try myoung at equityguardiangroup dot com
4 comments:
This just in from our Host:
"We are still working through the DNS and ServerCommand items. We are making progress on both items and expect to have both functional within the next 120 minutes.
In addition to the above, the network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online.
We do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning-- we will continue to provide hourly updates."
Yes, thank God no one was injured.
Thanks for letting us know, it sounds terrible. Good luck to you guys in getting the stuff up and running...
They're still working on it. They intend to start powering up at 6:00 but even if things work perfectly, it's still going to take a while.
We'll let you know of any further status reports up until 9:00 tonight, eastern time. After that, it'll be catch as catch can.
I think anyone who runs or who has run any sort of a business has to feel for those poor folks down in Houston. My appreciation goes out to the techs and their families who have had to do without them all weekend.
Mark
Post a Comment