Data Center Core Router

[Matt]

AtlantaNAP, our co-location facility, experienced a core router failure at approximately 5:50 AM EST on October 26th. Connectivity was restored by 12:01 PM. The cause of the outage nor duration are not immediately known; however, we were one of a multitude of hosting providers affected by this outage.

AtlantaNAP has not been communicative on this issue as it develops, but from excerpts gleaned from the forums, I know that a supervisor card inside the router failed precipitating the outage. Two core routers provide bandwidth to all servers within the data center. We were on core A that experienced failure.

http://www.webhostingtalk.com/showthread.php?t=900059
http://gnax.net/forum/showthread.php?t=1877

Monday and Friday are our most important days and likewise, your most important days. Sustaining an outage for any duration longer than a coffee break is disruptive. Although we provide no expressed SLA, clients affected by this outage will receive a one month credit for compensation. Please file a ticket within the control panel requesting credit. For those paying by PayPal, a refund will be issued to the e-mail address on record with the domain. Credit card-backed payments will receive a discount on the next transaction.

[dkozinn]

I know this was a pretty unusual situation, but since all the servers were cut off, we couldn't file a trouble ticket or even inquire as to the nature of the problem via email.

As you know, I saw your posting on Twitter, which is how I found out what the problem is; would you recommend that we monitor your twitter account (or possibly a new one, specifically for outage notifications) for these kinds of things in the future?

[Matt]

Originally Posted by dkozinn

I know this was a pretty unusual situation, but since all the servers were cut off, we couldn't file a trouble ticket or even inquire as to the nature of the problem via email.

As you know, I saw your posting on Twitter, which is how I found out what the problem is; would you recommend that we monitor your twitter account (or possibly a new one, specifically for outage notifications) for these kinds of things in the future?

I have @apisnetworks that is purposed for such a role. This account will be used to relay outages and maintenance, while @msaladna will continue to be internals. Handing out, "apisnetworks" is much easier than enunciating each letter in "Saladna"

No formal announcement yet on the 6 hour delay from the DC. At this point I doubt we'll see a statement nor changes to mitigate a similar outage. Gnax referenced multipath networks in the ticket escalated on Monday, but no clarification nor pricing has been posted to the ticket.