Matt
09-11-2006, 07:24 PM
Currently the data center that houses Gauss (66.98.192.43) is experiencing intermittent connectivity issues as of approximately 6 PM EDT (-0400 GMT) per reports. The data center has no ETA at this time, but hold tight and everything should be back again shortly.
6:41 PM: Appears to be back up now.
7:12 PM: Official word from the data center:
At approx. 16:40hrs US CDT September 11th, we experienced an outage at Datacenter 1. This issue also affected inbound traffic directed to DC2 phase 2, via the DC1 backbones. This issue has been resolved, and connectivity is funtioning normally.
The source of this issue was a misconfiguration of the Fireslayer Dos Mitigation system. After reviewing logs we have discovered that an important safety measure was mistakenly removed from the system several days ago. Upon removal of fireslayer from a targeted IP this afternoon, a flawed firewall change was written to the edge routers. This change resulted in all inbound traffic at the DC1 edge to be blocked.
We aplogize to those customers who were affected by this issue. We will be reviewing this issue with the personnel involved. In additon we will be examining and implementing additonal safegaurds to the fireslayer system to prevent this type of issue from occuring in the future.
6:41 PM: Appears to be back up now.
7:12 PM: Official word from the data center:
At approx. 16:40hrs US CDT September 11th, we experienced an outage at Datacenter 1. This issue also affected inbound traffic directed to DC2 phase 2, via the DC1 backbones. This issue has been resolved, and connectivity is funtioning normally.
The source of this issue was a misconfiguration of the Fireslayer Dos Mitigation system. After reviewing logs we have discovered that an important safety measure was mistakenly removed from the system several days ago. Upon removal of fireslayer from a targeted IP this afternoon, a flawed firewall change was written to the edge routers. This change resulted in all inbound traffic at the DC1 edge to be blocked.
We aplogize to those customers who were affected by this issue. We will be reviewing this issue with the personnel involved. In additon we will be examining and implementing additonal safegaurds to the fireslayer system to prevent this type of issue from occuring in the future.