| Month |
Measured Availability |
Planned Outages |
Reported Availability |
Notes: |
| 5/2012 |
100% |
0% |
100% |
None |
| 4/2012 |
100% |
0% |
100% |
None |
| 3/2012 |
99.89% |
0% |
99.89% |
Power Event at LWDC data center location affecting 2% of M5 customers. Duration 46 minutes mfor those affected customers. Uptime at SDTC data center was 100% |
| 2/2012 |
100% |
0% |
100% |
None |
| 1/2012 |
100% |
0% |
100% |
None |
| 12/2011 |
100% |
0% |
100% |
None |
| 11/2011 |
100% |
0% |
100% |
None |
| 10/2011 |
100% |
0% |
100% |
None |
| 9/2011 |
100% |
0% |
100% |
None |
| 8/2011 |
100% |
0% |
100% |
None |
| 7/2011 |
100% |
0% |
100% |
None |
| 6/2011 |
99.95% |
0% |
99.95% |
Our colocation provider (American Internet Services) suffered a power outage at our SDTC location. The power was out for 16 minutes. Please read their final Root Cause Analysis. The reported uptime here is 16 minutes plus 5 minutes for our infrastructure to reboot for a total of 21 minutes. While most customers (about 90%) were back on-line within 21 minutes, some were not and required additional time and/or support to return to full finctionality. |
| 5/2011 |
100% |
0% |
100% |
None |
| 4/2011 |
99.92% |
0% |
99.92% |
~5.8% of customers experienced some packet loss for 35 minutes on 4/22/11, during a DoS attck. In a second event, ~4% of customers experienced a 5 minute outage on 4/29/11, caused by human error by one of our network engineers. |
| 3/2011 |
100% |
0% |
100% |
None |
| 2/2011 |
100% |
0% |
100% |
None |
| 1/2011 |
100% |
0% |
100% |
None |
| 12/2010 |
100% |
0% |
100% |
None |
| 11/2010 |
100% |
0% |
100% |
None |
| 10/2010 |
100% |
0% |
100% |
None |
| 9/2010 |
100% |
0% |
100% |
None |
| 8/2010 |
100% |
0% |
100% |
None |
| 7/2010 |
100% |
0% |
100% |
None |
| 6/2010 |
100% |
0% |
100% |
None |
| 5/2010 |
100% |
0% |
100% |
None |
| 4/2010 |
100% |
0% |
100% |
None |
| 3/2010 |
100% |
0% |
100% |
None |
| 2/2010 |
100% |
0% |
100% |
None |
| 1/2010 |
100% |
0% |
100% |
Upstream provider AmericanIS has some trouble with their connection to Level3 which affected some cutomers on our "PremiumNet" More info here. ValueNet was unaffected. |
| 12/2009 |
100% |
0% |
100% |
Scheduled network upgrade resulting in seconds of downtime during the cutover to the new infrastructure More info here |
| 11/2009 |
100% |
0% |
100% |
None |
| 10/2009 |
100% |
0% |
100% |
None |
| 9/2009 |
100% |
0% |
100% |
None |
| 8/2009 |
100% |
0% |
100% |
None |
| 7/2009 |
100% |
0% |
100% |
None |
| 6/2009 |
100% |
0% |
100% |
None |
| 5/2009 |
100% |
0.01% |
100% |
Scheduled maintenance resulting in no downtime More info here |
| 4/2009 |
100% |
0% |
100% |
None |
| 3/2009 |
99.926% |
0.01% |
99.936% |
On 3/11/09 - Some customers experienced up to 33% packet loss for 18 minutes. More info here On 3/18/09 - There was a 90 second power outage in our main colo facility whch caused some follow-on issues for some customers More info here |
| 2/2009 |
100% |
0% |
100% |
None |
| 1/2009 |
100% |
0% |
100% |
None |
| 12/2008 |
100% |
0% |
100% |
None |
| 11/2008 |
99.96% |
0.01% |
99.97% |
Network Upgrades Description here |
| 10/2008 |
100% |
0% |
100% |
None |
| 9/2008 |
100% |
0% |
100% |
None |
| 8/2008 |
99.96% |
0.04% |
100% |
Network Upgrades Description here and a failover in the core switching Description here |
| 7/2008 |
100% |
0% |
100% |
None |
| 6/2008 |
99.998% |
0.002% |
100% |
Aproximately 25% of our customers experienced some intermittent connectivity when multiple interfaces on a core switch began flapping. Some of the affected customers only saw a few brief interruptions, and some may have had some impact over the course of 1hr (99.865% uptime for the month for those worst affected). Less than one minute of interruption during emergency maintenance on the same evening. Description here |
| 5/2008 |
100% |
0% |
100% |
None |
| 4/2008 |
100% |
0% |
100% |
None |
| 3/2008 |
100% |
0% |
100% |
None |
| 2/2008 |
100% |
0% |
100% |
Aproximately 8% of our dedicated server customers were affected by a cross connect failure within the data center facility cable plant. Total downtime was 24 minutes (99.943% uptime this month for those affected). Email explanation sent to those affected customers |
| 1/2008 |
100% |
0% |
100% |
None |
| 12/2007 |
100% |
0% |
100% |
None |
| 11/2007 |
100% |
0% |
100% |
None |
| 10/2007 |
100% |
0% |
100% |
None |
| 9/2007 |
100% |
0% |
100% |
None |
| 8/2007 |
99.989% |
0.011% |
100% |
Scheduled Maintenance during regular window for firewall upgrade and network redundancy improvement. Down time was about 5min. |
| 7/2007 |
100% |
0% |
100% |
None |
| 6/2007 |
100% |
0% |
100% |
None |
| 5/2007 |
100% |
0% |
100% |
None |
| 4/2007 |
99.97% |
0% |
99.97% |
14min outage caused by a border firewall which stopped passing packets until restarted |
| 3/2007 |
100% |
0% |
100% |
A bad cross connect caused packet loss for about 1hr for about 16% of our dedicated server customers. |
| 2/2007 |
100% |
0% |
100% |
None |
| 1/2007 |
100% |
0% |
100% |
None |
| 12/2006 |
100% |
0% |
100% |
None |
| 11/2006 |
99.857% |
0% |
99.857% |
Denial of Service attack against a customer affected performance and availability intermittently over the course of 1hr. A policy change has been made which will prevent this same event from happening again. A correponding firewall change has been implimented to enforce this policy change. |
| 10/2006 |
100% |
0% |
100% |
None |
| 9/2006 |
99.86% |
0% |
99.86% |
At 4:06am PDT Sunday 9/10/06, our border firewall stopped passing packets for an as yet undetermined reason. This event caused a 1hr 5min outage for discovery, recovery and investigation. |
| 8/2006 |
100% |
0% |
100% |
None |
| 7/2006 |
99.879% |
0% |
99.879% |
1) A new customer was the victim of a massive Denial of Service attack for 14min which also affected most of our customers with high packet loss. 2) The Denial of Service attack returned for two blasts during a 40min period. The customer was removed from network permanently and we are making further improvements to our network. |
| 6/2006 |
99.9977% |
0% |
99.9977% |
1) Upgraded to HSRP connection to Data Center core. This required a brief cable change (15 seconds). 2) Some routing issues caused by a corrupt CEF (Cisco Express Forwarding) table in one router caused some customers to have intermittent routing issues to specific Internet addresses. Not all customers were affected, and not all destinations were affected. However, some customers may have been affected for hours. Resolution of this issue required a reboot of one of the aggregation routers. With the HSRP connection, this affected connectivity for about 15 seconds. |
| 5/2006 |
100% |
0% |
100% |
None |
| 4/2006 |
100% |
0% |
100% |
None |
| 3/2006 |
100% |
0% |
100% |
None |
| 2/2006 |
99.993% |
0% |
99.993% |
During a major power upgrade of the Data Center facility on 2/23/06, there was an unplanned 90 second power outage on some circuits. This caused some customers servers to reboot. See the Data Center CEO's explanation (published shortly). The uptime recorded includes 3min downtime for system reboots. |
| 1/2006 |
99.505% |
0% |
99.505% |
Our first unplanned outages in more than 2yrs. Two unplanned outage events affected most customers on two subnets See explanation and mitigation actions here. Monitoring systems were disabled during a period of intermittent connectivity to reduce the number of alerts. |
| 12/2005 |
100% |
0% |
100% |
None |
| 11/2005 |
99.444% |
0.556% |
100% |
Planned Upgrade and outage - 4hr. outage to move all servers and network to new rack space within the same data center. Upgrade of border firewall, and network switching. |
| 10/2005 |
100% |
0% |
100% |
None |
| 09/2005 |
100% |
0% |
100% |
None |
| 08/2005 |
100% |
0% |
100% |
None |
| 07/2005 |
100% |
0% |
100% |
None |
| 06/2005 |
100% |
0% |
100% |
None |
| 05/2005 |
100% |
0% |
100% |
None |
| 04/2005 |
100% |
0% |
100% |
None |
| 03/2005 |
100% |
0% |
100% |
None |
| 02/2005 |
100% |
0% |
100% |
None |
| 01/2005 |
100% |
0% |
100% |
None |
| 12/2004 |
100% |
0% |
100% |
None |
| 11/2004 |
100% |
0% |
100% |
None |
| 10/2004 |
100% |
0% |
100% |
None |
| 09/2004 |
100% |
0% |
100% |
None |
| 08/2005 |
100% |
0% |
100% |
None |
| 07/2004 |
100% |
0% |
100% |
None |
| 06/2004 |
99.965% |
0.035% |
100% |
Scheduled Upgrade - Install new firewall |
| 05/2004 |
100% |
0% |
100% |
None |
| 04/2004 |
99.653% |
0.347% |
100% |
4/8/04 - Upgrade core routers to Cisco GSR 12000 series. Scheduled for 2am to 3am. 4/15/04 - Upgrade supervisor blade in core routers. Scheduled for 1am to 2am. 4/22/04 - Reboot of routers for BGP Vulnerability patch. |
| Month |
Measured Availability |
Planned Outages |
Reported Availability |
Notes: |
| 12/2011 |
100% |
0% |
100% |
None |
| 11/2011 |
100% |
0% |
100% |
None |
| 10/2011 |
100% |
0% |
100% |
None |
| 9/2011 |
100% |
0% |
100% |
None |
| 8/2011 |
100% |
0% |
100% |
None |
| 7/2011 |
100% |
0% |
100% |
None |
| 6/2011 |
99.95% |
0% |
99.95% |
Our colocation provider (American Internet Services) suffered a power outage at our SDTC location. The power was out for 16 minutes. Please read their final Root Cause Analysis. The reported uptime here is 16 minutes plus 5 minutes for our infrastructure to reboot for a total of 21 minutes. While most customers (about 90%) were back on-line within 21 minutes, some were not and required additional time and/or support to return to full finctionality. |
| 5/2011 |
100% |
0% |
100% |
None |
| 4/2011 |
100% |
0% |
100% |
None |
| 3/2011 |
100% |
0% |
100% |
None |
| 2/2011 |
100% |
0% |
100% |
None |
| 1/2011 |
100% |
0% |
100% |
None |
| 12/2010 |
100% |
0% |
100% |
None |
| 11/2010 |
100% |
0% |
100% |
None |
| 10/2010 |
100% |
0% |
100% |
None |
| 9/2010 |
100% |
0% |
100% |
None |
| 8/2010 |
100% |
0% |
100% |
None |
| 7/2010 |
100% |
0% |
100% |
None |
| 6/2010 |
100% |
0% |
100% |
None |
| 5/2010 |
100% |
0% |
100% |
None |
| 4/2010 |
100% |
0% |
100% |
None |
| 3/2010 |
100% |
0% |
100% |
None |
| 2/2010 |
100% |
0% |
100% |
None |
| 1/2010 |
100% |
0% |
100% |
Upstream provider AmericanIS has some trouble with their connection to Level3 which affected some cutomers on our "PremiumNet" More info here. ValueNet was unaffected. |
| 1/2010 |
100% |
0% |
100% |
None |
| 12/2009 |
100% |
0% |
100% |
Scheduled network upgrade resulting in seconds of downtime during the cutover to the new infrastructure More info here |
| 11/2009 |
100% |
0% |
100% |
None |
| 10/2009 |
100% |
0% |
100% |
None |
| 9/2009 |
100% |
0% |
100% |
None |
| 8/2009 |
100% |
0% |
100% |
None |
| 7/2009 |
100% |
0% |
100% |
None |
| 6/2009 |
100% |
0% |
100% |
None |
| 5/2009 |
100% |
0.01% |
100% |
Brief scheduled maintenance resulting in no downtime More info here |
| 4/2009 |
100% |
0% |
100% |
None |
| 3/2009 |
100% |
0% |
100% |
None |
| 2/2009 |
100% |
0% |
100% |
None |
| 1/2009 |
100% |
0% |
100% |
None |
| 12/2008 |
100% |
0% |
100% |
None |
| 11/2008 |
100% |
0% |
100% |
None |
| 10/2008 |
100% |
0% |
100% |
None |
| 9/2008 |
100% |
0% |
100% |
None |
| 8/2008 |
100% |
0% |
100% |
None |
| 7/2008 |
100% |
0% |
100% |
None |
| 6/2008 |
99.998% |
0.002% |
100% |
Less than one minute of interruption during an emergency maintenance periodDescription here |
| 5/2008 |
100% |
0% |
100% |
None |
| 4/2008 |
100% |
0% |
100% |
None |
| 3/2008 |
100% |
0% |
100% |
None |
| 2/2008 |
100% |
0% |
100% |
None |
| 1/2008 |
100% |
0% |
100% |
None |
| 12/2007 |
100% |
0% |
100% |
None |
| 11/2007 |
100% |
0% |
100% |
None |
| 10/2007 |
100% |
0% |
100% |
None |
| 9/2007 |
100% |
0% |
100% |
None |
| 8/2007 |
99.989% |
0.011% |
100% |
Scheduled Maintenance during regular window for firewall upgrade and network redundancy improvement. Down time was about 5min. |
| 7/2007 |
100% |
0% |
100% |
None |
| 6/2007 |
100% |
0% |
100% |
None |
| 5/2007 |
100% |
0% |
100% |
None |
| 4/2007 |
99.97% |
0% |
99.97% |
14min outage caused by a border firewall which stopped passing packets until restarted |
| 3/2007 |
100% |
0% |
100% |
None |
| 2/2007 |
100% |
0% |
100% |
None |
| 1/2007 |
100% |
0% |
100% |
None |
| 12/2006 |
100% |
0% |
100% |
None |
| 11/2006 |
99.857% |
0% |
99.857% |
Denial of Service attack against a customer affected performance and availability intermittently over the course of 1hr. A policy change has been made which will prevent this same event from happening again. A correponding firewall change has been implimented to enforce this policy change. |
| 10/2006 |
100% |
0% |
100% |
None |
| 9/2006 |
99.86% |
0% |
99.86% |
At 4:06am PDT Sunday 9/10/06, our border firewall stopped passing packets for an as yet undetermined reason. This event caused a 1hr 5min outage for discovery, recovery and investigation. |
| 8/2006 |
100% |
0% |
100% |
None |
| 7/2006 |
99.879% |
0% |
99.879% |
1) A new customer was the victim of a massive Denial of Service attack for 14min which also affected most of our customers with high packet loss. 2) The Denial of Service attack returned for two blasts during a 40min period. The customer was removed from network permanently and we are making further improvements to our network. |
| 6/2006 |
99.9282% |
0.0694% |
99.9977% |
1) Upgraded to HSRP connection to Data Center core. This required a brief cable change (15 seconds). 2) Some routing issues caused by a corrupt CEF (Cisco Express Forwarding) table in one router caused some customers to have intermittent routing issues to specific Internet addresses. Not all customers were affected, and not all destinations were affected. However, some customers may have been affected for hours. Resolution of this issue required a reboot of one of the aggregation routers. With the HSRP connection, this affected connectivity for about 15 seconds. 3) Planned upgrade of CPU on shared hosting server. Planned for 1hr, actual downtime about 30min |
| 5/2006 |
100% |
0% |
100% |
None |
| 4/2006 |
100% |
0% |
100% |
None |
| 3/2006 |
100% |
0% |
100% |
None |
| 2/2006 |
99.993% |
0% |
99.993% |
During a major power upgrade of the Data Center facility on 2/23/06, there was an unplanned 90 second power outage on some circuits. This caused some servers to reboot. See the Data Center CEO's explanation (published shortly). The uptime recorded includes 3min downtime for system reboots. |
| 1/2006 |
99.460% |
0.063% |
99.523% |
Three unplanned outage events. One was a kernel panic on the share hosting platform that was caused by a dead fan. Two affecting most customers on two subnets See explanation and mitigation actions here. Monitoring systems were disabled during a period of intermittent connectivity to reduce the number of alerts. Actual downtime was slightly less than the number reported here, but not our best month. Planned event announced here. |
| 12/2005 |
100% |
0% |
100% |
None |
| 11/2005 |
99.444% |
0.556% |
100% |
Planned Upgrade and outage - 4hr. outage to move all servers and network to new rack space within the same data center. Upgrade of border firewall, and network switching. |
| 10/2005 |
100% |
0% |
100% |
None |
| 09/2005 |
99.983% |
0% |
99.983% |
Two unplanned reboots required to resolve one possible security issue, and one hardware issue. Both reboots were performed after business hours. Total downtime about 7min. |
| 08/2005 |
100% |
0% |
100% |
None |
| 07/2005 |
99.444% |
0.556% |
100% |
Major upgrade performed on 7/30/05. Upgraded operating system to RHEL 3 ES, migrated to new hardware, upgraded Plesk control panel to latest version. This scheduled and previously announced work took about 4 hours to complete. Shared Hosting service may have been intermittent during the 4hr upgrade. |
| 06/2005 |
99.988% |
0.012% |
100% |
Scheduled Late Evening Reboot to clear a memory consuming dead process on "stout" and kernel patch all. Scheduled for 5 minutes. |
| 05/2005 |
100% |
0% |
100% |
None |
| 04/2005 |
100% |
0% |
100% |
None |
| 03/2005 |
100% |
0% |
100% |
None |
| 02/2005 |
100% |
0% |
100% |
None |
| 01/2005 |
100% |
0% |
100% |
None |
| 12/2004 |
100% |
0% |
100% |
None |
| 11/2004 |
100% |
0% |
100% |
None |
| 10/2004 |
99.893% |
0.107% |
100% |
Planned downtime for hardware move and kernel patches (46min) |
| 09/2004 |
100% |
0% |
100% |
None |
| 08/2004 |
100% |
0% |
100% |
None |
| 07/2004 |
100% |
0% |
100% |
None |