Skip to end of metadata
Go to start of metadata

Incident Review

AppNexus focuses on reliability and high availability and takes all possible precautions against downtime.  But occasionally we do have incidents.  When that happens, we investigate the problem and pass on to our customers everything we learn with full transparency.  Please visit this page to see a list of incidents and their explanations.

As always, please create a ticket at https://portal.appnexus.com/ or contact us at support@appnexus.com if you have any questions or concerns.

Date

Type

Severity

Time (ET)

Description

4-4-10

Network Connectivity

Moderate

01:22 to 01:23

Bandcon flapping affected the network connection between NYM1 and LAX1.

3-11-10

Power Outage in LAX1, Rack 03-01

Moderate

2010-03-10 22:25 UTC to 2010-03-11 03:19 UTC

Faulty ATS caused boxes in LAX1 rack 03-01 to lose power

2-1-10

Power Outage in NYM1, Rack 01-07

Moderate

6:37 to 8:32

Faulty ATS caused some web boxes in NYM1 rack 01-07 to lose power

11-18-09

API Unavailability, Isolated Power Outage

High

21:15 (Nov. 17) to 00:15 (Nov. 18)

Faulty ATS caused LAX1 rack to lose power. Some manual reboots required

7-29-09

Local Load Balancing Failover; API Unavailability

Moderate

05:00 to 06:00

 

7-26-09

Network Connectivity

Moderate

04:51 to 05:04

Flapping Router

7-23-09

Network Connectivity

Moderate

05:27 to 05:34

Intermittent Connectivity Problems between NYM1 and LAX1 datacenters

6-4-09

Internal DNS

High

13:58 to 15:00

Internal DNS unavailable in NYM1

5-6-09

Network Connectivity

Low

13:15

ISP's routing error caused minor packet loss for a small subset of customers in NYM1

2-25-09

CDN Disruption

Moderate

10:55 to 11:25

A half-hour DNS outage on the Panther CDN network affected certain AppNexus customer domains

1-10-09

Global Load Balancing Redundancy Loss

Moderate

10:00 Saturday to 19:00 Sunday

Minor blips for the NYM1 GTM meant all traffic was routed through the LAX1 GTM

12-28-08

Network Connectivity

Moderate

12:35 to 1:57

Intermittent connectivity problems in NYM1 and LAX1 due to ISP

12-26-08

API Unavailability

Moderate

03:00 to 09:00

Intermittent API Unavailability in NYM1 and LAX1

11-13-08

API Unavailability

Moderate

05:58 to 06:55

Intermittent API Unavailability in NYM1 and LAX1

11-5-08

API Unavailability

Moderate

22:00 to 23:10

Intermittent API Unavailability in the NYM1 datacenter

10-27-08

Content Delivery Network

Moderate

11:30 to 13:10

Occasional brief loss of CDN service for some European end users

10-3-08

Local Load Balancing Outage

Moderate

00:09 to 00:10

Spontaneous reboot of primary F5 load balancer in LAX1 with immediate failover

9-26-08

Local Load Balancing Outage

High

00:07 to 00:29

Spontaneous reboot of primary F5 load balancer in LAX1 with lingering connectivity issues for some VIPs

9-24-08

Network Connectivity

Moderate

22:58 to 23:00

Intermittent connectivity problems in NYM1 due to Equinix maintenance

9-11-08

Network Connectivity

Moderate

04:02 to 04:19

Intermittent connectivity problems in NYM1 due to ISP

9-9-08

Network Connectivity

Moderate

09:24 to 09:26

Brief connectivity problem in LAX1

9-5-08

Network Connectivity

Moderate

00:20 to 00:24
01:06 to 01:10

Intermittent connectivity problems in NYM1 due to ISP

8-9-08

Local Load Balancing Outage

Critical

08:40 to 09:00

Spontaneous reboot of both F5 load balancers in LAX1

6-30-08

Local Load Balancing Outage

High

21:22 to 21:23

Spontaneous reboot of both F5 load balancers in LAX1 with immediate failover

4-9-08

Storage Unavailable

Moderate

18:37 to 18:57

Network misconfiguration resulted in storage unavailability for 20 minutes

3-27-08

Network Connectivity

Low

18:02 to 18:03

Intermittent network connectivity

2-19-08

API Unavailability

Low

09:37 to 10:00

Failure of LAX1 master database

AppNexus went live February 1, 2008.

Severity range: Low, Moderate, High, Critical.  Severity is based on type and length of outage.

  • No labels
Page: 10-27-08 Page: 9-9-08 Page: 11-5-08 Page: 11-13-08 Page: 9-11-08 Page: 10-3-08 Page: 8-9-08 Page: 2-19-08 Page: 3-27-08 Page: 6-30-08 Page: 9-24-08 Page: 9-5-08 Page: 9-26-08 Page: 4-9-08 Page: 12-28-08 Page: 1-10-09 Page: 2-25-09 Page: 5-6-09 Page: 12-26-08 Page: 6-4-09 Page: 7-29-09 Page: 7-26-09 Page: 7-23-09 Page: 11-18-09 Page: 2-1-10 Page: 3-11-10 Page: 4-4-10