Incident Review
AppNexus focuses on reliability and high availability and takes all possible precautions against downtime. But occasionally we do have incidents. When that happens, we investigate the problem and pass on to our customers everything we learn with full transparency. Please visit this page to see a list of incidents and their explanations.
As always, please create a ticket at https://portal.appnexus.com/ or contact us at support@appnexus.com if you have any questions or concerns.
Date |
Type |
Severity |
Time (ET) |
Description |
|---|---|---|---|---|
Network Connectivity |
Moderate |
01:22 to 01:23 |
Bandcon flapping affected the network connection between NYM1 and LAX1. |
|
Power Outage in LAX1, Rack 03-01 |
Moderate |
2010-03-10 22:25 UTC to 2010-03-11 03:19 UTC |
Faulty ATS caused boxes in LAX1 rack 03-01 to lose power |
|
Power Outage in NYM1, Rack 01-07 |
Moderate |
6:37 to 8:32 |
Faulty ATS caused some web boxes in NYM1 rack 01-07 to lose power |
|
API Unavailability, Isolated Power Outage |
High |
21:15 (Nov. 17) to 00:15 (Nov. 18) |
Faulty ATS caused LAX1 rack to lose power. Some manual reboots required |
|
Local Load Balancing Failover; API Unavailability |
Moderate |
05:00 to 06:00 |
|
|
Network Connectivity |
Moderate |
04:51 to 05:04 |
Flapping Router |
|
Network Connectivity |
Moderate |
05:27 to 05:34 |
Intermittent Connectivity Problems between NYM1 and LAX1 datacenters |
|
Internal DNS |
High |
13:58 to 15:00 |
Internal DNS unavailable in NYM1 |
|
Network Connectivity |
Low |
13:15 |
ISP's routing error caused minor packet loss for a small subset of customers in NYM1 |
|
CDN Disruption |
Moderate |
10:55 to 11:25 |
A half-hour DNS outage on the Panther CDN network affected certain AppNexus customer domains |
|
Global Load Balancing Redundancy Loss |
Moderate |
10:00 Saturday to 19:00 Sunday |
Minor blips for the NYM1 GTM meant all traffic was routed through the LAX1 GTM |
|
Network Connectivity |
Moderate |
12:35 to 1:57 |
Intermittent connectivity problems in NYM1 and LAX1 due to ISP |
|
API Unavailability |
Moderate |
03:00 to 09:00 |
Intermittent API Unavailability in NYM1 and LAX1 |
|
API Unavailability |
Moderate |
05:58 to 06:55 |
Intermittent API Unavailability in NYM1 and LAX1 |
|
API Unavailability |
Moderate |
22:00 to 23:10 |
Intermittent API Unavailability in the NYM1 datacenter |
|
Content Delivery Network |
Moderate |
11:30 to 13:10 |
Occasional brief loss of CDN service for some European end users |
|
Local Load Balancing Outage |
Moderate |
00:09 to 00:10 |
Spontaneous reboot of primary F5 load balancer in LAX1 with immediate failover |
|
Local Load Balancing Outage |
High |
00:07 to 00:29 |
Spontaneous reboot of primary F5 load balancer in LAX1 with lingering connectivity issues for some VIPs |
|
Network Connectivity |
Moderate |
22:58 to 23:00 |
Intermittent connectivity problems in NYM1 due to Equinix maintenance |
|
Network Connectivity |
Moderate |
04:02 to 04:19 |
Intermittent connectivity problems in NYM1 due to ISP |
|
Network Connectivity |
Moderate |
09:24 to 09:26 |
Brief connectivity problem in LAX1 |
|
Network Connectivity |
Moderate |
00:20 to 00:24 |
Intermittent connectivity problems in NYM1 due to ISP |
|
Local Load Balancing Outage |
Critical |
08:40 to 09:00 |
Spontaneous reboot of both F5 load balancers in LAX1 |
|
Local Load Balancing Outage |
High |
21:22 to 21:23 |
Spontaneous reboot of both F5 load balancers in LAX1 with immediate failover |
|
Storage Unavailable |
Moderate |
18:37 to 18:57 |
Network misconfiguration resulted in storage unavailability for 20 minutes |
|
Network Connectivity |
Low |
18:02 to 18:03 |
Intermittent network connectivity |
|
API Unavailability |
Low |
09:37 to 10:00 |
Failure of LAX1 master database |
AppNexus went live February 1, 2008.
Severity range: Low, Moderate, High, Critical. Severity is based on type and length of outage.