Elevated 5xx error rates for some SparkPost and SparkPost Enterprise customers in the US

Incident Report for SparkPost

Postmortem

We have completed our postmortem analysis of the API elevated error rates that we reported over several days last week.

Last week beginning on Monday evening (10 PM EST Nov 6th) and continuing until midday on Thursday (12 PM EST Nov 9th), we experienced intermittent elevated error rates on all APIs (including SMTP API used for injection). The 5xx errors presented in very short and infrequent bursts (aka “alert storms”) occurred a handful of times each day and lasted only several minutes. For many customers the impact was limited due to caching. Therefore, many customer would not have observed the errors or experienced any impact. (No data was lost due to this incident and outbound message delivery was not impacted.)

The root cause of this event was related to performance issues with one of the data storage components within SparkPost used for authentication. Since this event, we have made a number of enhancements to the affected component (including provisioning more resources, tuning the system, and tweaking caching rules) that we are confident will reduce the likelihood and scope of this type of performance issue.

Posted Nov 17, 2017 - 14:17 EST

Resolved

The issue is resolved and we are continuing to closely monitor the affected systems

Posted Nov 09, 2017 - 14:51 EST

Monitoring

We've deployed a fix and are monitoring for further errors.

Posted Nov 09, 2017 - 10:01 EST

Identified

There have been some additional intermittent errors over the past 6 hours. We are aware of these and are continuing to take further action. This continues to be highest priority for our team. Thank you for your patience.

Posted Nov 09, 2017 - 06:33 EST

Monitoring

We made additional changes including adding significant capacity to some backend resources. We will leave open for next 6-12 hours for monitoring of any reoccurrence of the these transient errors.

Posted Nov 08, 2017 - 22:43 EST

Update

We have confirmed that our European hosted Enterprise customers are not affected.
We continue to investigate and will update this incident with more details as soon as we can

Posted Nov 08, 2017 - 22:03 EST

Investigating

We are seeing occasional brief periods of transient errors and timeouts followed by periods of no errors. This may affect users of both REST API and SMTP. Our team is investigating further. Please retry any 5xx errors.

Posted Nov 08, 2017 - 21:05 EST

This incident affected: Transmissions API (Transmissions API - USA), SparkPost Application (WebUI) (SparkPost Application - USA), and SMTP API (SMTP API - USA).