Elevated 5xx API error rates and increased latency for some SparkPost and SparkPost Enterprise customers (US hosted only)

Incident Report for SparkPost

Postmortem

During this incident, some customers hosted in the US region observed an elevated level of API errors, saw errors in the SparkPost application (UI) and experienced latency on API calls. In addition, outbound message delivery was slow during the impact period.

*Incident Period: *Aug 31 2018 from 4:38 PM to 5:00 PM EST.

Root Cause & Corrective Actions

The root cause was an availability/performance issue in an underlying component used for suppression checks that caused downstream impacts to the core messaging infrastructure.

We take reliability of our service very seriously. Changes to more fully isolate the core customer facing email services from the suppression check infrastructure are already underway and will be fully in place later this month.

Posted Sep 11, 2018 - 13:16 EDT

Resolved

This incident has been resolved.

Posted Aug 31, 2018 - 17:05 EDT

Investigating

We are experiencing an elevated level of API and UI errors and latency for our APIs for some SparkPost and SparkPost Enterprise customers (including Transmissions API and SMTP injection API). Please retry any 5xx error.
Note: This issue does not impact our SparkPost Enterprise customers hosted in the EU.

Posted Aug 31, 2018 - 16:53 EDT

This incident affected: Metrics API (Metrics API - USA), Transmissions API (Transmissions API - USA), Events API (Events API - USA), SparkPost Application (WebUI) (SparkPost Application - USA), SMTP API (SMTP API - USA), Sending Domains API (Sending Domains API - USA), and Relay Webhooks Delivery Service (Relay Webhooks - USA).