Summary

On Thursday May 19th there was a period of time where the Transmissions API (and Sending Domains API, Recipient List API, Tracking Domains API) had high error rates resulting in many failures to inject email to SparkPost. The WebUI was also unavailable during this incident due to dependencies on some of these affected APIs. There was a 30 minute window when there was a full Transmissions API outage and a full SMTP outage of 10 minutes as some services had to be brought down completely and restarted. Apart from the 10 minute SMTP outage, SMTP was working normally during this incident.

The issue was brought on by a bug in connection management within the MTA that was introduced two days prior on Tuesday the 17th as part of an improvement to connection pooling. This new bug resulted in a cascading failure in the Transmissions API when database connections leaked during recovery/retry. Once the cause was determined the software was rolled back to the previous version.

Timeline

• May 17th 16:40 UTC: an improvement to database connection management for the MTA was rolled out to production. This had a connection leak in a corner case in a recovery path. The MTA begins to slowly leak database connections.
• May 19th 13:00 UTC - Transmission API and some other APIs begin intermittently timing out, causing some sporadic alerts.
• May 19th 13:38 UTC - Transmission API database queries start failing completely, resulting in many HTTP 499 errors and additional alerts triggering.
• May 19th 14:05 UTC - restarting MTA service on some hosts cleared the 499 errors and the Transmission API was partially available at this time, but still giving errors on some hosts.
• May 19th 15:20 UTC - after unsuccessful attempts to restart services on other hosts we eventually killed the processes and rolled back the software to prior version and restarted.
• May 19th 15:30 UTC - All services are back online. Queues cleared out over next 15 minutes.

Security Implications

• None

Root Cause

1. Why did the Transmission API timeout? The MTA had too many database connections open.
2. Why did the MTA have too many open connections to the database? The MTA had a connection leak during database query failover, that is when a query failed on one node and would be retried on a different node. These connections built up over the prior 2 days.
3. How did this connection leak get introduced? The connection leak was introduced in a code change that was deployed on the 17th which was supposed to make database connections more stable. The connection leak only occurs during certain failover scenarios.
4. Why wasn't this connection leak noticed until the system became unresponsive? There was no monitoring or alerting on the number of db connections.
5. Why did it take so long to recover from the failure? The recovery took longer because the MTA would not shut down using normal procedures due to the open database operations that could not complete. As a result we had to use an alternate shutdown process which resulted in a brief period of time when the SMTP and Transmissions API were offline completely. After rollback and restart services came back online. During the entire incident any mail that was accepted by SparkPost (HTTP 200 or SMTP 250ok) was queued for delivery and processed normally.
   

Corrective Actions

• MTA service stopped and software rolled back

Follow up actions

• Fix MTA database connection leak on recovery case (done)
• Additional database connection # monitoring and alerting (done)
• Make it easier to shut down MTA if there are database connection problems (to do)
• Remove dependency on templates API to load the reports in the UI (done)
• Modularize UI to fail more gracefully if some APIs are not available (to do)