Summary

SparkPost APIs and the UI had high error rates during several periods of time starting on Wednesday May 11th 2016 through Saturday May May 14th. During these periods customers received the following error intermittently, and at times very frequently, HTTP code 503 "Unable to contact authorization service.” A similar authentication error was returned if using SMTP.

Timeline

During the following times we had elevated error rates on all APIs and the UI.

• May 11th 13:28 UTC - May 11th 14:15 UTC
• May 11th 14:59 UTC - May 11th 15:15 UTC
• May 11th 17:19 UTC - May 11th 17:42 UTC
• May 11th 20:17 UTC - May 11th 20:38 UTC
• May 12th 00:42 UTC - May 11th 00:49 UTC
• May 12th 05:17 UTC - May 12th 05:23 UTC
• May 12th 07:16 UTC - May 12th 07:22 UTC
• May 12th 09:03 UTC - May 12th 09:08 UTC
• May 12th 10:59 UTC - May 12th 11:16 UTC
• May 12th 18:29 UTC - May 12th 18:31 UTC
• May 13th 04:10 UTC - May 13th 04:25 UTC
• May 13th 08:17 UTC - May 13th 08:35 UTC
• May 14th 19:53 UTC - May 14th 20:16 UTC

This severely impacted the ability to submit new transmissions via REST and SMTP API. However, any transmission that was accepted with a 200 or a 250ok was queued for delivery and processed.

Impact ends after a reliable mitigation strategy is implemented and all alerts clear

• May 14th 20:16 UTC

Root cause fixed in production

• May 16th 21:30 UTC

Security Implications

None

Root Cause

• Why did the APIs return a 503 authentication error? The authentication system was unable to read from the database cluster.
• Why was the authentication system unable to read from the database? The database was not responding because it had reached its max heap size resulting in long Garbage Collection pauses.
• Why did the database reach max heap size? A query to retrieve and cache sending domains which randomly used any connection in the pool consumed large blocks of memory in the database that were not freed up quickly enough.
  
• Why did this this problem occur all of a sudden? The number of sending domains increased significantly over the prior 2 months until it hit a tipping point.
  
• Why did it take 3 days to implement a complete mitigation strategy? Initial mitigation strategies were not highly effective since the root cause was not known. No code changes were made that morning but the team did investigate possible code changes introduced in the prior week and also looked at database garbage collection tuning. However, the actual root cause was sending domain data volume growth and the way the connection pools were structured. Once the cause was identified then a proper mitigation strategy was implemented which involved automated rolling restarts of affected services that mitigated customer impact until the connection pooling was updated, tested, and deployed on Monday May 16th.

Corrective actions taken

• Improved monitoring and alerting on database heap size
• Augmented database diagnostic tools
• Fixed connection pooling to handle sending domains cache query

Follow up actions

Although the problem was fixed on the 16th there are several follow up actions:

• Upgrade database driver which will do a better job at connection pool management.
• Optimize the sending domains schema and tracking domains schema in database and the bulk loading into the cache
• Remove transient transmissions data from database to reduce the load.
• Launch a “status page” to better communicate status to customers. During the incident updates were posted to the public slack channel #announcements but this was not sufficient. (this was done on May 19th)