In the past few weeks, we experienced some performance challenges with our authentication services that caused an increase in API latency and 5xx error rates. (We notified our customers of these incidents via the SparkPost Status Page - please subscribe if you have not already done so.)
What happened?
Recent organic growth put pressure on the data storage infrastructure that we use for authentication and some other key services. This caused some downstream impacts and some customers would have seen the following during periodic episodes of performance issues:
Due to existing caching and high-availability features, many customers’ production email streams were minimally impacted by these events.
What did we do about it?
During the past few months, we performed some tuning and adjusted caching within the authentication system to support our growth trends; however, we determined these improvements alone were not enough to ensure we meet our performance and availability targets. So yesterday, January 25th, we made a significant architectural improvement -- we migrated some of the more intensive workloads to a separate cloud-based data storage service that is better suited to our needs. Since this migration 24 hours ago, we have seen a significant improvement with average time to fulfill authentication requests dropping by 75%, and we are no longer seeing high variability in the API response times. Customers will see more consistent and better performance from our APIs, and experience a more responsive SparkPost dashboard. We will continue to evaluate and perform additional infrastructure improvements where necessary as we stay ahead of our rapid growth.