This incident started on June 14th after a routine deployment included a change that caused one component of the platform/sending tier to intermittently deadlock. This produced an elevated rate of timeout errors for some customers using APIs served by this tier (primarily the Transmissions API). The code change was reverted on June 19th and will not be redeployed until further analysis and testing is complete.

Human error and oversight led to the extended duration of this incident. While our Engineering team was alerted to the increased error rate, it was not clear that the error rate was caused by the June 14th deployment. We mis-identified the root cause - other factors were thought to be the cause of this incident; only after performing some standard mitigation steps, was it clear that we had not identified the true culprit. In addition, the intermittent nature of this incident made it more difficult to fully understand the impact to our customers and we were slow to report this incident.

We realize that we need to be sure to communicate elevated error rates to our customers as soon as we are aware of them even if they are intermittent and not fully understood. Furthermore, we are doing an evaluation of our deployment regression testing and monitoring in order to help us track down root cause more quickly.