Summary

On Friday, March 13th, some webhook batches were delayed up to 4 hours in our US region. In addition to the delay, some webhook batches were incorrectly retried even after being accepted by customers’ webhook receivers.

Timeline and Impact

March 13th, all times in UTC

• 13:45 - Webhook delivery queues begin to backup [IMPACT PERIOD BEGINS]
• 14:22 - Team is alerted to backups and initial troubleshooting and mitigation steps are taken
• 17:00 - Material progress is made and the backlog begins to clear
• 23:00 - Webhook delivery queues are back to normal state and data is no longer delayed [IMPACT PERIOD ENDS]

Root Cause

A surge in message volume combined with two latent bugs in the webhooks delivery service to cause the data processing system to fall behind. Specifically, one data processing bug severely reduced overall throughput. The other bug caused messages to be removed from the queue only after a number of attempts, which resulted in the batches being sent multiple times even after a webhook consumer responded with an HTTP 200. The potential for these bugs to cause serious issues was not fully understood and no impact had been observed recently.

Corrective Actions

• Fix stream processing & batch deletion bugs - DONE
• Add APM profiling to confirm fixes and troubleshooting future issues - DONE
• Readjust settings to prior to incident - IN PROGRESS
• Add a separate delay queue for webhooks so that failed batches will not back up new messages under extreme load & perform additional load test of the production system (Target completion: June 1st)