Elevated 5xx Transmissions API error rates and Webhook delays for some SparkPost customers (US hosted only)

Incident Report for SparkPost

Resolved

This incident has been resolved.

Posted Sep 19, 2023 - 06:38 EDT

Update

We are continuing to monitor for any further issues.

Posted Sep 18, 2023 - 22:19 EDT

Update

We are continuing to monitor for any further issues.

Posted Sep 18, 2023 - 19:28 EDT

Monitoring

Transmissions API and Webhooks Delivery services are working within our internal SLOs.

We are monitoring the services and continue to work with our vendor to troubleshoot the issue and we will keep you updated on progress

Posted Sep 18, 2023 - 18:10 EDT

Identified

We are seeing continuing recovery on our services

5xx error rate on the Transmissions API is within our internal SLO.

Webhooks have caught up and running without delays.

We are continuing to work with our vendor to troubleshoot the issue and we will keep you updated on progress

Posted Sep 18, 2023 - 16:46 EDT

Update

We are continuing to investigate this issue.

Posted Sep 18, 2023 - 16:39 EDT

Update

We are continuing to work with our vendor to troubleshoot the issue and we are seeing some recovery.

5xx error rate on the Transmissions API is within our internal SLO. And we continue to monitor.

Webhooks are delayed by 45 minutes and catching up.

We will continue to keep you updated on progress

Posted Sep 18, 2023 - 15:59 EDT

Update

We are continuing to investigate the issue.

We are still seeing increased 5xx error on the Transmissions API (3%)

Webhooks are delayed by 60 minutes.

We are working with our vendor and will keep you posted on progress

Posted Sep 18, 2023 - 15:33 EDT

Investigating

We are experiencing an elevated level of errors on the Transmissions API (injection API) for some SparkPost customers. Please retry any 5xx error.

Our webhook data delivery services are also running behind and some customers may see a delay in data streamed to their webhook endpoints.

Note: This issue does not impact SparkPost customers hosted in the EU.

Posted Sep 18, 2023 - 14:09 EDT

This incident affected: Transmissions API (Transmissions API - USA) and Event Webhook Delivery Service (Event Webhooks - USA).