Elevated 5xx Events API error rates and data delays for some SparkPost and SparkPost Enterprise customers (US hosted only)

Incident Report for SparkPost

Resolved

This incident has been resolved.
Posted Nov 17, 2022 - 21:00 EST

Monitoring

The event data queues are caught up.
Posted Nov 17, 2022 - 20:36 EST

Update

We are processing the backlog of events data; we expect to be caught up in 1 hour.
Posted Nov 17, 2022 - 19:59 EST

Identified

The issue has been identified and we are working on a fix.
Posted Nov 17, 2022 - 18:52 EST

Update

We are continuing to work on this issue and will continue to provide updates; customers in the US will continue to see intermittent slowness and elevated error rates on our Events API and Events in the UI.

Message injection and outbound message delivery are not impacted - all messages are flowing as expected.

Note: This issue does not impact our SparkPost customers hosted in the EU.
Posted Nov 17, 2022 - 18:08 EST

Update

We are continuing to work on this issue and will continue to provide updates; customers in the US will continue to see intermittent slowness and elevated error rates on our Events API and events in the UI.

Message injection and outbound message delivery are not impacted - all messages are flowing as expected.
Note: This issue does not impact our SparkPost customers hosted in the EU.
Posted Nov 17, 2022 - 16:15 EST

Update

We are continuing to work on this issue; customers in the US will see intermittent slowness and elevated error rates on our Events API and events in the UI.

Message injection and outbound message delivery are not impacted - all messages are flowing as expected. Webhook events are also not impacted.
Note: This issue does not impact our SparkPost customers hosted in the EU.
Posted Nov 17, 2022 - 11:09 EST

Investigating

Our data processing services are running behind and we're seeing intermittent slowness and elevated error rates on our Events API. Specifically, some customers may see latency issues, 500 errors or a delay of data on:

SparkPost App (webUI)
Events API

Message injection and outbound message delivery are not impacted - all messages are flowing as expected. Webhook events are also not impacted.
Note: This issue does not impact our SparkPost customers hosted in the EU.
Posted Nov 17, 2022 - 09:19 EST
This incident affected: Events API (Events API - USA) and SparkPost Application (WebUI) (SparkPost Application - USA).