The webhook issue has been resolved. All of the queues have caught up and there should be no further delays.
Posted Nov 18, 2022 - 17:42 EST
Update
The webhook event data queues are being processed, we expect to be fully caught up in 1-2 hours from now. We will marked this incident as resolved when all the queues are processed.
Posted Nov 18, 2022 - 17:00 EST
Update
The webhook event data queues are being processed, we expect to be fully caught up in 4 - 5 hours. We will marked this incident as resolved when all the queues are processed.
Posted Nov 18, 2022 - 16:09 EST
Monitoring
We have implemented a fix and the backlog is beginning to drain, it will take some time to process all queued events.
Posted Nov 18, 2022 - 13:52 EST
Update
Webhook delivery streams are operational but data continues to run behind - up to 100 minutes for some customers and some batches. We are working to optimize the data processing services and will provide regular updates.
Posted Nov 18, 2022 - 12:27 EST
Update
Webhook delivery streams are operational but data is running behind - approximately 75 minutes for some customers and some batches. We are working to optimize the data processing services and will provide regular updates.
Posted Nov 18, 2022 - 11:30 EST
Update
Webhook delivery streams are operational but data is running behind - approximately 45 minutes for some customers and some batches. We are working to optimize the data processing services and will provide regular updates.
Posted Nov 18, 2022 - 10:22 EST
Identified
Webhook delivery streams are operational but data is running behind - approximately 20 minutes for some customers and some batches. We are working to optimize the data processing services and will provide regular updates.
Posted Nov 18, 2022 - 09:02 EST
Monitoring
A fix has been implemented and we are monitoring results.
Posted Nov 18, 2022 - 00:21 EST
Update
We are continuing to investigate this issue. Webhook delivery streams are operational and there is currently no data delay.
Posted Nov 17, 2022 - 23:26 EST
Update
We are continuing to investigate this issue. Webhook delivery streams are operational but data is running behind (up to 30 minutes for some customers and some batches).
Posted Nov 17, 2022 - 22:02 EST
Update
We are continuing to investigate this issue. Webhook delivery streams are operational but data is running behind (up to 50 minutes for some customers and some batches).
Posted Nov 17, 2022 - 21:04 EST
Update
We are continuing to investigate this issue. Webhook delivery streams are operational but data is running behind (up to 55 minutes for some customers and some batches).
Posted Nov 17, 2022 - 20:05 EST
Update
We are continuing to investigate this issue. Webhook delivery streams are operational but data is running behind (up to 40 minutes for some customers and some batches).
Posted Nov 17, 2022 - 18:05 EST
Investigating
Our webhook data delivery services are running behind and some customers may see a delay in data streamed to their webhook endpoints.
Message injection and outbound message delivery is not impacted - all messages are flowing as expected.
Note: This issue does not impact our SparkPost customers hosted in the EU.
Posted Nov 17, 2022 - 16:09 EST
This incident affected: Event Webhook Delivery Service (Event Webhooks - USA).