The replays have finished running. All historical engagement data is fully caught up.
Posted Feb 24, 2024 - 13:25 EST
Update
The replays continue to run nominally. We estimate that the replays will complete within the next 4-5 hours.
Posted Feb 24, 2024 - 10:58 EST
Update
The replays continue to run nominally. We anticipate historical engagement data will be fully caught up in the next 12-14 hours. We are intentionally throttling the replay to ensure we do not overwhelm customers' webhook endpoints. While we cannot guarantee ordering, we are generally sending the data from oldest to most recent.
The next update will be provided at the completion of the replay or early if needed.
Posted Feb 24, 2024 - 00:11 EST
Update
The replays continue to run nominally. To optimize the performance of the replay we are going to slightly change the manner in which we are sending the replayed data. Starting around 11PM ET we will slow the total overall rate limits slightly, but continue sending historical events without a pause between hours. We do not anticipate any impact to customers endpoints due to this change.
Posted Feb 23, 2024 - 22:54 EST
Monitoring
We started replaying historical missed data at 8:30ET. Customers will begin to see missed events delivered to their webhooks and continuing until all data has been replayed. We plan to replay 1 hour of open and click events from 2/22 1800UTC through (and including) 2/23 19:00UTC with a 5 minute pause between each hour of data.
We will provide future updates to this status page as needed, or at the completion of the replay.
Posted Feb 23, 2024 - 20:49 EST
Update
All of the tests for the replay process have been validated. We will begin replays at 8:30ET/7:30CT. We plan to replay 1 hour of data at a time, starting from the latest hour and working up to present time, with a pause of 5 minutes between each hour of data.
Posted Feb 23, 2024 - 20:15 EST
Update
The team is continuing to test and validate the replay process to for historical open and click events. We will post an update to this status page in the next 45 minutes with the exact start time that replays will begin. We plan to replay 1 hour of data at a time, starting from the latest hour and working up to present time, with a pause of 5 minutes between each hour of data.
Posted Feb 23, 2024 - 19:43 EST
Update
We have verified that the fix is working as expected and customers are seeing generation of live open and click events. We are actively finalizing testing on the replays and anticipate starting the recovery of historical open and click events within the next few hours. We plan to replay 1 hour of data at a time, starting from the latest hour and working up to present time, with a pause of 5 minutes between each hour of data.
Posted Feb 23, 2024 - 15:48 EST
Update
We are starting the deployment of this fix to production. We are anticipating that the production deployment should be completed within the next 20-30 minutes. We will provide an update as soon as the deployment is completed.
Posted Feb 23, 2024 - 14:43 EST
Update
The team encountered some unexpected delays in the deployment of the fixes. We are continuing to work on deploying the fixes to production. We will provide an updated estimate of the resolution within the next 30 minutes.
Posted Feb 23, 2024 - 14:23 EST
Update
The team has identified the fix for this issue and are actively working to deploy a fix to our production clusters. We anticipate that new open and click events will start processing within the next 60-90 minutes. Additionally, our team has confirmed that there was NO data loss due to this issue and are working on a replay of the missing data. We anticipate that replayed data will begin being shipped within the next 4-6 hours.
Posted Feb 23, 2024 - 10:40 EST
Update
We are continuing to work on a fix for this issue.
Posted Feb 23, 2024 - 09:14 EST
Update
We are continuing to work on a fix for this issue.
Posted Feb 23, 2024 - 09:13 EST
Identified
The issue has been identified and a fix is being implemented.
Posted Feb 23, 2024 - 09:10 EST
Investigating
Our data processing services are running behind; some customers may see a delay of data available via the:
Message injection and outbound message delivery are not impacted - all messages are flowing as expected. Note: This issue does not impact our SparkPost customers hosted in the EU region.
Posted Feb 23, 2024 - 08:10 EST
This incident affected: Events API (Events API - USA), SparkPost Application (WebUI) (SparkPost Application - USA), and Event Webhook Delivery Service (Event Webhooks - USA).