Delays on webhook batch delivery continue to go down, but are not yet completely caught up.
Posted Nov 26, 2021 - 16:14 EST
Update
Mail delivery has caught up and all messages are flowing as expected.
Webhook deliveries are still delayed.
Posted Nov 26, 2021 - 14:46 EST
Update
The delays in webhook batch delivery are reducing. Newer batches are delivering more quickly and older batches are delivering on retry, but there is still a backlog of undelivered batches.
Delays in message delivery have reduced.
Engineers are continuing to work to tune resources to improve webhook and mail delivery.
Posted Nov 26, 2021 - 14:34 EST
Identified
The metrics API is now caught up so the data in the SparkPost UI / metrics API is up to date. Webhooks deliveries continue to be impacted but the delays are reducing. The temporary failures on delivery with the error '451 4.4.2 [internal] connection closed by remote host' are reducing
Posted Nov 26, 2021 - 13:41 EST
Update
We are continuing to work on this issue. We are adding capacity which is starting to reduce the delays in providing data in the metrics API / SparkPost UI and in streaming data to webhook endpoints. Webhook batches will continue to be retried for up to 8 hours.
The number of messages getting temporary failures with the error '451 4.4.2 [internal] connection closed by remote host' is decreasing and messages are being delivered on retry.
Posted Nov 26, 2021 - 13:00 EST
Update
We are continuing to investigate this issue. We have also started to see an increase in temporary failures delivering messages with delays with the error '451 4.4.2 [internal] connection closed by remote host'. Message injection and outbound message delivery is otherwise not impacted - all messages are flowing as expected.
Posted Nov 26, 2021 - 11:40 EST
Update
Our data processing services are running behind and we're seeing intermittent slowness and elevated error rates on our Metrics API and Reporting page.
Our webhook data delivery services are running behind and some customers may see a delay in data streamed to their webhook endpoints.
We are also seeing an increase in temporary failures delivering messages with delays with the error '451 4.4.2 [internal] connection closed by remote host'
Message injection and outbound message delivery is otherwise not impacted - all messages are flowing as expected. Note: This issue does not impact our SparkPost customers hosted in the EU.
Posted Nov 26, 2021 - 11:00 EST
Investigating
Our webhook data delivery services are running behind and some customers may see a delay in data streamed to their webhook endpoints.
Message injection and outbound message delivery is not impacted - all messages are flowing as expected.
Note: This issue does not impact our SparkPost customers hosted in the EU.
Posted Nov 26, 2021 - 10:08 EST
This incident affected: Event Webhook Delivery Service (Event Webhooks - USA).