Resolved -
This incident is resolved, and delivery to Microsoft mailboxes is back to normal.
From about 17:30 UTC on 12 November, Microsoft's mail service returned temporary errors for most delivery attempts. The cause was an incident on Microsoft's side (Microsoft incident EX1550543), and delivery to other mailbox providers was not affected. Microsoft's errors dropped sharply from about 13:30 UTC on 13 November. Delivery was back to normal rates by 14:35 UTC, and Microsoft's error levels have been normal since about 15:10 UTC (09:10 CST).
We queued all affected mail and retried it automatically. Most of it has now been delivered, and the rest of the retry queue will finish over the next few hours. You don't need to do anything, and you don't need to resend.
If your messages use a short expiration (TTL) setting, some may have expired before Microsoft accepted them. Those messages were returned as bounces and appear in your bounce events. You may want to resend any time-sensitive messages that expired.
Thank you for your patience while Microsoft resolved this issue.
Nov 13, 11:20 EST
Update -
Delivery to Microsoft (Outlook.com, Hotmail and Office 365) has recovered
Delivery to Microsoft mailboxes is back to normal. Starting around 17:30 UTC on 12 November, an incident on Microsoft's side (Microsoft incident EX1550543) caused delivery delays. The errors from that incident have now returned to normal levels, and new mail to Microsoft is being accepted at normal rates and speed.
We are finishing the automatic retry of mail that was delayed during the incident. Most of it has already been delivered, and the rest is draining steadily. Some older delayed messages may still arrive over the next while. You don't need to do anything, and you don't need to resend.
If your messages use a short expiration (TTL) setting, some may have expired before Microsoft accepted them. Those messages were returned as bounces and appear in your bounce events.
Nov 13, 10:27 EST
Update -
Delivery delays to Microsoft (Outlook.com, Hotmail and Office 365)
Microsoft's mail service has been degraded since about 17:30 UTC on 12 November (Microsoft incident EX1550543). Delivery to Microsoft has improved a lot over the past few hours. Since about 13:30 UTC (07:30 CST), almost all of the errors from the outage itself have stopped.
Microsoft is still deferring some mail while it clears its own backlog, so some messages to Microsoft mailboxes may still be delivered late. SparkPost is retrying all deferred mail automatically. You don't need to do anything, and you don't need to resend.
If your messages use a short expiration (TTL) setting, some of them may have expired before Microsoft accepted them. Those messages were returned as bounces, and you'll see them in your bounce events. This issue was on Microsoft's side and did not affect delivery to other mailbox providers.
We'll keep monitoring and will post another update once delivery to Microsoft is back to normal.
Nov 13, 09:33 EST
Monitoring -
Delivery to Microsoft (Outlook.com, Hotmail, Microsoft 365) is recovering
Microsoft is still recovering from its service incident (Microsoft reference EX1550543). The problem was on Microsoft's side. Our platform has run normally throughout, and delivery to other mailbox providers was not affected.
Since about 11:30 UTC (5:30 AM US Central), delivery to Microsoft recipients has improved a lot. Delivery volume is back in its normal range. We're also working through the backlog of mail that was queued during the outage, and those retries make up a large share of what Microsoft is accepting right now. About 3 in 10 delivery attempts to Microsoft are still being deferred, compared with more than 9 in 10 at the worst point. That's still above normal, so some messages to Microsoft recipients may take longer than usual to arrive while Microsoft finishes recovering.
Microsoft reports that recovery is moving forward across Europe, Asia Pacific and North America, and that some users may still see intermittent issues.
What this means for you: You don't need to do anything. We'll keep retrying any deferred mail automatically. If you set short expiration windows on your messages, some messages to Microsoft recipients may have expired before Microsoft accepted them.
We'll keep watching delivery to Microsoft closely and will post again once it's fully back to normal.
Nov 13, 07:45 EST
Update -
Delivery delays to Microsoft (Outlook.com, Hotmail and Microsoft 365)
Microsoft is still having a service incident (Microsoft reference EX1550543). Mail sent to Outlook.com, Hotmail and Microsoft 365 recipients is still being delayed. The problem is on Microsoft's side. Their servers are still turning away most delivery attempts with temporary errors. Our platform is working normally, and delivery to all other mailbox providers is unaffected.
As of 04:45 UTC (10:45 PM US Central), about three in four delivery attempts to Microsoft are still being deferred. Microsoft reports partial recovery in some regions but says it has not held, and they are still working on a fix.
What this means for you: You don't need to do anything. We keep accepting your mail, queue anything Microsoft defers and retry it automatically. Most mail is still getting through, but more slowly than usual. If you have set a short expiration on your messages, some messages to Microsoft recipients may expire before Microsoft accepts them.
Next update: This is our last scheduled update tonight. Our automated monitoring is still watching delivery to Microsoft around the clock. We will post again as soon as we see Microsoft's servers sustaining recovery, or tomorrow morning (US Central).
Nov 12, 23:50 EST
Update -
Microsoft reports that the changes it applied to restore service are not working as expected, and it is pursuing other ways to add capacity. Delivery to Outlook.com, Hotmail, Live, MSN and Office 365 recipients remains significantly delayed, and about two-thirds of delivery attempts to Microsoft are still being deferred. Microsoft has not given an estimated time for recovery. The incident is tracked as EX1550543, and Microsoft's updates are at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect.
SparkPost's systems are working normally. Messages still waiting for Microsoft remain queued and are being retried automatically. Please don't resend them, because that will create duplicates once they're delivered. The problem has now lasted nearly ten hours, so messages with expiration times of several hours or more may expire before Microsoft accepts them. Each expired message generates a bounce event, and since expired messages are no longer queued, you can decide whether to resend those.
Nov 12, 22:19 EST
Update -
Delivery to Outlook.com, Hotmail, Live, MSN and Office 365 recipients is improving, but it hasn't fully recovered. Microsoft has restarted affected servers, is adding capacity, and reports that some regions are coming back online. Over the past hour the rate of temporary errors from Microsoft has dropped by about half, and more queued mail is being delivered on retry. Many delivery attempts are still being deferred, so recipients at Microsoft may keep seeing delays for a while. Microsoft is tracking the issue as EX1550543 and posting updates at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect.
SparkPost's systems are working normally. Messages still waiting for Microsoft remain queued and will keep being retried automatically. Please don't resend them, because that will create duplicates once they're delivered. Messages whose expiration time passed during the outage have expired and generated a bounce event. They are no longer queued, so you can decide whether to resend those.
Nov 12, 22:10 EST
Update -
Microsoft has identified a likely cause of incident EX1550543. A process on its servers that establishes secure connections is becoming overwhelmed, which stops the servers from handling requests. Microsoft has started rolling back a recent network configuration change that coincides with the start of the problem. It is also testing a fix that it plans to apply across all affected servers. Microsoft expects to post its next update by 02:00 UTC, and you can follow it at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect.
Delivery to Outlook.com, Hotmail, Live, MSN and Office 365 recipients is still heavily delayed. More mail is getting through than earlier this evening, but most delivery attempts to Microsoft are still being deferred, and we have not yet seen the effect of Microsoft's changes.
SparkPost's systems are working normally. Messages still waiting for Microsoft are queued and retried automatically, so please don't resend them, because that would create duplicates once they are delivered. Because the problem has now lasted more than seven hours, messages with an expiration time of up to several hours may expire before Microsoft accepts them. Each expired message generates a bounce event, and since expired messages are no longer queued, you can decide whether to resend those specific messages.
Nov 12, 20:11 EST
Update -
We're still monitoring Microsoft incident EX1550543. Microsoft says it has begun bringing affected servers back into service and is seeing limited recovery in some regions. We're seeing a gradual improvement too. More mail is getting through to Outlook.com, Hotmail, Live, MSN and Office 365 recipients than an hour ago, but most delivery attempts to Microsoft are still being deferred. Delays will continue until Microsoft completes its recovery. Microsoft expects to post its next update by 00:30 UTC, and you can follow it at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect.
SparkPost's systems are working normally. Messages still waiting for Microsoft are queued and retried automatically, so please don't resend them. That would create duplicates once they are delivered. Because the problem has now lasted more than six hours, messages with an expiration time of up to several hours may expire before Microsoft accepts them. Each expired message generates a bounce event in your event data. Expired messages are no longer queued, so you can decide whether to resend those specific messages.
Nov 12, 18:54 EST
Update -
We're still monitoring Microsoft incident EX1550543. Delivery to Outlook.com, Hotmail, Live, MSN and Office 365 recipients is still heavily delayed, and we haven't seen a meaningful improvement.
In its latest update, Microsoft said its earlier recovery steps did not bring the expected relief. It has traced the problem to a core process on its servers that is becoming overwhelmed. Microsoft is now isolating part of its infrastructure, restarting it and slowly bringing traffic back to test a recovery approach before applying it more widely. It has not given a time for recovery, and its next update is expected by 23:30 UTC. You can follow it at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect.
SparkPost's systems are working normally. We're still queuing messages to Microsoft and retrying them automatically, and some continue to get through on a retry. Please don't resend affected messages, because that will create duplicates once the queued mail is delivered. Because this has now lasted several hours, messages with an expiration time of up to a few hours may also expire before Microsoft accepts them, not only very short ones. You'll get a bounce event for each expired message.
Nov 12, 17:26 EST
Update -
We're still monitoring the Microsoft incident (EX1550543). Delivery to Outlook.com, Hotmail, Live, MSN and Office 365 recipients is still heavily delayed, and we haven't seen a meaningful improvement yet.
Microsoft says it has found a way to bring a test group of its affected servers back into service and is now rolling that fix out more widely. It is also restarting servers that stop responding. Delivery may improve gradually as that rollout continues, and Microsoft expects to post its next update by 22:30 UTC. You can follow it at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect.
SparkPost is still queuing and retrying affected messages automatically, and some continue to get through on a retry. Please don't resend them. Messages with a short expiration time may still expire before Microsoft accepts them.
Nov 12, 16:37 EST
Identified -
Microsoft has confirmed an incident affecting Exchange Online and Microsoft 365 connectivity in all regions. It’s tracked as EX1550543, and you can follow Microsoft’s updates at https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect.
Since about 17:30 UTC on November 12, Microsoft’s mail servers have been answering a large share of delivery attempts with temporary errors. This affects Outlook.com, Hotmail, Live, MSN and Office 365 recipients. The errors include “451 4.7.0 Temporary server error”, “421 4.3.2 The maximum number of concurrent server connections has exceeded a limit” and “451 4.7.700 PFA agent busy”. The problem is on Microsoft’s side and affects senders across the industry. SparkPost’s own systems are working normally.
Affected messages are not lost. SparkPost queues them and keeps retrying automatically, and many are already going through on a retry. You’ll see these attempts as delay events in your event data. Recipients at Microsoft may get mail later than usual until Microsoft resolves EX1550543. Delivery to other mailbox providers is not affected.
You don’t need to do anything, and please don’t send these messages again. Resending creates duplicates once the queued mail goes through. If you’ve set a short message expiration (for example on one-time passcodes), those messages may expire before Microsoft recovers.
We’re watching this closely and will post an update as soon as things change.
Nov 12, 15:01 EST
