IAD3 Outage
- Resolved ·
This matter is now closed.
- Monitoring ·
...
- Identified ·
...
- Identified ·
As of 8:56am Most IP connectivity has been restored. We are leaving this issue in critical status for the next 90 minutes.
- Identified ·
Update from Data Center Personnel @ 8:15am: Please be advised that the maintenance activity that was originally scheduled to run between 10/27/23 00:00 EDT and 10/27/23 04:00 EDT has been extended for an additional two (2) hours due to unforeseen circumstances. Further updates will be provided as they become available. A final completion notice will only be sent once this maintenance has been confirmed completed. If you are experiencing any manner of service interruption as a result of this maintenance, rest assured that this event is being actively addressed by our senior engineers and that every effort is being made to restore your services as quickly as is possible.
- Identified ·
Update from Data Center Personnel @7:03am: Please be advised that this maintenance activity that began between 10/27/23 00:00 Eastern and 10/27/23 04:00 Eastern has been extended due to unforeseen circumstances. Further updates will be provided as they become available. A final completion notice will only be sent once this maintenance has been confirmed completed.
- Identified ·
Some customers are seeing restored IP connectivity, however, most of our IAD3 network remains offline at this moment.
- Identified ·
The 6am Window has now arrived with no resolution. Awaiting further updates from Data Center Personnel.
- Identified ·
Update from Data Center Personnel: Please be advised that this maintenance activity that began between 10/27/2023 at 00:01 AM EDT and 10/27/2023 at 4:00 AM EDT has been extended to 10/27/2023 at 6:00 AM EDT due to unforeseen circumstances. Further updates will be provided as they become available. A final completion notice will only be sent once this maintenance has been confirmed completed.
- Identified ·
There has been no word nor any replies to 8Dweb inquiries from Data Center personnel for the IAD3 outage for one hour. All Primary and Secondary network connectivity remains offline at this time.
- Identified ·
Here is the latest from the Data Center Personnel:
"Please be advised that this maintenance activity that began between 10/27/2023 at 00:01 AM EDT and 10/27/2023 at 4:00 AM EDT has been extended due to unforeseen circumstances. Further updates will be provided as they become available." - Identified ·
The window has now progressed more than 15 minutes past what was promised. Data center personnel had indicated it would be less than the window, then would be the window, and now the time has extended. 8D personnel are awaiting clarification and response from 365 at this time. Please be advised that we will keep this notice updated as we gain information.
- Identified ·
...
- Investigating ·
At this moment, despite being notified in writing that this event would be "shorter" than expected (once again 8D received no prior notice - the data center team is working to rectify that matter) - it appears the event may last the full 240 minute window.
- Investigating ·
The data center team has responded that this was a planned event - However - it is confirmed that no email notice regarding such a major event was provided. Here is the information received: Facility(s): IAD3 – Reston, VA
Ticket #: 2671997
Maintenance Type:
Scheduled Network MaintenancePlease be advised that from
approximately/27/2023 at 00:01 AM EDT to 10/27/2023 at 4:00 AM EDT, 365 Data
Centers will be conducting scheduled network maintenance in the Reston, VA
area.Please see the details below:
Duration of Window: 4 Hours
Maintenance
Location: Reston, VAReason for Maintenance: Core network switch replacement
Customer Estimated Impact & Duration:
You are directly connected to a
device that will be impacted by this maintenance. Your service is expected to
be
impacted, which may result in a loss of connectivity of up to 240 minutes - Investigating ·
It has taken our team some time to contact data center personnel. The issue as they report is that there is a maintenance window at the Coresite facility. This maintenance
window was not previously announced, but does not appear to be an emergency window. Information is still flowing. - Investigating ·
There is an as yet unexplained network outage at IAD3 - CR7. Please be advised that our team is still seeking information on this issue.