IAD3 - XCP-NG Cloud System Partial Failure

IAD3 - XCP-NG Cloud System Partial Failure

Minor Resolved Started Nov 11, 2022 23:46 UTC · resolved Nov 17, 2022 13:04 UTC (5 days)

1. Resolved · Nov 17, 2022 13:04 UTC

   Closing this issue.

2. Monitoring · Nov 13, 2022 02:17 UTC

   The SAN devices in IAD3 have been repaired and all devices and SentryServers brought back online as of 9:38pm Saturday, November 12.
   This will remain in Monitoring Status for 72 hours. Please open a ticket at https://help.8dweb.com if any problems persist for your SentryServer.

3. Identified · Nov 12, 2022 17:56 UTC

   We have continued to attempt recovery of the failed SAN and backup SAN that service a number of our SentryServer clients in IAD3.
   This is affecting a handful of customers at this time. Restoration will continue as replacement hardware becomes available over the weekend.

   At this point, our intent is to have the failed customers migrated or restored from backup systems by Monday morning at 8am. We will not give any specific time-frames to specific customers since the repair and restoration of data must proceed in a particular manner.

   We know this is a major inconvenience, however, the failure of the primary and backup SAN devices for a portion of our cloud clients at IAD3 almost simultaneously has caused significant delays.

4. Investigating · Nov 11, 2022 23:46 UTC

   We have encountered what appears to be a catastrophic SAN failure in our IAD3 infrastructure. If you cannot currently reach your SentryServer it is likely it is due to this issue. There is currently NO specific timeframe for resolution as we need to work through a number of steps to assess.