- Status Closed
- Percent Complete
- Task Type Tasks
- Category DataCenter / DC
- Assigned To No-one
- Operating System All
- Severity Low
- Priority Medium
- Reported Version Development
- Due in Version Undecided
-
Due Date
Undecided
- Votes
- Private
FS#27 - Electrical maintenance from 12/28/2015
Following a problem with one of our electrical panels, we had to perform emergency maintenance on one of our hosting rooms (nearly 500 services). We therefore notified our clients of a 45-minute power outage 30 minutes beforehand. Right in the middle of the holiday season, with a reduced team.
Unfortunately, during the 30-minute restoration, one of our core network switches rebooted with an old configuration, causing a global incident that was difficult to manage. As a result, we experienced an incident that lasted not just 45 minutes in a single room, but more than two hours across the entire OrdCom network.
We had to check the services one by one. A significant number of machines did not restart properly. Configuration issues, disk problems, etc., were the cause. Also, some didn't actually restart until late in the day.
Finally, due to this important recovery work and the period, we were unable to communicate properly and we apologize for this.
RSS