OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#11717 — p19-63-n5
Incident
Report for Network & Infrastructure
Resolved
This Nexus is no longer responding. All the traffic in this infrastructure is routed via p19-64-n5
We will investigate.
Update(s):
Date: 2014-09-29 17:17:01 UTC
The 6 FEXs are up:
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|salle5-1|T06 Online N2K-C2148T-1GE FOX1343GPE6
101 fex101|salle5-1|T03 Online N2K-C2248TP-1GE SSI14200M3B
103 fex103|salle5-1|T06 Online N2K-C2148T-1GE FOX1343GPAT
104 fex104|salle5-1|T12 Online N2K-C2148T-1GE FOX1340GDT0
105 fex105|salle5-1|T12 Online N2K-C2148T-1GE FOX1344G3QQ
106 fex106|salle5-1|T18 Online N2K-C2148T-1GE FOX1343GP67
Date: 2014-09-29 17:16:44 UTC
The switch spent a lot of time thinking and finally accepted the FEX in the 6 bays. The service is remounting.
Date: 2014-09-29 17:16:01 UTC
p19-63-n5 rebooted - it is updating the 6 impacted FEX. The service will be back up in 2 to 5 minutes.
Date: 2014-09-29 17:15:18 UTC
The 1st switch of the spare is now up. The FEX modules that the machines are connected to will go up in a few seconds.
Date: 2014-09-29 17:14:11 UTC
A climatisation module in room 5.1 broke down. Following this, it produced some water that dropped on to 2 racks. The p19-63 and p19-64 switches are hosted in two different racks but on the same climatisation module, hence the breakdown despite the redundancy in place. We will replace the switches There's no other impact.
In 2 weeks, we will remove these 2 switches to reconnect all the racks to a new central 40G system.
Date: 2014-09-29 16:03:43 UTC
The rest of the infrastructure is working without issue. For sites hosted on filers impacted by the breakdown, everything should be back up within 5 to 10 minutes.
Date: 2014-09-29 16:02:06 UTC
p19-63-n5 and p19-64-n5 are still down following a climatisation issue in the shared hosting infrastructure in Paris P19 (room 5.1). This pair of switches connects around 30% of the filers in the infrastructure and 30% of hosts.
Our teams are currently replacing one of the switches in this pair with a spare with minimal configuration in order to re-establish service asap.
Date: 2014-09-29 16:00:10 UTC
p19-64-n5 has also just gone down.
We will investigate.
Update(s):
Date: 2014-09-29 17:17:01 UTC
The 6 FEXs are up:
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|salle5-1|T06 Online N2K-C2148T-1GE FOX1343GPE6
101 fex101|salle5-1|T03 Online N2K-C2248TP-1GE SSI14200M3B
103 fex103|salle5-1|T06 Online N2K-C2148T-1GE FOX1343GPAT
104 fex104|salle5-1|T12 Online N2K-C2148T-1GE FOX1340GDT0
105 fex105|salle5-1|T12 Online N2K-C2148T-1GE FOX1344G3QQ
106 fex106|salle5-1|T18 Online N2K-C2148T-1GE FOX1343GP67
Date: 2014-09-29 17:16:44 UTC
The switch spent a lot of time thinking and finally accepted the FEX in the 6 bays. The service is remounting.
Date: 2014-09-29 17:16:01 UTC
p19-63-n5 rebooted - it is updating the 6 impacted FEX. The service will be back up in 2 to 5 minutes.
Date: 2014-09-29 17:15:18 UTC
The 1st switch of the spare is now up. The FEX modules that the machines are connected to will go up in a few seconds.
Date: 2014-09-29 17:14:11 UTC
A climatisation module in room 5.1 broke down. Following this, it produced some water that dropped on to 2 racks. The p19-63 and p19-64 switches are hosted in two different racks but on the same climatisation module, hence the breakdown despite the redundancy in place. We will replace the switches There's no other impact.
In 2 weeks, we will remove these 2 switches to reconnect all the racks to a new central 40G system.
Date: 2014-09-29 16:03:43 UTC
The rest of the infrastructure is working without issue. For sites hosted on filers impacted by the breakdown, everything should be back up within 5 to 10 minutes.
Date: 2014-09-29 16:02:06 UTC
p19-63-n5 and p19-64-n5 are still down following a climatisation issue in the shared hosting infrastructure in Paris P19 (room 5.1). This pair of switches connects around 30% of the filers in the infrastructure and 30% of hosts.
Our teams are currently replacing one of the switches in this pair with a spare with minimal configuration in order to re-establish service asap.
Date: 2014-09-29 16:00:10 UTC
p19-64-n5 has also just gone down.