Date: 2014-09-29 17:16:44 UTC The switch spent a lot of time thinking and finally accepted the FEX in the 6 bays. The service is remounting.
Date: 2014-09-29 17:16:01 UTC p19-63-n5 rebooted - it is updating the 6 impacted FEX. The service will be back up in 2 to 5 minutes.
Date: 2014-09-29 17:15:18 UTC The 1st switch of the spare is now up. The FEX modules that the machines are connected to will go up in a few seconds.
Date: 2014-09-29 17:14:11 UTC A climatisation module in room 5.1 broke down. Following this, it produced some water that dropped on to 2 racks. The p19-63 and p19-64 switches are hosted in two different racks but on the same climatisation module, hence the breakdown despite the redundancy in place. We will replace the switches There's no other impact.
In 2 weeks, we will remove these 2 switches to reconnect all the racks to a new central 40G system.
Date: 2014-09-29 16:03:43 UTC The rest of the infrastructure is working without issue. For sites hosted on filers impacted by the breakdown, everything should be back up within 5 to 10 minutes.
Date: 2014-09-29 16:02:06 UTC
p19-63-n5 and p19-64-n5 are still down following a climatisation issue in the shared hosting infrastructure in Paris P19 (room 5.1). This pair of switches connects around 30% of the filers in the infrastructure and 30% of hosts.
Our teams are currently replacing one of the switches in this pair with a spare with minimal configuration in order to re-establish service asap.
Date: 2014-09-29 16:00:10 UTC p19-64-n5 has also just gone down.