OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#13460 — mtl <> bhs
Scheduled Maintenance
Report for Network & Infrastructure
Completed
We currently have 10 links between Montreal and Beauharnois down.
Traffic is going through Newark <> BHS. We are in contact with our provider. We think it may be due to a fiber cut.
Update(s):
Date: 2015-05-28 12:57:27 UTC
The 3 pairs of optical fiber cables are up. Huge service interruptions lasted 1 hour 40 - longer than we expected. We will continue to weld other tubes, but we're no longer impacted.
We're sorry for the incident. We're continuing to work on fiber 2) and 3) which will enable us to fully secure the BHS - Montreal connection, and the BHS - New York connection.
Date: 2015-05-28 12:34:11 UTC
MTL/NWK 20x10G Level3 is up
Date: 2015-05-28 12:33:50 UTC
MTL/BHS 1x100G is UP.
Date: 2015-05-28 12:33:14 UTC
Our BHS/MTL 6x100G link is up.
Date: 2015-05-28 12:32:49 UTC
The welding is bad quality. We need to redo it. We will cut everything off, we will breathe and start again.
Welding is in progress. One of the fibers in our tube is up. We're receiving signal from BHS to MTL. We're waiting to receive it in the other direction.
Date: 2015-05-28 12:03:12 UTC
During the work, the connection seriously downgraded :( We're getting packet loss
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. p19-55-6k.fr.eu 0.5% 392 2.1 19.7 0.2 279.1 51.0
2. p19-2-6k.fr.eu 0.0% 392 1.8 14.4 0.2 219.2 38.7
3. th2-g1-a9.fr.eu 0.0% 392 1.0 1.0 0.6 34.0 1.8
4. rbx-g1-a9.fr.eu 0.0% 392 4.5 4.9 4.2 11.3 0.9
5. ldn-1-6.uk.eu 99.7% 392 104.6 104.6 104.6 104.6 0.0
6. nwk-1-6k.nj.us 0.0% 392 76.7 76.7 76.5 83.9 0.6
7. mtl-1-6k.qc.ca 0.0% 392 84.0 93.5 83.8 265.8 31.1
8. bhs-g1-6k.qc.ca 7.2% 392 138.9 139.8 138.6 155.6 1.4
9. bhs-3a-a9.qc.ca 4.9% 391 140.7 141.0 139.0 163.1 2.3
10. bhs.proof.ovh.net 5.4% 391 141.8 141.7 141.0 150.5 1.1
Date: 2015-05-28 12:02:16 UTC
Work should be complete in a few minutes.
Date: 2015-05-28 12:01:56 UTC
Work is continuing. We're welding, we're welding ...
Date: 2015-05-28 10:53:16 UTC
The fibers have been cut.
Date: 2015-05-28 09:13:07 UTC
We've started to install the new cable. The welders are about to insert a new optical fiber cable into the connection box. We will then prepare the cable. We estimate that the cut-off will happen around 5:00AM (11.00 GMT+2)
Date: 2015-05-28 09:07:38 UTC
Hello,
There is currently a very delicate situation at our datacentre in BHS. We have 4 optical fiber cables that connect the datacenter to the world.
1) BHS to Montreal via the North. The link is up and we have 3 pairs of optical fiber cables: to Montreal, New York (600G capacity) and to level 3 (which provides 200G capacity to New York)
2) 2) BHS to Montreal via the South. The link is not yet up. Construction of the connection is delayed in the Indian reserve. We will have 3 pairs of optical fiber cables by July 2015
3) BHS to New York via the South. This link is not yet up either. It takes a particularly long time to get approval to perform work on the Canada/USA border: 3 years. The go-live date: September 2015
4) A backup of the MTL/BHS backup is UP, but with low capacity (10G).
Last night, a car hit one of the poles along the BHS/MTL via the North route, damaging the optical fiber cable: 2 of the 3 pairs of cables are down. The 3rd one, to Level 3, is still UP.
We’ve identified where the incident happened and the maintenance teams are currently working to feed a new 96 FO cable at 300m. This will take 12 hours.
Around 7-8am local time (1-2pm GMT+2), we will begin maintenance on the cable: we will cut off the optical fiber cables that are still up to weld them on to the new cable. To reduce the welding time, we have 2 teams on site at both ends of the cable. We estimate that it will take 30 minutes to cut off and remount our tube of optical fiber cables, in which we have 3 pairs of optical fiber cables. During this intervention, the datacenter will remain connected, but only via the backup of the backup, and therefore the connection will be seriously downgraded.
We will post more information before the work begins.
Best,
Octave Klaba
CTO
Date: 2015-05-28 04:37:57 UTC
Ovh network need an urgent repair, a 30 minute service interruption is necessary during this time the service will be degraded. (no ETA on when) We are working closely with our technicians on the ground to resolve this issue.
Date: 2015-05-28 02:30:21 UTC
The technicians are on site and looking for the damaged fiber.
Date: 2015-05-28 01:30:02 UTC
It's an incident with the cable but not a clear cut. Several fibers are affected but not the whole cable. The problem could be caused by a rodent.
Since there are available pairs, the repair of the cable is not expected immediately. The maintainer will transfer the affected links to other pairs in good health. It's expected therefore a recovery of the links BHS <> Montreal in a few hours. No accurate ETA can be provided in the current situation.
Traffic is going through Newark <> BHS. We are in contact with our provider. We think it may be due to a fiber cut.
Update(s):
Date: 2015-05-28 12:57:27 UTC
The 3 pairs of optical fiber cables are up. Huge service interruptions lasted 1 hour 40 - longer than we expected. We will continue to weld other tubes, but we're no longer impacted.
We're sorry for the incident. We're continuing to work on fiber 2) and 3) which will enable us to fully secure the BHS - Montreal connection, and the BHS - New York connection.
Date: 2015-05-28 12:34:11 UTC
MTL/NWK 20x10G Level3 is up
Date: 2015-05-28 12:33:50 UTC
MTL/BHS 1x100G is UP.
Date: 2015-05-28 12:33:14 UTC
Our BHS/MTL 6x100G link is up.
Date: 2015-05-28 12:32:49 UTC
The welding is bad quality. We need to redo it. We will cut everything off, we will breathe and start again.
Welding is in progress. One of the fibers in our tube is up. We're receiving signal from BHS to MTL. We're waiting to receive it in the other direction.
Date: 2015-05-28 12:03:12 UTC
During the work, the connection seriously downgraded :( We're getting packet loss
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. p19-55-6k.fr.eu 0.5% 392 2.1 19.7 0.2 279.1 51.0
2. p19-2-6k.fr.eu 0.0% 392 1.8 14.4 0.2 219.2 38.7
3. th2-g1-a9.fr.eu 0.0% 392 1.0 1.0 0.6 34.0 1.8
4. rbx-g1-a9.fr.eu 0.0% 392 4.5 4.9 4.2 11.3 0.9
5. ldn-1-6.uk.eu 99.7% 392 104.6 104.6 104.6 104.6 0.0
6. nwk-1-6k.nj.us 0.0% 392 76.7 76.7 76.5 83.9 0.6
7. mtl-1-6k.qc.ca 0.0% 392 84.0 93.5 83.8 265.8 31.1
8. bhs-g1-6k.qc.ca 7.2% 392 138.9 139.8 138.6 155.6 1.4
9. bhs-3a-a9.qc.ca 4.9% 391 140.7 141.0 139.0 163.1 2.3
10. bhs.proof.ovh.net 5.4% 391 141.8 141.7 141.0 150.5 1.1
Date: 2015-05-28 12:02:16 UTC
Work should be complete in a few minutes.
Date: 2015-05-28 12:01:56 UTC
Work is continuing. We're welding, we're welding ...
Date: 2015-05-28 10:53:16 UTC
The fibers have been cut.
Date: 2015-05-28 09:13:07 UTC
We've started to install the new cable. The welders are about to insert a new optical fiber cable into the connection box. We will then prepare the cable. We estimate that the cut-off will happen around 5:00AM (11.00 GMT+2)
Date: 2015-05-28 09:07:38 UTC
Hello,
There is currently a very delicate situation at our datacentre in BHS. We have 4 optical fiber cables that connect the datacenter to the world.
1) BHS to Montreal via the North. The link is up and we have 3 pairs of optical fiber cables: to Montreal, New York (600G capacity) and to level 3 (which provides 200G capacity to New York)
2) 2) BHS to Montreal via the South. The link is not yet up. Construction of the connection is delayed in the Indian reserve. We will have 3 pairs of optical fiber cables by July 2015
3) BHS to New York via the South. This link is not yet up either. It takes a particularly long time to get approval to perform work on the Canada/USA border: 3 years. The go-live date: September 2015
4) A backup of the MTL/BHS backup is UP, but with low capacity (10G).
Last night, a car hit one of the poles along the BHS/MTL via the North route, damaging the optical fiber cable: 2 of the 3 pairs of cables are down. The 3rd one, to Level 3, is still UP.
We’ve identified where the incident happened and the maintenance teams are currently working to feed a new 96 FO cable at 300m. This will take 12 hours.
Around 7-8am local time (1-2pm GMT+2), we will begin maintenance on the cable: we will cut off the optical fiber cables that are still up to weld them on to the new cable. To reduce the welding time, we have 2 teams on site at both ends of the cable. We estimate that it will take 30 minutes to cut off and remount our tube of optical fiber cables, in which we have 3 pairs of optical fiber cables. During this intervention, the datacenter will remain connected, but only via the backup of the backup, and therefore the connection will be seriously downgraded.
We will post more information before the work begins.
Best,
Octave Klaba
CTO
Date: 2015-05-28 04:37:57 UTC
Ovh network need an urgent repair, a 30 minute service interruption is necessary during this time the service will be degraded. (no ETA on when) We are working closely with our technicians on the ground to resolve this issue.
Date: 2015-05-28 02:30:21 UTC
The technicians are on site and looking for the damaged fiber.
Date: 2015-05-28 01:30:02 UTC
It's an incident with the cable but not a clear cut. Several fibers are affected but not the whole cable. The problem could be caused by a rodent.
Since there are available pairs, the repair of the cable is not expected immediately. The maintainer will transfer the affected links to other pairs in good health. It's expected therefore a recovery of the links BHS <> Montreal in a few hours. No accurate ETA can be provided in the current situation.