OVHCloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#7221 — vss-2-6k
Incident Report for Network & Infrastructure
Resolved
The card 4 crashed.
4 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX


Update(s):

Date: 2012-08-26 21:09:53 UTC
The router is stable.

In all we had many short interruptions during 2h30,
the necessary time to find the cards which are not good
and take them off.

Date: 2012-08-26 21:06:34 UTC
So, we progressively lost 3 cards on the router
one after the other dugin 2 hours.
It's very rare and it explains that there are
impacts during 2h30.

So
the card 4 is dead
the card 6 is dead
the card 8 is dead

That' too much.

We interrupted vss-2b-6k. We are running on vss-2-6k.
We are checking to get back the spare cards
until we receive them.

Date: 2012-08-26 21:03:18 UTC
We restarted the chassis.

Date: 2012-08-26 21:02:59 UTC
We notice ping problems on the networks connected to the replaced card and also losses on the other networks. We are ready to restart completely the chassis.

Date: 2012-08-26 15:30:52 UTC
We are still noticing anomalies on the networks routed via vss-2-6k. We are looking for the origin of the problem.

Date: 2012-08-26 15:20:06 UTC
The card is replaced.

Date: 2012-08-26 14:00:03 UTC
We are going to get back the card 8 of vss-2b (router of backup) to insert it instead of card 4 of vss-2 which is damaged.

Date: 2012-08-26 13:17:57 UTC
Aug 26 15:13:35 vss-2-6k.fr.eu 151949: Aug 26 14:13:09 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:13:35 vss-2-6k.fr.eu 151950: Aug 26 14:13:09 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:13:35 vss-2-6k.fr.eu 151951: .Aug 26 14:13:09 GMT: %XDR-6-XDRIPCNOTIFY: Message not sent to slot 4/0 (4) because of IPC error queue flush. Disabling linecard. (Expected during linecard OIR or system reloads)
Aug 26 15:13:37 vss-2-6k.fr.eu 151952: Aug 26 14:13:09 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)
Aug 26 15:13:47 vss-2-6k.fr.eu 151953: Aug 26 14:13:20 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (admin request)


It's dead. We interrupt it electrically and then we search for a spare.

Date: 2012-08-26 13:16:47 UTC
Aug 26 14:54:25 vss-2-6k.fr.eu 151891: Aug 26 13:54:00 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)

Aug 26 14:56:00 vss-2-6k.fr.eu 151893: Aug 26 12:55:35.946: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 14:56:08 vss-2-6k.fr.eu 151894: Aug 26 13:55:41 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 14:56:47 vss-2-6k.fr.eu 151895: Aug 26 13:56:22 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 14:56:47 vss-2-6k.fr.eu 151896: Aug 26 13:56:22 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 14:56:49 vss-2-6k.fr.eu 151897: Aug 26 13:56:22 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)


Aug 26 14:58:27 vss-2-6k.fr.eu 151898: Aug 26 12:58:02.843: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 14:58:33 vss-2-6k.fr.eu 151899: Aug 26 13:58:08 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 14:59:17 vss-2-6k.fr.eu 151900: Aug 26 13:58:51 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 14:59:17 vss-2-6k.fr.eu 151901: Aug 26 13:58:51 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 14:59:18 vss-2-6k.fr.eu 151902: Aug 26 13:58:51 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)
Aug 26 15:01:04 vss-2-6k.fr.eu 151903: Aug 26 14:00:37 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 15:01:04 vss-2-6k.fr.eu 151904: Aug 26 13:00:30.900: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 15:01:49 vss-2-6k.fr.eu 151905: Aug 26 14:01:24 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:01:49 vss-2-6k.fr.eu 151906: Aug 26 14:01:24 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:01:50 vss-2-6k.fr.eu 151907: Aug 26 14:01:24 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)


Aug 26 15:03:28 vss-2-6k.fr.eu 151908: Aug 26 14:03:03 GMT: %OIR-SP-6-REMCARD: Card removed from slot 4, interfaces disabled
Aug 26 15:05:02 vss-2-6k.fr.eu 151916: Aug 26 13:04:38.914: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 15:05:12 vss-2-6k.fr.eu 151917: Aug 26 14:04:44 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 15:05:54 vss-2-6k.fr.eu 151920: Aug 26 14:05:28 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:05:54 vss-2-6k.fr.eu 151921: Aug 26 14:05:28 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:05:54 vss-2-6k.fr.eu 151922: Aug 26 14:05:28 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)



Aug 26 15:07:31 vss-2-6k.fr.eu 151926: Aug 26 13:07:07.060: %SYS-DFC4-5-RESTART: System restarted --

Aug 26 15:07:40 vss-2-6k.fr.eu 151928: Aug 26 14:07:12 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 15:08:16 vss-2-6k.fr.eu 151929: Aug 26 14:07:50 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:08:16 vss-2-6k.fr.eu 151930: Aug 26 14:07:50 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:08:18 vss-2-6k.fr.eu 151931: Aug 26 14:07:50 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)
Posted Aug 26, 2012 - 13:16 UTC