Multi ISP Providers flap
Incident Report for RackNation
Resolved
VEX switch in question was updated to recommended version by J-Tac, installation was transparent, no further traffic flaps were caused during or after upgrade. This incident is marked as resolved.
Posted Mar 31, 2017 - 02:58 CST
Update
It has been detected that the issue in question was related to a bug in the JunOS version of one of our Core VEX switches, this bug confused high amounts of SNMP traffic which we use to monitor our switching appliances with a ddos, this triggered the backplane of the formentioned switch to stress out and eventually drop protocols and in some cases reboot the backplane as per Juniper Tac explanation.
We have been instructed to apply an urgent patch on this particular switch, we have moved traffic into alternate providers while we perform the switch upgrade to fix this particular bug.
Once the upgrade is finalized we will provide an update.
Posted Mar 31, 2017 - 02:08 CST
Investigating
At 11:35pm CST-6 we suffered a multiple providers flap which caused a complete blackout in providers: TIGO, ICE, Cable and Wireless
We are investigating the root cause. The flap recovery time was 3 minutes.
Posted Mar 30, 2017 - 23:57 CST