A contributing factor was a bug in JunOS 10.4, which responded to a broadcast BGP route with an invalid attribute by restarting BGP sessions or dumping core and crashing. Joy!
A summary is available here: Time Warner Cable outage traced to Level 3 router glitch and the vendor note follows in gory detail.
1. View Bulletin PSN-2011-08-327
2. Title MX Series MPC crash in Ktree::createFourWayNode after BGP UPDATE
3. Products Affected This issue can affect any MX Series router with port concentrators based on the Trio chipset -- such as the MPC or embedded into the MX80 -- with active protocol-based route prefix additions/deletions occurring.
4. Platforms Affected
6. JUNOS 11.x
8. JUNOS 10.x
9. SIRT Security Advisory
10. SIRT Security Notice
11. Revision Number 1
12. Issue Date 2011-08-08
14. PSN Issue :
15. MPCs (Modular Port Concentrators) installed in an MX Series router may crash upon receipt of very specific and unlikely route prefix install/delete actions, such as a BGP routing update. The set of route prefix updates is non-deterministic and exceedingly unlikely to occur. Junos versions affected include 10.0, 10.1, 10.2, 10.3, 10.4 prior to 10.4R6, and 11.1 prior to 11.1R4. The trigger for the MPC crash was determined to be a valid BGP UPDATE received from a registered network service provider, although this one UPDATE was determined to not be solely responsible for the crashes. A complex sequence of preconditions is required to trigger this crash. Both IPv4 and IPv6 routing prefix updates can trigger this MPC crash.
17. There is no indication that this issue was triggered maliciously. Given the complexity of conditions required to trigger this issue, the probability of exploiting this defect is extremely low.
19. The assertions (crash) all occurred in the code used to store routing information, called Ktree, on the MPC. Due to the order and mix of adds and deletes to the tree, certain combinations of address adds and deletes can corrupt the data structures within the MPC, which in turn can cause this line card crash. The MPC recovers and returns to service quickly, and without operator intervention.
21. This issue only affects MX Series routers with port concentrators based on the Trio chipset, such as the MPC or embedded into the MX80. No other product or platform is vulnerable to this issue.
24. The Ktree code has been updated and enhanced to ensure that combinations and permutations of routing updates will not corrupt the state of the line card. Extensive testing has been performed to validate an exceedingly large combination and permutation of route prefix additions and deletions.
26. All Junos OS software releases built on or after 2011-08-03 have fixed this specific issue. Releases containing the fix specifically include: 10.0S18, 10.4R6, 11.1R4, 11.2R1, and all subsequent releases (i.e. all releases built after 11.2R1).
28. This issue is being tracked as PR 610864. While this PR may not be viewable by customers, it can be used as a reference when discussing the issue with JTAC.
30. KB16765 - "In which releases are vulnerabilities fixed?" describes which release vulnerabilities are fixed as per our End of Engineering and End of Life support policies.
33. No known workaround exists for this issue.