Updated on 8/25/2015: I had no idea when I wrote this blog earlier this month that I’d be proven right by recent events so quickly.  Someone evidently found a similar issue with Cisco’s closed, proprietary ROMMON firmware, and instead of disclosing it responsibly, they weaponized it and secretly used it for nefarious purposes in the wild.  There is no way to know how long this has been used surreptitiously by malicious actors before someone noticed it and published it.

Gregory Pickett of Hellfire Security reached out to me last Wednesday about some interesting research he is presenting tomorrow at Black Hat USA. There are two parts to his research: a security bug in Cumulus Linux (that we already patched) and other network operating systems, and a serious design issue with how all network switches are designed and built.

The security bug was the easy part: it is not exploitable in our default configuration, and Gregory politely gave us a heads up well ahead of time, so we put the fix out last Friday to protect customers who have modified their sudoers configuration in a way that exposed them to the vulnerability. You can see the details in our security fix announcement from last Friday. (If you’re interested in being notified about future security fixes in Cumulus Linux, please sign up for our security mailing list.)

The much more serious issue he will present is the exploitability of firmware in all network switches. This same exploitability has been known about in servers, laptops and PCs for years (and in some cases mitigated with technologies like Trusted Platform Modules), but its application to networking devices is new.

This issue means that once an attacker has gotten full root-level access to a switch (via whatever means), their firmware-based malware cannot be disabled even by reinstalling the operating system. Note that this issue applies to proprietary vendor switches’ firmware as well as those switches that use ONIE, the Open Network Install Environment. The vulnerability is not specific to software-defined or open networking.

I believe Gregory chose to demonstrate the issue using ONIE because it is developed openly, and thus far easier to do research on and experiment with. Proprietary vendors often claim that attacking their systems is harder because it requires reverse engineering. This may have been true in the past, but with the stakes for security as high as they are now, we repeatedly see that people are willing to do the work to reverse engineer. And after doing all that work, they don’t always publish their results openly. When someone else executed a related attack against the firmware in Cisco gear in the past, they didn’t publish their findings, they sold them to these guys.

As a result, I believe that open source and open development of standards like ONIE increase overall security! We (as an industry) should all be thankful to Gregory for presenting his research in the open, so that we can fix problems.

The ONIE community has come together, from OS vendors to hardware manufacturers, to address this issue in an open, transparent way. Hopefully the proprietary vendors will do the same for their customers, though I doubt they will do so out in the open.

Unfortunately I’m in Mountain View working, not having fun and learning at Black Hat, but if you’re more fortunate than I am, you should definitely attend Gregory’s talk tomorrow at 11am.[/fusion_text]