It’s always a treat to be working for an organization that has supportive aficionados (and production customers too!).
Recently I ran across The Class-C Block show on all things Cumulus. The episode clearly captures die-hard Linux converts and skeptical Cisco purists. Both of these extreme viewpoints are completely valid and shouldn’t be quickly dismissed. It was a great pleasure to see the real-time dialogue between these two groups, even if the participants themselves don’t firmly have a foot in either camp.
As someone who has been plumbing IP networks for over 15 years, it’s been an interesting personal career change from “building” to “evangelizing”. At the heart of Cumulus Networks is an engineering-led organization, which translates to very clear communication around our product capabilities and addressing assumptions – negative or positive. I know that I’ve certainly found myself in a place where I wish that the vendor or supplier were just a bit more transparent. With that in mind, here are some points I’d like to further elaborate on:
0:07:05 – Cumulus Linux is a fork of Debian (just as Ubuntu), and the current 1.5.x & 2.0.x releases are derived from the “wheezy” branch. While “piecing together a Debian server with a bunch of NIC cards” is theoretically the same as Cumulus Linux, this is not at all a technical equivalent. Under hardware-based switches running Cumulus Linux, the majority of network functions are hardware accelerated – such as programming the FIB, ACLs or firewall rules, etc. This arrangement offers line rate performance – very different than a “PC router”. In addition, several of the software packages have been modified by Cumulus.
0:11:08 – “switchd is the only thing that is special”. Linux distributions are much more than just a set of packages. Cumulus Linux is comprised of a number of changes besides switchd – this includes bug fixes, performance enhancements, contributed software, documentation, and of course, support. For example, the Cumulus build of Quagga has over a hundred patches based on our internal testing and customer feedback. Where possible, Cumulus submits patches upstream, however not every project adopts these and releases the contributed changes. All of these patch sets are available on our OSS site.
0:14:56– cl-brctl is a wrapper for brctl, using the alternatives framework. Most of the other cl- commands are specific to Cumulus Linux, for example image or license management. Over time some cl- commands will be deprecated; my personal goal is to have as few of these Cumulus-specific commands as possible. Generally these exist for lack of an existing standard command set. The overall goal is to have command parity matching a stock Linux x86 distribution.
0:14:20 – ONIE (correctly pronounced by Matt & Willard) has been critical to disaggregating software from once proprietary hardware platforms. It’s very humbling where lead developer Curt Brune has taken this effort in such a short time. In less than a year, Curt has taken my dry erase scribblings and rantings, and shaped it into a widely embraced project. I like to call it “PXE that doesn’t suck”. In addition to Cumulus & Big Switch – many other partners have also pledged to ship ONIE compatible hardware: Dell, Quanta, & Accton. Monitoring the mailing list gives a good preview of additional organizations proposing to adopt ONIE, too.
0:16:30 – While Cumulus doesn’t offer a proprietary modal CLI as historical NOSs have done, we have chosen to fully expose Linux. Again, I want to be very clear here – no limited shell, full root access, and the freedom of choice. This offers users 3 different configuration approaches: real-time, persistent, and/or automated. The Linux server industry has produced dozens of tools to tackle configuration abstraction (Puppet, Chef, Ansible, CFEngine) and we prefer to leverage these rich communities, instead of invent another domain-specific language.
Cumulus Linux does offer some wrapper tools to ease configuration, with one example being our non-modal interface to Quagga (see cl-ospf & cl-bgp, both of which have awesome bash tab completion). Our intention is blunt here: we consider Linux the baseline API. We’re not here to push a prescriptive “thou shall use X approach or else suffer!” message. Some users write C programs directly against the netlink subsystem, others author simple bash scripts, some take advantage of their existing configuration management system – it’s all completely up to them.
0:17:31 – As noted earlier, a Debian host can provide some configuration parity – but not performance or the same daemon codebase. Cumulus does plan to offer a development partner oriented VM in the future. The key here is this will not be representative of network silicon performance or basic system telemetry (think of emulated calls to dmidecode or lmsensors).
The most accurate environment is an actual hardware switch running Cumulus Linux, to that end Cumulus makes available several “customer workbenches”. These are small pods of switches, console servers, PDUs, management, and front-panel attached servers. This allows a prospective customer to “try before buying”, and is obviously a great way to gauge familiarity on the product.
0:20:24 – We’re always looking to improve the user experience around non-destructive configuration changes. To help close the gap between interactive and persistent syntax, along with a huge list of enhancements & bug fixes, Cumulus Linux 2.1 will ship with ifupdown2. This is a completely rewritten replacement for the current network interface configuration system.
This large topic deserves a dedicated post on all of the new features, which I will do before release time. However, I did want to point out for the particular syntax example given – we’ve been there and done that. Ifupdown2’s “ifquery” program offers a “dump the active running configuration as a compatible interfaces file to stdout” directive. This allows for numerous commands to be run and tested on the command line, with a “snapshot” to write out a persistent configuration. Also, this rewrite supports the secondary IP address example too, extending the syntax lexicon of the interfaces file.
0:23:22 – Under Cumulus Linux 2.0 the VXLAN implementation effectively behaves just as the standard Linux VXLAN interface. Commands such as “ip link add vxlan0 type vxlan …” just work and are appropriately mapped to the network silicon. We have two partners utilizing this technology today: VMware NSX and Midokura MidoNet.
The NSX implementation uses OVS for northbound configuration; VMware (formerly Nicira) has published the VTEP schema that we utilize. As for more adoption of OVS, Cumulus has demonstrated novel uses here – most recently detecting heavy top talkers or “elephant performance handling”.
0:27:44 – “Everything is all over the place… it’s a very fractured interface”. I think both viewpoints were reflected well here. Unlike traditional NOS vendors, Cumulus doesn’t preach that customers need to learn our proprietary or arcane scripting language. It frustrates me when tweets fly by proclaiming that “network engineers need to learn Python” to stay relevant in the industry. That’s leading with such a false message. Not everyone is an auto mechanic, but many of us have replaced a windshield wiper or changed a tire.
I bet a lot more “traditional network engineers” have played with a Raspberry Pi or Arduino, than have taken up learning TCL or XSLT in their spare time. For users that already have experience with Bash, Ruby, Perl, and/or Python – they’re ready to go. Just our support of baseline scripting languages sets Cumulus Linux vastly apart from other vendors, which are “Linux under the hood”.
Following the true Unix spirit, part of the decoupling is historical on how systems are built to scale and operate reliably. For example, if you don’t need STP – why run a daemon that provides this? It’s amazing how powerful it is to take a customer from the “cut-n-paste notepad” mentality to introducing simple one-liners with seq, awk, and sed. Linux is powerful: take advantage of it.
0:30:29 – Willard hit the nail on the head, as Cumulus attempts to push as much code upstream as possible. As clearly demonstrated by the traditional vendors, forking from Linux makes it extremely difficult to evolve with a changing environment. Moving the wider community forward benefits our customers and has a direct impact on code sustainability.
0:34:06 – Restarting switchd is NOT the correct way to reload any network configuration; switchd should never have to be restarted. In the case of interface configuration, this is handled by ifup/ifdown’s /etc/network/interfaces file – bridges (aka VLAN’s), bond’s (link aggregates), and routed L3 interfaces are all defined here. Ifupdown2 improves upon this greatly. For example, a bridge which has a particular member or interface being removed or added will only affect that changing membership – leaving the rest of the bridge intact. Additionally, ifupdown2 provides template support along with glob & regex matching – ideally suited for a large instantiation.
0:36:44 – “Does 10work?” Cumulus Linux ships with a modified version of Quagga. As much as possible, we work with the various Quagga communities to contribute our efforts upstream. The suite of daemons is well tested for BGP & OSPF functionality in both IPv4 & IPv6 deployments. One of our customers has well over 5,000 switches configured in an OSPFv2 (IPv4) environment running our build of Quagga. BIRD is also available in our testing apt-get repo.
0:40:59 – Many organizations already have extensive Linux knowledge; where automation, scripting, monitoring, and other lifecycle management are well understood. This is not commonplace on the network side, which is traditionally treated as a silo or decoupled from compute. Part of the advantage of Cumulus Linux is tapping into that vast resource, which isn’t possible under traditional network platforms.
In my previous network deployment roles, I never once hired a traditional network admin – instead I courted generalists who could be taught networking skills. We approached the network just as any other infrastructure – plan, prototype, monitor, deploy, automate, and document. Then we’d move onto the next project, be this storage, virtualization, or anything else. Many IT organizations have already moved towards a converged staffing model, removing silos and spreading knowledge across supporting teams.
0:42:30 – gcc, make, and ancillary build packages are available in the add-ons apt-get repo. Other packages can be sourced from the Debian.org mirrors, too. If you have a particular package or utility you’d like to see available under Cumulus Linux, please don’t hesitate to reach out. So far some trolls have asked for R and Mathematica.
The support strategy is built around a continuum of expectations. Packages or software that ships pre-installed is fully supported, which means we’ll wake up an engineer at 3am to solve a problem. Partner-supplied or authored software is co-supported. For example, we wrote a minor portion of Puppet (our netdev module), but obviously didn’t write the complete Puppet software package. Software that is installed outside of these parameters is best effort. Obviously, though, we wouldn’t hang up the phone on a customer and would do whatever it takes to restore production availability.
0:45:22 – Bob brings a very good point around vendors pointing fingers in the case of support issues. Cumulus publishes an established table of supported hardware, creatively named the Cumulus Linux Hardware Compatibility List (or HCL). Part of this HCL process is not just porting Cumulus Linux correctly to the platform, but also harmonizing the RMA and support escalation procedures. Cumulus Networks is the “one throat to choke”, and users reach out to us directly – regardless if the problem is software- or hardware-related. Our support team will verify if the issue is hardware-related, and if so, will shepherd that process with the equipment manufacturer.
0:50:18 – ONIE is purely a secondary boot loader (equivalent to PXE). On PowerPC platforms U-Boot is still used as the first boot loader. Similarly on X86, a traditional BIOS does the initial system loading. In both cases, ONIE is triggered after these first stage loaders complete.
Part of the Cumulus Networks HCL process is bringing up platforms, which may look similar externally but are radically different underneath. Each board has a different layout for LEDs, the I2C bus, how the PHYs are wired, storage device, etc. Drivers have to be modified or built around the unique parts of the platform. In most cases, these platforms share a common network silicon chip (Broadcom being quite popular). But just because the chip is common, that doesn’t mean that an unknown board could be “jailbroken”. Sadly, it’s not quite that simple, otherwise we’d all be buying surplus hardware to rechristen under Cumulus Linux.
0:53:20 – Cumulus has a number of OpenStack partners today that are already supporting VMs under a L3 routed environment – some using IGP (OSPF or BGP), others with VXLAN, or even a combination of both. Chris described a classic Clos architecture, where routes are summarized at the leaf layer. This mitigates L2 frustrations of VLAN exhaustion & STP, with the blessing of ECMP flow scalability.
Thanks again to Bob McCouch, Willard Dennis, Matt Stone, Matt Oswalt, and Chris Marget for sharing your thoughts around all things Cumulus Networks. Hopefully my behind the scenes thoughts were helpful and interesting.