Many people have been asking me what Cumulus Linux is, and how it is different from the software that runs on other networking gear. After all, much of the networking gear sold today has varying degrees of Linux inside, despite many vendor’s attempts to hide this fact. So what makes Cumulus Linux so different? Probably the best way to explain this is by way of a story…

Let’s say you’re an IT manager for a company, in charge of their data center. Some engineers in the company are working on a new application and need to do some experimentation and testing. So they provide you with 3 servers and ask you to install them in the data center. Off in the corner of the data center is an empty rack which you decide to use. The only problem is that you don’t have a spare top of rack switch to attach these servers to each other and the rest of data center network.

As you contemplate ordering a switch, and the time and money that will take, you notice an old server sitting on a bench, unused. It’s got a reasonably fast processor with an empty socket for another, plenty of RAM, and, most importantly, 2 dual-port NICs. You get the idea to create your own top of rack switch, using this server. After loading up your favorite distribution of Linux, installing the latest NIC drivers, and configuring the eth0, eth1, eth2, and eth3 interfaces you’ve got the basis for a switch. The interfaces can be manipulated with ifconfig or the ip commands. The three interfaces connected to the servers are put in a bridge, created by using the standard brctl command. The interface to the rest of the data center network is a routed interface, with routing protocol support provided by the open source package Quagga. You “apt-get install” all of the other apps you are accustomed to on your servers: emacs, autofs, collectd, and whatever else you may need. You’ve created a server which is also a switch.

As you smile to yourself at the ingenuity of your solution, you realize that this is not a good long-term solution. As network traffic grows and as additional servers are added, this solution will run out of steam. What you really need is to add a custom processor to that extra CPU socket which is capable of offloading all of the packet forwarding tasks from the server’s CPU, and which has lots of high speed network ports built-in. Such co-processors do exist. They are switching chips and are made by companies like Broadcom. And when such a chip is added to a server/switch along with the software which syncs the Linux kernel to this switching chip, it creates a system like those enabled by Cumulus Linux. There are many companies which already sell such devices. Companies like Accton, Agema, Penguin Computing and Quanta. These companies are also the manufacturers who build the networking switches for most of the major switching vendors. If you have switching equipment in your data center, you most likely already have equipment manufactured by these companies.

Cumulus Linux is a Linux distribution which has support for networking co-processors. It is not simply a networking device which uses Linux as a base OS. Linux is the OS. All of the device’s interfaces are standard Linux interfaces. So when you type “ifconfig”, you’ll see all of the interfaces, just like on a Linux server. Want to bring a link up? You can use “ip link set”. “brctl” sets up and configures bridges. Standard open-source routing protocol suites, like Quagga, can be used.

This also means that you can do things like you would on any other Linux device, like mount remote file systems, or install and use standard monitoring, administration, and reporting tools. Puppet, collectd, Nagios, bwm-ng, and most other Linux-based tools can be easily downloaded and installed using apt-get. You could even run custom scripts written in Python, or Perl, or C and, for example, put them in a crontab to periodically run. And since the network co-processor is handling all of the datapath forwarding functions, the switch operates at wirespeed.

This is Cumulus Linux.