We’ve all been there. That “non-disruptive” maintenance window that should “only be a blip”. You sit down at the terminal at 10pm expecting that adding a new server to the MLAG domain or upgrading a single switch will be a simple process, only to lose the rack of dual-attached servers and spend the rest of your Thursday night frantically trying to bring the cluster back online.

If I never spend another evening troubleshooting an outage caused by MLAG, I’ll die happy!

While MLAG provides higher availability than single attaching or a creating multi-port bond to a single switch, it comes with the cost of a delicate balancing act. What if there was a way to provide redundancy without MLAG’s fragility and its risk to maintenance windows?

We at Cumulus Networks have seen many of our customers solve these problems by leveraging Cumulus Quagga, our enhanced version of the routing suite, on their server hosts, so we’ve decided to call it Routing on the Host and make it broadly available for download.

By leveraging the routing protocols OSPF or BGP all the way to the server, we can resolve that MLAG problem once and for all.


Over the last five years, data centers have evolved to deploy more routing and less bridging. In the early days of data center architecture, we routed down to a pair of aggregation switches, with a large layer 2 domain below it. Then the network evolved to include routing to the top of rack switches, where a local MLAG pair acted as the means to connect to the rack of servers below. Because routing limits loops and broadcasts while allowing for equal cost multipath (ECMP) for all-active link utilization, Routing on the Host lets you finally realize these advantages all the way to the server and start looking at more resilient networks running at higher speeds.



With Routing on the Host, you can use ECMP for high I/O workloads, like Ceph storage or Apache Spark; you can connect servers to more than two top of rack switches and share them all equally.



For clustered solutions, like Ceph storage, the loss of a rack of servers due to an MLAG issue can have a major impact on the network as the cluster must replicate and rebalance all of the data that was just temporarily lost. The added network stability from Routing on the Host can prevent these small outages that have a massive impact.

When it’s 10pm on Thursday and it’s time to execute maintenance, Routing on the Host allows for zero packet loss due to network changes and upgrades. You can change the OSPF link metric or BGP AS Path on the top of rack switch that is about to undergo maintenance, gracefully removing it from the data path. This signals the server to send traffic to other attached top of rack switches. Once all traffic has routed around the switch undergoing maintenance, you can execute the software upgrades or configuration changes. You can even try this yourself with a demo I made in Cumulus VX.


Finally, Routing on the Host provides a superior network design for bare metal and containers. By relying on OSPF or BGP unnumbered, a server can easily connect anywhere in the data center and advertise IP addresses for itself or containers. With containers, you can use Routing on the Host to advertise the overlay endpoint and allow Docker networking or a networking plugin to dynamically build overlays.

Running Cumulus Quagga directly to a server provides real benefits in a simple manner. Never again do you have to worry about MLAG maintenance taking down your environment or a server administrator confusing server bridge and bond configurations, causing a catastrophic bridging loop. Service providers and web scale companies have relied on entirely routed environments to provide the highest scalability and reliability in their networks.

Routing on the Host is a way to get this same power and flexibility throughout the entire data center.

Download Cumulus Quagga and start your journey of independence from late night troubleshooting and L2 data centers!