Recently there was a conversation in the Cumulus community (details in the debriefing below) about the best way to build a redundant backup IP link for multi-chassis link aggregation (MLAG). Like all good consulting-led blogs, we have a healthy dose of pragmatism that goes with our recommendations and this technology is no different. But if you’re looking for the short answer, let’s just say: it depends.

The MLAG backup IP feature goes by many names in the industry. In Cisco-land you might call this the “peer keepalive link,” in Arista-ville you might call this the “peer-address heartbeat” and in Dell VLTs it is known as the “backup destination.” No matter what you call it, the functionality offered is nearly the same.

What does it do?

Before we get into the meat of the recommendation, let’s talk about what the backup IP is designed to do. The backup IP link provides an additional value for MLAG to monitor, so a switch knows if its peer is reachable. Most implementations use this backup IP link solely as a heartbeat, meaning that it is not used to synchronize MAC addresses between the two MLAG peers. This is also the case with Cumulus Linux. In the Cumulus Linux MLAG implementation, UDP port 5342 is used by default for the MLAG backup IP configuration, which you can see by running the net show clag params command below.

Note that the backup IP is not a required configuration[a][b][c][d] today (that might change in the future); it is used simply to provide added redundancy for the classic “split-brain” MLAG failure scenario. In this case — the loss of a peerlink — each MLAG peer is unaware that the other is still alive and will continue to forward traffic. However, the forwarding of return traffic is suboptimal as the MAC address tables are no longer being synchronized and the return traffic may need to be flooded to all participants in a destination VLAN.

How can I build it?

With Cumulus Linux, there are four main approaches to deploying an MLAG backup IP.

1. Use the loopback. Another option is to leverage the loopback IP addresses as targets for the backup IP link. Dynamic routing is a must for this option and requires the stable advertisement of loopback IP addresses from one MLAG peer to another. We now document this option as it can be appropriate for many environments.

2. Use a dedicated dataplane link. One of the least common yet still valid methods to configure a backup IP link is to dedicate an interface to it. In this case you could dedicate one or more low speed interfaces, possibly in an LACP bond, which is built in a dedicated VRF to connect directly to the peer switch. This is very similar to adding more links to your peerlink bond; however, these links are not used for synchronizing MAC addresses. We don’t document this method currently, but it is supported.

3. Use the management network. The most common method of implementing the backup IP is through the use of the management plane. The existing isolation of the management VRF provides an ideal shared segment between both MLAG peers. This segment is generally via a totally different path than the loopback option described above and provides redundancy in a way that does not depend on the health of layer 3 protocols and dynamic routing.

4. Use back-to-back management ports. This approach like #3 above is not very common. There are very particular use cases where it makes sense which we will discuss below. The main idea is that you connected the eth0 port from one MLAG peer directly to the eth0 port of another MLAG peer. We don’t document this method currently either as it’s not something we see all that often, although it too is supported.

How SHOULD I build it?

You can leverage multiple options but what is the right choice for your environment? If you can use option #1 or #2 above either of those would be preferred over options #3 or #4 but all of them will serve the intended purpose. Here we’ll discuss the pros and cons of each approach while identifying some of the logical use cases for each.

When to use a loopback interface for the MLAG backup IP
This is a great approach to use when deploying MLAG. If you’re deploying MLAG in an all Layer 2 environment it may not be possible to leverage this technique. If you’ve never worked in detail with routing policy this approach has more complexity but on the flipside it is the most robust and does not sacrifice any links to be dedicated to this purpose.

Logical Use Cases
  • Each of your MLAG devices has routing relationships with a series of common devices (leafs to spines, spines to superspines, and so forth)
  • L3 uplinks across a routed fabric that connects MLAG peers has rich multipathing
Don’t Use It When
  • You have back-to-back/layered/nested/2-tier MLAG environments
  • You’re concerned about the stability of the routing domain between MLAG peers
  • You don’t want to deal with ASN concerns in BGP and the possibility of using “allowas-in 1” (see the note in our documentation)

When to use a dedicated dataplane link for the MLAG backup IP
This approach, although not seen very often also offers a significant amount of robustness, and unlike the option above it can be used in any kind of environment. If you have a network which can use this method, and not all can, you should. This option is another great choice but due to port density concerns is not always considered.

Logical Use Cases
  • Can be used in ANY network type
  • There is no management network (like there are with MLAG in OOB environments)
  • Switches are not fully populated and extra ports remain available for use
Don’t Use It When
  • Port density is of primary importance or switches are at full port capacity

When to use the management network for the MLAG backup IP
This has been our tried and true approach to deploying MLAG for a long time now. It’s still used very often because many environments typically have a management network and this is dead simple to setup with very low configuration complexity and without the need to sacrifice ports specifically for the task. This approach works great for dataplane networks but OOB networks may not have their own OOB network to leverage so it is not as versatile of a solution.

Logical Use Cases
  • Typically used with switches in the dataplane of the network (not out-of-band switches)
  • You’re already using management VRF
  • Layer 2 (L2) uplinks prevent you from using a loopback interface
Don’t Use It When
  • There are concerns regarding the stability of the management network

When to use back-to-back management ports for the MLAG backup IP
This method is used infrequently but it works just fine. It is a great way to setup MLAG for an OOB network that does not have it’s own OOB network and needs every single port for switch-to-switch connectivity, leaving you with unutilized Eth0 ports which could be dedicated to the backup IP link.

Logical Use Cases
  • There is no management network (like there are with MLAG in OOB environments)
  • There are no available dataplane (front-panel) ports
  • There are L2 uplinks or you have a back-to-back/layered/nested/2-tier MLAG environment
Don’t Use It When
  • The distribution tier of OOB switches is in different racks and cross-rack cabling is not allowed.

Debriefing a recent issue

Some of the recent discussion around the MLAG backup IP arose from an incident where a hardware issue was causing the dataplane of a switch to become stuck. When that happened, the peerlink went down on the secondary MLAG switch but the backup IP was still accessible to the management plane (eth0) of the primary MLAG peer. This peculiar failure caused the secondary to think the primary was still alive and not promote itself, which led to all host-facing links being down.

This particular issue would have been caught if the dataplane was integrated into the forwarding of the backup IP link via the use of the loopback IP as the backup IP. Ironically, the issue would not have manifested itself if NO backup IP was configured!

While using a loopback IP as the backup IP would have prevented this specific issue, this was a pretty anomalous case and not typical of other commonly seen failures.

Which method is the best though?

Option 1 or 2 from the above list are the most robust and are the recommended way to deploy MLAG today. However, nothing is free in life, and you must be willing to accept the added complexity of making the reachability of the backup IP part of the routing domain, or in the case of option 2, you must be willing to sacrifice a dedicated link(s) to the purpose.

If the tradeoffs mentioned above are not desirable or not possible based on the constraints of your environment consider using the management network or a back-to-back connection over the management port. Without a doubt, the most general use case for the backup IP continues to be the use of the management network, as it is the least complex and the most widely applicable method, since you can use it in either all-L2 or mixed L2/L3 environments.