In my previous post, I focused on the concepts of what is called off box routing and centralized routing. They were two different yet similar solutions. The first one being the simplest solution leveraging an external gateway to route between VXLANs. The second solution integrated the edge device to be both an external gateway and VXLAN end point (VTEP).

To expand on my previous post, the next logical place to put a gateway in VXLAN designs is to distribute them all on the top of rack (TOR), also known as the leaf. This TOR acts as a VTEP in the VXLAN solution. Its primary purpose is to encapsulate and decapsulate traffic. This solution is also colloquially known as Anycast Gateway VXLAN Routing. Anycast Gateway VXLAN Routing can only be performed on ASICs that support routing in and out of tunnels (RIOT), as discussed in the previous post. For the rest of this post, when I refer to VXLAN Routing, I specifically mean Anycast Gateway VXLAN Routing unless otherwise noted.

In the simplest form, VXLAN Routing allows the TOR to perform a route lookup on the inbound packet before encapsulating the traffic into a VXLAN tunnel. There are two ways that the lookup can occur: (1) asymmetric and (2) symmetric. This post won’t go into the details of symmetric mode compared to asymmetric mode, as I will detail that at a later time. Our latest release, Cumulus Linux 3.5.0, supports both asymmetric and symmetric modes. For comparison sake, Cisco and Arista implement their VXLAN routing solution using symmetric mode. Juniper implements its VXLAN routing solution using asymmetric mode

Instead of focusing on the difference of how inter-VXLAN routing works, I wanted to focus on where the gateways live and what architectural elements to consider in designing a VXLAN routing solution.

Let’s take our previous topology and replace it with an asymmetric VXLAN Routing design.

In the previous post, the gateway was set to, which lived on the edge firewall. This doesn’t change for all traffic going out to the Internet. But we also add an additional anycast gateway on every TOR. This anycast gateway is used by the TOR to directly route traffic to the VTEP that owns the destination rather than having to trombone traffic through the exit leaf.

This case is a bit advanced, as this solution requires two unique routes on the server to handle both VXLAN Routing and Internet access. In the case where the server is only set up with a single gateway, it introduces a second issue of how EVPN advertisements are handled.

The most common type of EVPN advertisements is Type 2 messages. These messages are MAC address advertisements. Each VTEP advertises any MAC address learned in its MAC and IP Neighbor table into BGP as a Type 2 EVPN advertisement. Then BGP transfers all these messages across all routing members so that every VTEP knows which MAC address is owned by which other VTEP.

Finally, we’ve recently added new functionality to EVPN routing. With routes added into EVPN, each TOR can perform a route lookup into the VXLAN overlay and forward traffic based on L3 routing information rather than just L2 switch information. This specific feature is covered in detail in this white paper. I recommend you check it out! 

As an aside, the one benefit of symmetric mode compared to asymmetric mode is that asymmetric mode requires every VNI to exist on every VTEP. Symmetric mode does not have that same limitation.

VXLAN routing has full VRF support, so that allows you to create the full end-to-end tenant isolation that you would normally have with a VLAN only solution. In our perfect world, VXLAN Routing with Type 5 EVPN advertisements and symmetric VXLAN Routing provides the full comprehensive end-to-end scalable data center solution that enables tenancy, L2 adjacency and scalability.

If you’re interested in getting hands-on experience with open networking, then sign up for one of our boot camps! Whether you prefer in-person or online training, our networking experts will teach you everything you need to know.