We all know and love EVPN as a control plane for VXLAN tunnels over a layer 3 infrastructure (Need a refresher? Check out our blog post on the topic). EVPN gives us the ability to deploy VXLAN tunnels without controllers. Plus, it offers a range of other benefits such as reduction of data center traffic through ARP suppression, quick convergence during mobility, one routing protocol for both underlay and overlay and the inherent ability to support multi-tenancy (just to name a few). So EVPN for VXLAN for all of your layer 2 needs, right? Well it’s a little more complicated than that.
Customers need to also communicate between VXLANs and between a VXLAN tunnel and the outside world, so VXLAN routing must also be enabled in the network — which is what I cover in this post. Previous generation merchant silicon does not internally support VXLAN routing, so customers implement a workaround — adding an external loopback cable, sometimes called hyperloop, to the switch or adding an external router for this purpose. The newer chips that support VXLAN routing allow us to route directly on the ASIC, eliminating the need for the hyperloop or the extra router.
VXLAN routing can be performed with one of two architectures – centralized or distributed. Centralized routing performs all the VXLAN routing on one (or two) centralized routers, which can cause additional east-west traffic in the data center. The distributed architecture provides the VXLAN routing closest to the hosts on the directly-connected leaf switches, which simplifies the traffic flow. This is where VXLAN routing with EVPN comes in. BGP EVPN is used to communicate the VXLAN layer 3 routing information to the leafs.
Using the distributed architecture, The IETF defines two models to accomplish inter-subnet routing with EVPN – asymmetric integrated routing and bridging (IRB) and symmetric IRB. Some vendors offer symmetric model and others offer the asymmetric model. Here at Cumulus, we believe you are in charge of your network. Since both models have value — depending on how your network is set up and who might have built your legacy network systems — we offer both solutions so that you can choose whichever method is right for your network.
So, what’s the difference between the asymmetric EVPN IRB model and symmetric EVPN IRB model, anyway?
The main difference between the asymmetric IRB model and symmetric IRB model is how and where the routing lookups are done, which results in differences concerning which VNI the packet travels on through the infrastructure. Because of these differences, there are variations in how they need to be configured on the switch and how they are deployed in your network.
The asymmetric model allows routing and bridging on the VXLAN tunnel ingress, but only bridging on the egress. This results in bi-directional VXLAN traffic traveling on different VNIs in each direction (always the destination VNI) across the routed infrastructure.
Consider the above example. Host A wants to communicate with Host B, which is located on a different VLAN and a different rack, thus reachable via a different VNI. Since Host B is on a different subnet from Host A, Host A sends the frame to its default gateway, which is Leaf01 (this is generally an Anycast Gateway, but we can cover that in a later post). Leaf01 recognizes that the destination MAC address is itself, looks up the routing table and routes the packet to the Green VNI while still on Leaf01. Leaf01 then tunnels the frame in the Green VNI to Leaf02. Leaf02 removes the VXLAN header from the frame, and bridges the frame to Host B. Likewise, the return traffic would behave similarly. Host B sends a frame to Leaf02. Leaf02 recognizes its own destination MAC address and routes the packet to the Orange VNI on Leaf02. The packet is tunneled within the Orange VNI to Leaf01. Leaf01 removes the VXLAN header from the frame and bridges it to Host A.
With the asymmetric model, all the required source and destination VNIs (e.g. orange and green) must be present on each leaf, even if that leaf doesn’t have a host in that VLAN in its rack. This may increase the number of IP/MAC addresses the leaf must hold, which results in somewhat limited scale. However, in many instances, all VNIs in the network are configured on all leaves anyway to allow VM mobility and to simplify configuration of the network as a whole, in which case asymmetric model is desirable.
While it is not hugely scalable, deployment with the asymmetric model is a simple solution as no additional VNIs or VLANs need to be configured. Additionally, less routing hops occur to communicate between VXLANs, which results in lower latency.
Where multi-tenancy is required, each “set of VLANs” can also be placed into separate VRFs and route between the VLANS within a VRF.
The symmetric model routes and bridges on both the ingress and the egress leafs. This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name. However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI. All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure, routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
Now consider the scenario with a symmetric model, as shown above. Let’s say Host A on VLAN A needs to communicate with Host B on VLAN B. Since the destination is a different subnet from Host A, Host A sends the frame to its default gateway, which is Leaf01.
Leaf01 recognizes that the destination MAC address is itself and will use the routing table to route the packet to the L3VNI and nexthop Leaf02. The VXLAN-encapsulated packet will have the egress leaf’s MAC as the destination MAC address and this L3VNI as the VNI. Leaf02 performs VXLAN decapsulation and recognizes that the destination MAC address is itself and routes the packet on to the destination VLAN, to reach the destination host. The return traffic will be routed similarly over the same L3VNI.
With symmetric model, the leaf switches only need to host the VLANs and the corresponding VNIs that are located on its rack, as well as the L3VNI and its associated VLAN, since the ingress leaf switch doesn’t need to know the destination VNI. The ability to host only the local VNIs (plus one extra) helps with scale. However, the configuration is more complex as an extra VXLAN tunnel and VLAN in your network are required. The data plane traffic is also more complex as an extra routing hop occurs and could cause extra latency.
Multitenancy requires one L3VNI per VRF, and all switches participating in that VRF must be configured with the same L3VNI. The L3VNI is used by the egress leaf to identify the VRF in which to route the packet.
So how do I know which IRB model is right for my network?
The hardest part of choosing an IRB model is knowing the difference between symmetric and asymmetric methods. Now that you know the difference, you can make an informed decision regarding the best option for your network.
Generally, if you configure all VLANs/subnets/VNIs on all leafs anyway (e.g. for mobility or ease of configuration), the asymmetric model is for you. It’s simpler to configure and doesn’t require extra VNIs to have to potentially troubleshoot and may have slightly less latency. The asymmetric model will also work well if your data center can be broken down into PODS with VLANs/subnets contained in a POD. Each leaf within the POD is configured with all VLANs/subnets/VNIs in that local POD. Other PODS and external networks will be reachable via EVPN external routes (EVPN external routing with the asymmetric model is supported in Cumulus Linux 3.6 release – using the L3VNI for external routing only).
If your VLANs/subnets/VNIs are widely dispersed and/or provisioned on the fly, choose the symmetric model. The symmetric model supports reachability to external networks with Cumulus Linux 3.5.
Cumulus Networks believes you own and control your network, not a proprietary vendor, so we provide both solutions to allow you to choose what is right for you — instead of limiting your choice to one or the other. If you’d like to learn more about how to configure and deploy EVPN with VXLAN routing in your environment, check out our detailed white paper.