This post was updated on 2/22/17 to reflect the official launch of EVPN. You can now access EVPN in general availability. Read the white paper to learn more about this exciting new feature.
When we set out to build new features for Cumulus Linux, we ask ourselves two questions: 1) How can we make network operators’ jobs easier? And 2) How can we help businesses use web-scale IT principles to build powerful, efficient and highly-scalable data centers? With EVPN, we believe we nailed both.
Many data centers today rely on layer 2 connectivity for specific applications and IP address mobility. However, an entire layer 2 data center can bring challenges such as large failure domains, spanning tree complexities, difficulty troubleshooting, and scale challenges as only 4094 VLANS are supported.
Therefore, modern data centers are moving to a layer 3 fabric, which means running a routing protocol, such as BGP or OSPF between the leaf and spine switches. In order to provide layer 2 connectivity, between hosts and VMs on different racks as well as maintain multi-tenant separation, layer 2 overlay solution is deployed such as VXLAN. However, VXLAN does not define a control plane to learn and exchange MAC addresses. Therefore, VXLAN cannot, by itself, deal efficiently with host mobility and fast convergence. Practical deployments of VXLAN without a control plane often use a controller.
Traditional bridging and L2VPN solutions like VPLS and native VXLAN retrieve host location via the data plane, meaning it learns the location of the hosts based on the data frames that pass through the switch. When the bridge receives a frame on a switchport, it will take note of its source MAC address and add it, along with the received port, to the switch’s MAC table. This MAC address table directs traffic to the correct destination. See below.
The same data plane learning happens over the MPLS (or other) tunnel also as there is no control plane protocol to exchange the MAC addresses learned on the local switch ports with the remote PEs or VXLAN Virtual Tunnel End Points (VTEPs). Learning solely via the data plane has its own set of limitations, such as limited redundancy, no per-flow load balancing, lack of traffic engineering, and slow convergence.
How does EVPN help me?
Cumulus Networks EVPN is a next-generation control-plane solution for VXLAN tunnels that uses the BGP routing protocol to provide high scale, redundancy, traffic engineering, multi-tenant separation, and fast convergence for host and VM mobility — all while interoperating between vendors. BGP is a well known, mature routing protocol that powers the Internet. It is very robust and scalable. The Multiprotocol BGP (MP-BGP) extensions allow BGP to carry all kinds of information by introducing support for address-families and sub-address-families. All of the above characteristics make BGP very attractive to use for a VXLAN control plane.
In early 2017, Cumulus Quagga will natively contain the new BGP address-family that was created by the IETF for EVPN — the EVPN address-family, as seen below.
The MP-BGP EVPN address-family advertises MAC addresses learned on a switch to the remote VTEPs while providing for multi-tenant separation and overlapping addresses. Advertising MAC addresses via BGP allow remote switches to dynamically learn the location of hosts or VMs without requiring data packets to traverse the VXLAN tunnel first. After the MAC address is learned on one switch, then all peered switches will learn it as well, which can reduce broadcast traffic in the network and allow fast convergence when a host or VM moves to a different rack.
For example, in a setup with three leafs and three hosts, Host 1’s MAC address would be learned by leaf01, host 2’s address by leaf02, and host 3’s address by leaf03. This often happens as soon as the link between the leaf and the host comes up, through Gratuitous ARP (GARP). At this point, each leaf would advertise its locally learned MAC address to the other leafs via BGP. Now, all of the leafs have all of the hosts MAC addresses and their locations without requiring a data packet to traverse the switch first.
Using the above setup as an example, the command sh evpn vni 10100 mac outputs all the MAC addresses and their locations associated within that VNI, or VXLAN tunnel. For example, EVPN Table for leaf01 shows:
Whereas the table for leaf02 similarly shows:
If host 1 then moves to rack 3, leaf03 will learn of the new location and immediately update the other two leafs of the move. It is not necessary to wait for MAC table timeouts or a new data packet sent by the host for data center wide convergence.
Can you summarize the benefits of deploying EVPN?
Cumulus EVPN provides many benefits to a data center, including:
- Controller-less VXLAN: No controller is needed with EVPN, as it enables VTEP peer discovery through BGP.
- Scale and Robustness: EVPN uses the standard BGP routing protocol for the control plane. BGP is a mature well-known protocol that powers the internet. For data centers that already run BGP, this involves just adding another address-family.
- Fast convergence/mobility: The BGP EVPN address family includes features to track host moves across the datacenter, allowing for very fast convergence.
- Multi-vendor interoperable: Since EVPN is a standard, it will be interoperable with other vendors that adhere to the standard.
- Support for Active/Active VxLAN: Cumulus EVPN supports host redundancy to switch pairs with an MLAG configuration.
- Multi-tenancy: Cumulus EVPN supports VXLAN tunnel separation