I started coveting IP encapsulated network virtualization back in 2005 when I was working to build a huge IP fabric. However, we needed to have layer 2 (L2) adjacencies to some servers for classic DSR load balancing. The ideal solution was to have something that looked like a bridge as far as the load balancers and servers were concerned, yet would tunnel unmodified L2 frames through the IP fabric. Alas, we were way ahead of our time.

Thank the IT gods that things have changed quite a bit in the last 12 years. Today, we as an IT community have VXLAN, which is embodied in most modern networking silicon and (a bit more importantly) realized as part of the Linux networking model so that it’s really straightforward to deploy and scale. IT geeks have a bunch of ways to build L2 domains that are extended across IP fabrics using VXLAN. There are dedicated SDN controllers, such as Contrail, Nuage, Midonet and VMware NSX; there are orchestration-hosted controllers in OpenStack Neutron and Docker Swarm; and there are simple tools like the lightweight network virtualization that we built at Cumulus Networks.

This all leads me to EVPN. We recently made EVPN available as part of Cumulus Linux, and I had the opportunity to deploy it as part of an OPNFV project. I wanted to take this opportunity to express my excitement for the release and explain a little bit about how we used EVPN in the OPNFV project.

The challenge, accepted

The team wanted to deploy an L3 fabric with a few L2 domains for operational purposes. They wanted to use VXLAN/EVPN to build the L2 domains, and for added flair, some of those L2 domains carried VXLAN encapsulated traffic to build tenant networks and chain services. We used an ODL controller variant that leveraged EVPN to build the tenant-facing connections, and with no lack of ambition, we wanted all of the EVPN instances to be aware of each other.

I have to admit, I was slightly intimidated… I was familiar with all of the underlying technology, but so many times, the “operationalization” of it gets hairy. What blew me away was how easy it was to get the system together!!!

Touchdown, EVPN!

We had three racks which each consisted of six servers, and a switch with two spine switches aggregating it all. Each server had an interface that connected both the BMC and the OS to the management networks (192.0.4.x VLAN 192) and an interface trunked to a list of VLANs/networks used for operational purposes. Given that we had an L3 fabric, I deployed management VRF on all of the switches. One of the leaf switches had an interface connected the out-of-band network to the outside world, and another leaf switch had an interface that trunked to a router for access to the Internet (on VLANs 157 and 158). Here’s a simplified view of the network:

EVPN

I used Ansible for the configuration with two roles, one for the spines and one for the leafs. While I love using Ansible, I’ll cop to not writing the best playbooks — but hey… they work!! Here is the set of Ansible variables and playbooks that I used for this deployment.

Variables

[ops@vco-jump group_vars]$ cat group_vars/all
#  networks/VLANS
# 
OOB_VLAN:
    - 192

INET_VLANS:
    - 157
    - 158

OSP_VLANS:
    - 120
    - 130
    - 140
    - 150
    - 1001
    - 1002
    - 1003
    - 1004

# spine interface info
#
AGG_PORTS: 'swp1,swp2,swp3'

# common leaf interface info
#
# note: there is host specific stuff in the playbooks
#
UPLINKS:      'swp49,swp50'
OOB_PORTS:    'swp1,swp2,swp3,swp4,swp5,swp6'
TRUNK_PORTS:  'swp11,swp12,swp13,swp14,swp15,swp16'
GE_PORTS:     'swp1,swp2'
XE_PORTS:     'swp11,swp12'

# IP addresses and ASNs
#

switchvars:
    spine1:
         ASN: 65000
         LOOPBACK_4: 10.0.0.21
         LOOPBACK_6: fd00:0:0:1::21
    spine2:
         ASN: 65000
         LOOPBACK_4: 10.0.0.22
         LOOPBACK_6: fd00:0:0:1::22
    leaf1:
         ASN: 65001
         LOOPBACK_4: 10.0.0.11
         LOOPBACK_6: fd00:0:0:1::11
         INET_ADDR: 172.21.158.145/24
         ODL_PEER: 172.21.158.145
    leaf2:
         ASN: 65002
         LOOPBACK_4: 10.0.0.12
         LOOPBACK_6: fd00:0:0:1::12
    leaf3:
         ASN: 65003
         LOOPBACK_4: 10.0.0.13
         LOOPBACK_6: fd00:0:0:1::13

L3 Leaf tasks

[ops@vco-jump tasks]$ cat roles/l3-leaf/main.yaml 
#-----------------------------------------------------------------------------------------
#
# L3 fabric and EVPN VTEPs for L2 connectivity
#
---

- set_fact:
     myvars: "{{ switchvars[inventory_hostname] }}"

# L3 fabric 
#
- name: Clean the slate and set up unnumbered BGP fabric and VNI advertisement
   nclu:
     commands:
     - add bgp autonomous-system {{ myvars.ASN }}
     - add bgp router-id {{ myvars.LOOPBACK_4 }}
     - add loopback lo ip address {{ myvars.LOOPBACK_4 }}/32
     - add bgp ipv4 unicast network {{ myvars.LOOPBACK_4 }}/32
     - add loopback lo ipv6 address {{ myvars.LOOPBACK_6 }}/128
     - add bgp ipv6 unicast network {{ myvars.LOOPBACK_6 }}/128
     - add interface {{ UPLINKS }} mtu 9216
     - add bgp neighbor {{ UPLINKS }} interface remote-as external 
     - add bgp ipv4 unicast neighbor {{ UPLINKS }} activate 
     - add bgp ipv6 unicast neighbor {{ UPLINKS }} activate 
     - add bgp evpn neighbor {{ UPLINKS }} activate
     - add bgp evpn advertise-all-vni
     atomic: true
     description: l3-fabric

# VLANS/VNIS
#
# This is kind of tricky... no commit in the first loops, only after the OOB VLAN
#
- name: VLANs and VTEPs
  nclu:
    commands:
     - add vxlan vtep{{ item }} vxlan id {{ item }}
     - add vxlan vtep{{ item }} vxlan local-tunnelip {{ myvars.LOOPBACK_4 }}
     - add vxlan vtep{{ item }} bridge access {{ item }}
     - add vxlan vtep{{ item }} bridge learning off
     - add vxlan vtep{{ item }} mtu 9216
  with_items: "{{ OSP_VLANS }}"
- nclu:
   commands:
    - add vxlan vtep{{ item }} vxlan id {{ item }}
    - add vxlan vtep{{ item }} vxlan local-tunnelip {{ myvars.LOOPBACK_4 }}
    - add vxlan vtep{{ item }} bridge access {{ item }}
    - add vxlan vtep{{ item }} bridge learning off
    - add vxlan vtep{{ item }} mtu 9216
  with_items: "{{ INET_VLANS }}"
 - nclu:
    commands:
    - add vxlan vtep{{ item }} vxlan id {{ item }}
    - add vxlan vtep{{ item }} vxlan local-tunnelip {{ myvars.LOOPBACK_4 }}
    - add vxlan vtep{{ item }} bridge access {{ item }}
    - add vxlan vtep{{ item }} bridge learning off
    - add vxlan vtep{{ item }} mtu 9216
    commit: yes
    description: VLANS and VTEPS
  with_items: "{{ OOB_VLAN }}"

- name: Access ports
  nclu:
   commands:
    - add interface {{ GE_PORTS }} link speed 1000
    - add interface {{ XE_PORTS }} link speed 10000
    - add interface {{ OOB_PORTS }} bridge access {{ OOB_VLAN | join(",") }}
    - add interface {{ TRUNK_PORTS }} bridge trunk vlans {{ OSP_VLANS | join(",") }}
    - add interface {{ TRUNK_PORTS }} bridge trunk vlans {{ INET_VLANS | join(",") }}
    atomic: true
    description: access port link speeds
  
# leaf specific config
#
- name: OOB on leaf1 port 48
  nclu:
   commands:
    - add interface swp48 link speed 10000
    - add interface swp48 bridge access {{ OOB_VLAN | join(",") }}
    atomic: yes
    description: connect to OOB network
  when: inventory_hostname == "leaf1"

- name: INET trunk on leaf3 port 47
  nclu:
    commands:
   - add interface swp47 link speed 1000
 - add interface swp47 bridge trunk vlans {{ INET_VLANS | join(",") }}
 atomic: yes
 description: connect to INET networks
 when: inventory_hostname == "leaf3"

- name: BGP peering on leaf1
  nclu:
    commands:
    - add vlan 158 ip address {{ myvars.INET_ADDR }}
    - add bgp ipv4 unicast neighbor {{ myvars.ODL_PEER }} activate
    - add bgp evpn neighbor {{ myvars.ODL_PEER }} activate
    atomic: yes
    description: ODL EVPN peering
...

L3 Spine tasks

[ops@vco-jump tasks]$ cat roles/l3-spine/main.yaml 
#-----------------------------------------------------------------------------------------
#
# L3 fabric and EVPN for L2 connectivity
#
---

- set_fact:
   myvars: "{{ switchvars[inventory_hostname] }}"

- name: Clean the slate and set up unnumbered BGP fabric and VNI advertisement
  nclu:
    commands:
    - add bgp autonomous-system {{ myvars.ASN }}
    - add bgp router-id {{ myvars.LOOPBACK_4 }}
    - add loopback lo ip address {{ myvars.LOOPBACK_4 }}/32
    - add bgp ipv4 unicast network {{ myvars.LOOPBACK_4 }}/32
    - add loopback lo ipv6 address {{ myvars.LOOPBACK_6 }}/128
    - add bgp ipv6 unicast network {{ myvars.LOOPBACK_6 }}/128
    - add interface {{ AGG_PORTS }} mtu 9216
    - add bgp neighbor {{ AGG_PORTS }} interface remote-as external 
    - add bgp ipv4 unicast neighbor {{ AGG_PORTS }} activate 
    - add bgp ipv6 unicast neighbor {{ AGG_PORTS }} activate 
    - add bgp evpn neighbor {{ AGG_PORTS }} activate
   atomic: true
   description: l3-fabric with EVPN
...

Bringing it all home

As you can tell, I’m pretty jacked about how far this technology has come. It’s become relatively trivial for people to leverage IP fabrics in their networks and use VXLAN/EVPN to “plumb” any L2 domains that they need to propagate. An awesome benefit of the Cumulus Linux VXLAN/EVPN technology is that it’s based on the Linux networking model and open components — you can peer a Linux host into your EVPN fabric and build some pretty bitchin’ networks.

Stay tuned for more on the OPNFV demo, it was really cool!

If you’d like to learn more about EVPN, check out this blog post.

#flybefree
JR