Blog

QFX controllerless overlay with all-active Ethernet segments.

June 1, 2017 11:56 am

The QFX is a very versatile switch with a lot of deployment options. In this article, I want to go over a scenario in which 4 QFX switches are deployed to form a controllerless-overlay:

topology

The main characters here are the QFX switches. As an overlay, they will offer an all-active layer 2 connection to both MX routers.  

In this walkthrough, I will cover the following configuration steps:

  1. Underlay.
  2. Signaling for the overlay.
  3. QFX-switch level EVPN.
  4. Multihoming and the addition of a VXLAN segment.
  5. Adding an additional VXLAN segment.

 

Configuring the underlay.

The underlay network has one main function: providing IP connectivity between the loopback IP addresses of the switches. To achieve our goal, really any protocol would work. We can use BGP, IS-IS, OSPF or something else. For this example, I chose to use OSPF:

qfx-only

vQFX1:

set protocols ospf area 0.0.0.0 interface xe-0/0/6.100
set protocols ospf area 0.0.0.0 interface xe-0/0/6.103
set protocols ospf area 0.0.0.0 interface lo0.0 passive


vQFX2:

set protocols ospf area 0.0.0.0 interface xe-0/0/6.100
set protocols ospf area 0.0.0.0 interface xe-0/0/6.101
set protocols ospf area 0.0.0.0 interface lo0.0 passive


vQFX3:

set protocols ospf area 0.0.0.0 interface xe-0/0/6.101
set protocols ospf area 0.0.0.0 interface xe-0/0/6.102
set protocols ospf area 0.0.0.0 interface lo0.0 passive


vQFX4:

set protocols ospf area 0.0.0.0 interface xe-0/0/6.102
set protocols ospf area 0.0.0.0 interface xe-0/0/6.103
set protocols ospf area 0.0.0.0 interface lo0.0 passive

 

Configuring signaling for the overlay.

 

When Juniper QFX switches are used to build a controllerless overlay, control plane signaling is taken care of by BGP. The overlay is not very big, so instead of turning to any of the BGP scaling mechanisms, we simply configure a full-mesh. Every BGP session needs to support the EVPN address family. For the sake of stability, we source the BGP sessions from the loopback interface on every QFX:

 

qfx-only-lo0

 

vQFX1:

set routing-options autonomous-system 65400

set protocols bgp group overlay type internal
set protocols bgp group overlay local-address 10.0.0.1
set protocols bgp group overlay family evpn signaling
set protocols bgp group overlay neighbor 10.0.0.2
set protocols bgp group overlay neighbor 10.0.0.3
set protocols bgp group overlay neighbor 10.0.0.4


vQFX2:

set routing-options autonomous-system 65400

set protocols bgp group overlay type internal
set protocols bgp group overlay local-address 10.0.0.2
set protocols bgp group overlay family evpn signaling
set protocols bgp group overlay neighbor 10.0.0.1
set protocols bgp group overlay neighbor 10.0.0.3
set protocols bgp group overlay neighbor 10.0.0.4


vQFX3:

set routing-options autonomous-system 65400

set protocols bgp group overlay type internal
set protocols bgp group overlay local-address 10.0.0.3
set protocols bgp group overlay family evpn signaling
set protocols bgp group overlay neighbor 10.0.0.1
set protocols bgp group overlay neighbor 10.0.0.2
set protocols bgp group overlay neighbor 10.0.0.4


vQFX4:

set routing-options autonomous-system 65400

set protocols bgp group overlay type internal
set protocols bgp group overlay local-address 10.0.0.4
set protocols bgp group overlay family evpn signaling
set protocols bgp group overlay neighbor 10.0.0.1
set protocols bgp group overlay neighbor 10.0.0.2
set protocols bgp group overlay neighbor 10.0.0.3

 

Configure QFX-switch level EVPN

 

To enable a single QFX to be part of a controllerless overlay, we need to define several switch-level parameters. For this scenario, we want to build a controllerless overlay that uses VXLAN encapsulation and ingress replication. We also want to allow the switch to import any signaled VXLAN segments we configure later on with minimal configuration overhead. To this end, we configure the following parameters under the [ protocols evpn ] stanza:

vQFX1:

set protocols evpn encapsulation vxlan
set protocols evpn extended-vni-list all
set protocols evpn multicast-mode ingress-replication

 

Next up is the configuration of the switch-options.  Here we configure the vtep-source-interface, the route-distinguisher and the route-target:

 

vQFX1:

set switch-options vtep-source-interface lo0.0
set switch-options route-distinguisher 10.0.0.1:65400
set switch-options vrf-target target:100:100
set switch-options vrf-target auto

 

We have configured two vrf-target values. The first value, vrf-target:100:100 will be attached to type 1 routes. The auto statement will ensure that for every VXLAN segment, a route-target is generated automatically. Both vrf-target statements will cause for the QFX to create and apply ‘hidden’ policies. These policies take care of the route-target import and export.

 

Last step here is to configure a router-ID. In order to have the QFX commit, the router-id has to match the lo0 address:

 

vQFX1:

set routing-options router-id 10.0.0.1

 

The QFX-switch level configuration shown here is for vQFX1. The other vQFX switches can be configured with a similar configuration. Only two things differ, the route-distinguisher value and the router-id. On the other switches, instead of using 10.0.0.1, the IP address configured as the loopback is used.

 

Configure Multihoming and a VXLAN segment.

 

Even though we have prepared every QFX for switch-level EVPN, nothing is happing yet. There is no signaling going on in our overlay and forwarding across VXLAN tunnels is not possible. Let’s change that by adding the multihomed connections the MX routers as well as adding a VXLAN segment.

During this part of the configuration, we need to:

  • enable each QFX for LAGs
  • assign interfaces to the LAGs
  • configure the LAGs as multihomed Ethernet Segments
  • create the VXLAN segment and add that to the LAG interfaces

As far as the LAG is concerned, we have a few things to bear in mind. First of all, the MX routers need to think that they are communicating with a single device. To this end, we configure the same lacp system-id on the QFX pairs. Another thing is the esi configuration. We will configure the Ethernet segment to be all-active and we need to specify a unique ESI on each QFX pair. The ESI identifier needs to be unique on a network-wide basis.

After taking into account those specificities, the rest of the LAG configuration is exactly like a standard trunk interface. On each QFX, we conclude the configuration with the creation of a VXLAN segment. In this case, this is really nothing more than the configuration of a VLAN with a VNI value.

 

vQFX1:

set chassis aggregated-devices ethernet device-count 1

set interfaces xe-0/0/3 ether-options 802.3ad ae0

set interfaces ae0 esi 00:12:12:12:12:12:12:12:12:12
set interfaces ae0 esi all-active
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-id 12:12:12:12:12:12
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members 100

set vlans vlan-100 vlan-id 100 vxlan vni 100



vQFX2:

set chassis aggregated-devices ethernet device-count 1

set interfaces xe-0/0/2 ether-options 802.3ad ae0

set interfaces ae0 esi 00:12:12:12:12:12:12:12:12:12
set interfaces ae0 esi all-active
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-id 12:12:12:12:12:12
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members 100

set vlans vlan-100 vlan-id 100 vxlan vni 100


vQFX3:

set chassis aggregated-devices ethernet device-count 1

set interfaces xe-0/0/2 ether-options 802.3ad ae0

set interfaces ae0 esi 00:34:34:34:34:34:34:34:34:34
set interfaces ae0 esi all-active
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-id 34:34:34:34:34:34
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members 100

set vlans vlan-100 vlan-id 100 vxlan vni 100


vQFX4:

set chassis aggregated-devices ethernet device-count 1

set interfaces xe-0/0/2 ether-options 802.3ad ae0

set interfaces ae0 esi 00:34:34:34:34:34:34:34:34:34
set interfaces ae0 esi all-active
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-id 34:34:34:34:34:34
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members 100

set vlans vlan-100 vlan-id 100 vxlan vni 100

 

 Adding an additional VXLAN segment.

 

Now that we have completed our controllerless overlay, let’s have a look at the configuration overhead for adding additional VXLAN segments.

To add a VXLAN segment for VLAN 101 on every multihomed interface, we need to configure the following on every QFX device:

 

set interfaces ae0 unit 0 family ethernet-switching vlan members 101
set vlans vlan-101 vlan-id 101 vxlan vni 101

 

Verification:

 

Let’s start out verifying what we can from the MX point of view. We can see that the LAG between MX1 and the QFX switches is operational:

jncie@vMX1> show lacp interfaces extensive      
Aggregated interface: ae0
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/2       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/2     Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/3       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/3     Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      xe-0/0/2                  Current   Fast periodic Collecting distributing
      xe-0/0/3                  Current   Fast periodic Collecting distributing
    LACP info:        Role     System             System       Port     Port    Port
                             priority         identifier   priority   number     key
      xe-0/0/2       Actor        127  00:05:86:72:fb:c0        127        3       1
      xe-0/0/2     Partner        127  12:12:12:12:12:12        127        1       1
      xe-0/0/3       Actor        127  00:05:86:72:fb:c0        127        4       1
      xe-0/0/3     Partner        127  12:12:12:12:12:12        127        1       1

 

Sending traffic across both VXLAN segments:

jncie@vMX1> ping 192.168.1.2 rapid           
PING 192.168.1.2 (192.168.1.2): 56 data bytes
!!!!!
--- 192.168.1.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 52.196/65.551/73.717/7.806 ms


jncie@vMX1> ping 192.168.2.2 rapid   
PING 192.168.2.2 (192.168.2.2): 56 data bytes
!!!!!
--- 192.168.2.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 54.500/135.347/285.082/90.370 ms

 

Now on to the QFX. The easiest verification command is the ‘show evpn instance extensive’ command. The information that the command outputs is incredibly valuable. If you have examined it in detail a few times over, you’ll be able to scan through the information quickly:

 

jncie@vQFX1> show evpn instance extensive
Instance: __default_evpn__
  Route Distinguisher: 10.0.0.1:0
  Number of bridge domains: 0
  Number of neighbors: 1
    10.0.0.2
      Received routes
        Ethernet Segment:                       1


Instance: default-switch
  Route Distinguisher: 10.0.0.1:65400
  Encapsulation type: VXLAN
  MAC database status                     Local  Remote
    MAC advertisements:                       0       6
    MAC+IP advertisements:                    0       0
    Default gateway MAC advertisements:       0       0
  Number of local interfaces: 1 (1 up)
    Interface name  ESI                            Mode             Status
    ae0.0           00:12:12:12:12:12:12:12:12:12  all-active       Up   
  Number of IRB interfaces: 0 (0 up)
  Number of bridge domains: 2
    VLAN  VNI    Intfs / up    IRB intf   Mode             MAC sync  IM route label
    100   100        1   1                Extended         Enabled   100   
    101   101        1   1                Extended         Enabled   101   
  Number of neighbors: 3
    10.0.0.2
      Received routes
        MAC address advertisement:              2
        MAC+IP address advertisement:           0
        Inclusive multicast:                    2
        Ethernet auto-discovery:                2
    10.0.0.3
      Received routes
        MAC address advertisement:              2
        MAC+IP address advertisement:           0
        Inclusive multicast:                    2
        Ethernet auto-discovery:                2
    10.0.0.4
      Received routes
        MAC address advertisement:              2
        MAC+IP address advertisement:           0
        Inclusive multicast:                    2
        Ethernet auto-discovery:                2
  Number of ethernet segments: 2
    ESI: 00:12:12:12:12:12:12:12:12:12
      Status: Resolved by IFL ae0.0
      Local interface: ae0.0, Status: Up/Forwarding
      Number of remote PEs connected: 1
        Remote PE        MAC label  Aliasing label  Mode
        10.0.0.2         100        0               all-active  
      Designated forwarder: 10.0.0.1
      Backup forwarder: 10.0.0.2
    ESI: 00:34:34:34:34:34:34:34:34:34
      Status: Resolved
      Number of remote PEs connected: 2
        Remote PE        MAC label  Aliasing label  Mode
        10.0.0.4         101        0               all-active  
        10.0.0.3         101        0               all-active  
  Router-ID: 10.0.0.1
  Source VTEP interface IP: 10.0.0.1

 

There are dozens of other verification commands at our disposal, but to keep things simple, I would like to wrap up with some of the most common commands used to examine the switching table:

 

jncie@vQFX1> show ethernet-switching table                                    
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 4 entries, 4 learned
Routing instance : default-switch
   Vlan                MAC                 MAC      Logical                Active                       
   name                address             flags    interface              source                      
   vlan-100            00:00:00:00:00:aa   DR       ae0.0               
   vlan-100            00:00:00:00:00:bb   DR       esi.1702               00:34:34:34:34:34:34:34:34:34 
   vlan-101            00:00:00:00:00:aa   DR       ae0.0               
   vlan-101            00:00:00:00:00:bb   DR       esi.1702               00:34:34:34:34:34:34:34:34:34 

 

jncie@vQFX1> show evpn database   
Instance: default-switch
VLAN  VNI  MAC address        Active source                  Timestamp        IP address
      100   00:00:00:00:00:aa  00:12:12:12:12:12:12:12:12:12  May 23 09:12:44
      100   00:00:00:00:00:bb  00:34:34:34:34:34:34:34:34:34  May 23 09:12:44
      101   00:00:00:00:00:aa  00:12:12:12:12:12:12:12:12:12  May 23 09:09:19
      101   00:00:00:00:00:bb  00:34:34:34:34:34:34:34:34:34  May 23 09:12:46

 

Wrapping up:

 

We used 4 QFX switches to form an overlay network between two MX routers. On the QFX, the MX-facing interfaces were configured as all-active interfaces. This way, each MX router view the adjacent QFX-pair as a single device. The addition of VXLAN segments was made possible with minimal configuration overhead. We chose to have the VXLAN segments tagged with automatically generated route-targets.

The iNET ZERO JNCIE-DC workbook that currently being developed will include far more sophisticated labs, a more thorough walkthrough and additional theory on all the JNCIE-DC related topics.

Said van de Klundert

Said van de Klundert

Netherlands based networking enthusiast and Juniper Networks ambassador. Has spent most of his career on the service-providers’ side of things. Known to lab up and write about whatever sparked his interest networking wise. Other than that, he is a father to his son, husband to his wife and he enjoys long dinners with friends.

More Posts - Website - Twitter

Categorised in:

  • Hannes says:

    Hi,

    I like this setup very much, but I am wondering:

    how is a broadcast packet coming from one of the MX routers handled?
    It will pass over one of the LACP links to one of the QFX switches and then it will be passed on to the other QFXs. Won’t it be passed back to the originating MX via the other LACP link?

    • EVPN introduces several rules to handle BUM (Broadcast, Unknown unicast and Multicast) traffic. These rules prevent BUM traffic from endlessly looping between the PE and the CE.
      When we look at the scenario in this article, vQFX1 can receive BUM traffic directly from the MX, from the remote Ethernet Segment (vQFX3 and vQFX4) or from the other vQFX serving the same multihomed Ethernet segment (vQFX2). Let’s go over these three options.

      1: BUM traffic arriving from the MX.
      The vQFX1 switch will forward this traffic to all the other vQFX devices.

      2: BUM traffic arriving from vQFX3 or vQFX4.
      The vQFX1 and vQFX2 switches will elect a DF, Designated Forwarder, for the Ethernet Segment. The DF will be the only vQFX that will forward BUM traffic out interfaces connected to the Ethernet Segment. The non-DF vQFX will simply drop the BUM traffic.
      If vQFX1 is acting as the DF, it will forward any BUM traffic received from vQFX3 and vQFX4 towards the MX.
      If vQXF1 is the non-DF, the BUM traffic will not be send out to the MX. Instead, the BUM traffic will simply be discarded.

      3: BUM traffic sent by vQFX2:
      When vQFX2 receives BUM traffic from the MX, it will forward that traffic to all the other vQFX switches, including vQFX1. The DF and the non-DF do not play any role here. What prevents this traffic from looping is the ‘split horizon’ rule. This rule makes sure that BUM traffic coming from the same Ethernet Segment will not be forwarded out any interfaces that are attached to the same Ethernet Segment.

      For more information, you can check a white paper from Juniper called ‘Juniper Networks EVPN Implementation for Next-Generation Data Center Architectures’.

Leave a Reply

Your email address will not be published. Required fields are marked *