Blog

Interprovider Layer 3 VPN option C

September 19, 2015 9:42 am

What if, for whatever reason, a customer L3VPN would need to span multiple service providers? RFC 4364 describes three options of Interprovider VPNs that do this. This article is about option C, the most scalable of the three.

The RFC describes option C as follows: ‘Multi-hop EBGP redistribution of labeled VPN-IPv4 routes between source and destination ASes, with EBGP redistribution of labeled IPv4 routes from AS to neighboring AS’.

That is a bit of a mouthful, but if you’re familiar with regular L3VPNs, the Interprovider VPN option C is really not that hard to understand. Let’s walkthrough the following topology:

Interprovider layer 3 VPN option C

 

In this topology, the setup of AS1 and AS2 is practically identical. Both ASes contain an edge, a P and a PE router. Inside the AS, these routers are running OSPF and LDP. Both PE routers connect to a CPE. The connection to this CPE is placed inside an L3VPN that will be made active on as1-pe and as2-pe. This L3VPN will span two ASes, making it an Interprovider layer 3 VPN option C.

When comparing the Interprovider VPN option C, to a ‘normal’ L3VPN setup, there are two differences that stand out the most. The first difference is that an LSP needs to be established between PE routers located in different ASes. This is achieved by using BGP to distribute the label for the loopback IP addresses of the PE routers. The second difference is that the PE routers will use an EBGP multihop peering session to exchange routing information for routes inside the L3VPN.

This second difference, the exchange of routes between the PE routers, is what makes this option of Interprovider VPN the most scalable one. None of the routers in between the two PE routers need to be aware of any VPN related routing information. The only extra routing information that the other routers are burdened with are the labeled routes towards the PE routers located in the other AS.

Let’s start off by briefly touching the situation inside the AS and then continue with the establishment of the LSPs between the two PE routers. After that, we’ll look into the EBGP peering session between the PE routers as well as the L3VPN configuration on the PE routers.

 

The situation inside the ASes.

To keep things simple, both ASes are running the exact same setup. As stated before, the establishment of LSPs between the routers inside the ASes is achieved by activating OSPF, LDP and MPLS on the relevant interfaces.

 

Since this configuration is (almost) the same on all routers, showing the configuration of the as1-p router should give you a clear indication on how things are configured:

as1-p:

set interfaces xe-0/2/0 unit 1505 description as1-edge
set interfaces xe-0/2/0 unit 1505 vlan-id 1505
set interfaces xe-0/2/0 unit 1505 family inet address 192.168.200.1/30
set interfaces xe-0/2/0 unit 1505 family mpls

set interfaces ae1 unit 4001 description as1-pe
set interfaces ae1 unit 4001 vlan-id 4001
set interfaces ae1 unit 4001 family inet address 192.168.200.6/30
set interfaces ae1 unit 4001 family mpls

set interfaces lo0 unit 3 family inet address 192.168.0.3/32

set protocols mpls interface xe-0/2/0.1505
set protocols mpls interface ae1.4001

set protocols ospf area 0.0.0.0 interface xe-0/2/0.1505 interface-type p2p
set protocols ospf area 0.0.0.0 interface lo0.3
set protocols ospf area 0.0.0.0 interface ae1.4001 interface-type p2p

set protocols ldp interface xe-0/2/0.1505
set protocols ldp interface ae1.4001

By applying this configuration to routers in both ASes, the connectivity between the routers inside each AS is taken care of.

 

Establishing an LSP between the PE-routers.

We need to provide the as1-pe with an LSP towards the as2-pe, and vice versa:

 

To make this happen, we will first need to configure BGP with the ‘inet labeled-unicast’ address family. By doing so, we enable BGP to advertise both IP addresses and MPLS label information. We first configure this for the edge routers of both AS1 and AS2. These routers will need to be configured to exchange a prefix and a label for the PE located in their own AS. To accomplish this, we can configure something like the following:

 

as1-edge:

set interfaces ae1 unit 4000 description as2-edge
set interfaces ae1 unit 4000 vlan-id 4000
set interfaces ae1 unit 4000 family inet address 10.0.0.1/30
set interfaces ae1 unit 4000 family mpls

set protocols mpls traffic-engineering bgp-igp-both-ribs

set protocols bgp group ebgp type external
set protocols bgp group ebgp export export-label
set protocols bgp group ebgp neighbor 10.0.0.2 description as2-edge
set protocols bgp group ebgp neighbor 10.0.0.2 family inet labeled-unicast
set protocols bgp group ebgp neighbor 10.0.0.2 peer-as 2

set policy-options policy-statement export-label term as1-pe from route-filter 10.1.0.1/32 exact
set policy-options policy-statement export-label term as1-pe then accept
set policy-options policy-statement export-label term reject then reject

set routing-options autonomous-system 1

as2-edge:

set interfaces ae1 unit 4000 description as1-edge
set interfaces ae1 unit 4000 vlan-id 4000
set interfaces ae1 unit 4000 family inet address 10.0.0.2/30
set interfaces ae1 unit 4000 family mpls

set protocols mpls traffic-engineering bgp-igp-both-ribs

set protocols bgp group ebgp type external
set protocols bgp group ebgp export export-label
set protocols bgp group ebgp neighbor 10.0.0.1 description as1-edge
set protocols bgp group ebgp neighbor 10.0.0.1 family inet labeled-unicast
set protocols bgp group ebgp neighbor 10.0.0.1 peer-as 1

set policy-options policy-statement export-label term as2-pe from route-filter 192.168.0.2/32 exact
set policy-options policy-statement export-label term as2-pe then accept
set policy-options policy-statement export-label term reject then reject

set routing-options autonomous-system 2

The ‘inet labeled-unicast’ will make the routers not only advertise an IP address using BGP, but a corresponding label as well. The export policy will make sure that the only address advertised is the one belonging to the PE in the originating AS. To see if this is really the case, we can look at the following:

inetzero@as1-edge> show route advertising-protocol bgp 10.0.0.2 detail

inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden)
* 10.1.0.1/32 (2 entries, 2 announced)
 BGP group ebgp type External
     Route Label: 301168
     Nexthop: Self
     Flags: Nexthop Change
     MED: 1
     AS path: [1] I

Another important aspect of the configuration that was applied to both edge routers happened inside the [ protocols mpls traffic-engineering ] stanza. The default value here is ‘bgp’. By configuring either ‘bgp-igp-both-ribs’ or ‘bgp-igp’, we can make the edge routers use the LSP towards the PE router located in their own AS. Since the necessity and explanation of this command was something I needed to read several times, I’d like to try and elaborate a little further on why this is necessary.

The as1-edge has an LDP route towards the as1-pe. This route is placed in the inet.3 table. However, as soon as the as2-edge forwards packets to the as1-pe, the as1-edge router will not use the mpls path information from table inet.3 by default. This is because by default, only BGP can use the inet.3 table recursively.

If the as2-edge sends a labeled packet towards the as1-pe, the as1-edge will ‘pop’ the label and forward the packet on to as1-pe using the OSPF route in the inet.0 table. When building an Interprovider VPN, this is a problem. This is a problem because after that label, there is a VPN label. When the as1-edge pops the top label, it will confront the P router with a VPN label. And that P router will not know what to do with it.

Let’s examine the situation before enabling ‘bgp-igp-both-ribs’:

 

The as1-egde has the following OSPF route towards the as1-pe: 

 inetzero@as1-edge> show route 10.1.0.1 table inet.0

inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.1.0.1/32        *[OSPF/10] 00:00:05, metric 2
                    > to 192.168.200.1 via xe-0/3/0.1505

 

Let’s verify what the as1-edge router does with labeled traffic by examining the mpls.0 table. First, we need to know what label the as1-edge advertised to the as2-edge for the as1-pe:

inetzero@as1-edge> show route advertising-protocol bgp 10.0.0.2 detail

inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
* 10.1.0.1/32 (1 entry, 1 announced)
 BGP group ebgp type External
     Route Label: 301312
     Nexthop: Self
     Flags: Nexthop Change
     MED: 2
     AS path: [1] I

So the as1-edge told the as2-edge that it can use label 301312 to reach the as1-pe. When we consult the mpls table on the as1-edge, we can find out what the as1-edge will do with this label:

inetzero@as1-edge> show route table mpls.0 label 301312

mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301312             *[VPN/170] 00:00:17
                    > to 192.168.200.1 via xe-0/3/0.1505, Pop
301312(S=0)        *[VPN/170] 00:00:17
                    > to 192.168.200.1 via xe-0/3/0.1505, Pop

As stated before, since we want to exchange VPN traffic, this behavior needs to change. We can do this by configuring the router with ‘set protocols mpls traffic-engineering bgp-igp-both-ribs’:

 

 

 Let’s examine the changes on the router by issuing the same commands:

inetzero@as1-edge> show route 10.1.0.1 table inet.0

inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.1.0.1/32        *[LDP/9] 00:02:55, metric 1
                    > to 192.168.200.1 via xe-0/3/0.1505, Push 299792
                    [OSPF/10] 00:02:55, metric 2
                    > to 192.168.200.1 via xe-0/3/0.1505

The ‘bgp-igp-both-ribs’ configuration command copied the route from the inet.3 table into the inet.0 table. So now, the route towards the as1-pe has changed from OSPF into LDP. After using the configuration command ‘set protocols mpls traffic-engineering bgp-igp-both-ribs’, the advertised label changed. So before looking at the MPLS table, we need to check what label the as1-edge router is currently advertising:

inetzero@as1-edge> show route advertising-protocol bgp 10.0.0.2 detail

inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden)
* 10.1.0.1/32 (2 entries, 2 announced)
 BGP group ebgp type External
     Route Label: 301296
     Nexthop: Self
     Flags: Nexthop Change
     MED: 1
     AS path: [1] I

So now, the as1-edge instructed the as2-edge to use label 301296 when it wants to reach the as1-pe. This time, when we consult the mpls table, we should see a ‘swap’ operation:

inetzero@as1-edge> show route table mpls.0 label 301296

mpls.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301296             *[VPN/170] 00:03:02
                    > to 192.168.200.1 via xe-0/3/0.1505, Swap 299792

Even though it might look so now, the end-to-end LSP is not complete yet. The PE routers don’t have the information necessary to reach the other PE router. Let’s look at the configuration needed to accomplish this in AS1:

as1-edge:

set protocols bgp group ibgp type internal
set protocols bgp group ibgp local-address 192.168.0.1
set protocols bgp group ibgp family inet labeled-unicast 
set protocols bgp group ibgp export ibgp
set policy-options policy-statement ibgp term remote-pe then next-hop self
set protocols bgp group ibgp neighbor 10.1.0.1

as1-pe:

set protocols bgp group ibgp type internal
set protocols bgp group ibgp local-address 10.1.0.1
set protocols bgp group ibgp family inet labeled-unicast resolve-vpn
set protocols bgp group ibgp neighbor 192.168.0.1

The ‘family inet labeled-unicast’ is enough to make BGP advertise the necessary labels, but the PE requires an additional knob: ‘resolve-vpn’. This command will allow the PE router to place the route in the inet.3 table. Without this configuration command, the PE routers would not be able to resolve any of the VPN routes we intend to make them exchange.

So, after configuring IBGP for the routers in both ASes, a usable LSP exists between the PE routers:

 

On to the multihop EBGP exchange of routing information for the L3VPN.

 

The multihop EBGP exchange of routing information for the L3VPN.

After establishing an LSP between the two PE routers, there are two things left to configure. These are the L3VPN itself and the EBGP multihop session between the PE routers.

Let’s have a look at the L3VPN first. The configuration is kept the exact same on both PE routers:

set interfaces xe-2/0/1 unit 2 description cpe-as1
set interfaces xe-2/0/1 unit 2 vlan-id 2
set interfaces xe-2/0/1 unit 2 family inet mtu 1500
set interfaces xe-2/0/1 unit 2 family inet address 172.16.10.2/30
set interfaces lo0 unit 4000 family inet address 172.16.0.1/32

set policy-options policy-statement to-cpe term bgp from protocol bgp
set policy-options policy-statement to-cpe term bgp then accept
set policy-options policy-statement to-cpe term direct from protocol direct
set policy-options policy-statement to-cpe term direct then accept

set routing-instances ipvpn instance-type vrf
set routing-instances ipvpn interface xe-2/0/1.2
set routing-instances ipvpn interface lo0.4000
set routing-instances ipvpn route-distinguisher 1:1
set routing-instances ipvpn vrf-target target:1:1
set routing-instances ipvpn vrf-table-label
set routing-instances ipvpn protocols ospf export to-cpe
set routing-instances ipvpn protocols ospf area 0.0.0.0 interface xe-2/0/1.2 interface-type p2p

OSPF is used to exchange routes between the PE and the CPE. But to make the as1-pe and the as2-pe exchange routes, an EBGP session between the two PE routers is needed:

 

This configuration could be done in the following way:

as1-pe:

set protocols bgp group pe-as-1 type external
set protocols bgp group pe-as-1 multihop ttl 10
set protocols bgp group pe-as-1 local-address 192.168.0.2
set protocols bgp group pe-as-1 family inet-vpn unicast
set protocols bgp group pe-as-1 peer-as 1
set protocols bgp group pe-as-1 neighbor 10.1.0.1 description as1-pe

as2-pe:

set protocols bgp group pe-as-2 type external
set protocols bgp group pe-as-2 multihop ttl 10
set protocols bgp group pe-as-2 local-address 10.1.0.1
set protocols bgp group pe-as-2 family inet-vpn unicast
set protocols bgp group pe-as-2 peer-as 2
set protocols bgp group pe-as-2 neighbor 192.168.0.2 description as2-pe

Other than the multihop configuration statement, this peering session contains nothing really out of the ordinary.

Let’s verify things from the as1-pe by checking the BGP session with the as2-pe router:

inetzero@as1-pe> show bgp summary group pe-as-2
Groups: 2 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l3vpn.0
                       3          3          0          0          0          0
inet.0
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.0.2               2         23         23       0       0        8:21 Establ
  bgp.l3vpn.0: 3/3/3/0
  ipvpn.inet.0: 3/3/3/0

This shows us that the session is up and that there are three active, received and accepted routes. Since there is only one L3VPN active on this PE, the number of routes received in the ‘bgp.l3vpn.0’ table is equal to the number of routes received in the ‘ipvpn.inet.0’ table.

If all is well, the CPEs should be able to send traffic to each other’s loopback IP address:

 inet-zero-interprovider-vpn-option-c-9

 

 inetzero@cpe-as1> ping 192.168.250.2 source 192.168.250.1 count 5 rapid

PING 192.168.250.2 (192.168.250.2): 56 data bytes
!!!!!
--- 192.168.250.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.792/0.987/1.706/0.360 ms

And the Interprovider Layer 3 VPN option C is up!

In closing, let’s look at the complete setup once more and examine how the as1-pe was able to forward those packets:

  

 

 Routers in both AS1 and AS2 are all running OSPF and LDP. This way, inside their own AS, all routers have LDP routes to each other.

The as1-edge and as2-edge routers are exchanging labeled routes for the PE routers located in their own AS. These labeled routes are exchanged by using an EBGP session.

Both as1-edge and as2-edge routers are also running IGBP with the PE routers located inside their own AS. The edge routers use this IBGP session to advertise the labeled route for the PE located in the other AS. This way, both as1-pe and as2-pe learn of a labeled unicast route towards the other routers’ loopback address.

Both as1-pe and as2-pe are engaged in an EBGP session with each other to exchange the routes for the L3VPN.

Let’s examine the forwarding table entry on the as1-pe router and look at the loopback address of cpe-as2:

inetzero@as1-pe> show route forwarding-table vpn ipvpn destination 192.168.250.2
Logical system: as1-pe
Routing table: ipvpn.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
192.168.250.2/32   user     0                    indr  1048575     4
                              192.168.200.6     Push 16, Push 301488, Push 299776(top)      630     2 ae1.4001

This is a forwarding entry with three labels; a VPN label, a label to reach the other PE and a label to reach the as1-edge router. Let’s briefly examine how these labels were learned again and start with the VPN label:

inetzero@as1-pe> show route receive-protocol bgp 192.168.0.2 table ipvpn.inet.0 192.168.250.2/32 detail

ipvpn.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
* 192.168.250.2/32 (1 entry, 1 announced)
     Import Accepted
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: 192.168.0.2
     MED: 1
     AS path: 2 I
     Communities: target:1:1 rte-type:0.0.0.0:1:0

The next-hop address is the loopback of the as2-pe. The labeled route towards this PE was learned from the as1-edge router:

inetzero@as1-pe> show route receive-protocol bgp 192.168.0.1 detail

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
* 192.168.0.2/32 (1 entry, 1 announced)
     Accepted
     Route Label: 301488
     Nexthop: 192.168.0.1
     MED: 1
     Localpref: 100
     AS path: 2 I

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

* 192.168.0.2/32 (1 entry, 1 announced)
     Accepted
     Route Label: 301488
     Nexthop: 192.168.0.1
     MED: 1
     Localpref: 100
     AS path: 2 I

Because of the ‘next-hop self’ policy that was applied to the as1-edge router, the next-hop for this route is the as1-edge routers’ own loopback IP address. To be able to reach the as1-edge loopback IP address, the top label that the as1-pe uses corresponds with the route towards the as1-edge loopback IP address. We can see how this label was learned simply by consulting the inet.3 table, asking for the route towards this loopback IP address:

inetzero@as1-pe> show route 192.168.0.1 table inet.3

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.0.1/32     *[LDP/9] 00:50:18, metric 1
                    > to 192.168.200.6 via ae1.4001, Push 299776

After analyzing that, let’s follow the packet up until the as1-edge router by examining what the as1-p router does with this top label. The as1-p router will be confronted with the top label. To find out what that router will do, the easiest thing is to consult the mpls.0 tabel:

inetzero@as1-p> show route table mpls.0 label 299776

mpls.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299776             *[LDP/9] 00:54:29, metric 1
                    > to 192.168.200.2 via xe-0/2/0.1505, Pop
299776(S=0)        *[LDP/9] 00:54:29, metric 1
                    > to 192.168.200.2 via xe-0/2/0.1505, Pop

The as1-p router POPs the label. This makes sense, since the next hop is the destination. This leaves the as1-edge with the two remaining labels: 16 and 301488.

inetzero@as1-edge> show route table mpls label 301488

mpls.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301488             *[VPN/170] 00:56:31
                    > to 10.0.0.2 via ae1.4000, Swap 301056

The label is swapped to 301056. This label, used to reach the PE in the other AS, was learned from the as2-edge:

inetzero@as1-edge> show route receive-protocol bgp 10.0.0.2 detail

inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden)
* 192.168.0.2/32 (1 entry, 1 announced)
     Accepted
     Route Label: 301056
     Nexthop: 10.0.0.2
     MED: 1
     AS path: 2 I

Since AS2 is sort of a mirror image of the setup in AS1, I’ll stop the walkthrough right here.

Hopefully, I have been able to shed some light on how the Interprovider VPN option C works and how you can configure it. 

In closing, I wanted to mention two more things. First of all, I configured a Layer 3 VPN example. The Interprovider VPN option C can also be used for a Layer 2 VPNs. Second, I configured the PE routers to exchange the routes directly with each other because this was easiest. To further enhance the scalability, you could choose to have the multi-hop EBGP session exist between route-reflectors.

More interprovider L3VPN scenario’s can be found in iNETZERO’s JNCIE-SP lab workbooks

 

 

Said van de Klundert

Said van de Klundert

Netherlands based networking enthusiast and Juniper Networks ambassador. Has spent most of his career on the service-providers’ side of things. Known to lab up and write about whatever sparked his interest networking wise. Other than that, he is a father to his son, husband to his wife and he enjoys long dinners with friends.

More Posts - Website - Twitter

Categorised in:

  • George G Fouad says:

    Good post.

    One note…

    The main goal of option C is to have a routing entry for the remote PE loopback in inet.0 as well as inet.3 on each PE. (why? the entry in inet.0 is essential for MP-eBGP session establishment, while the entry in inet.3 is essential for VPN routes resolution).

    There two methods to achieve this.

    One method described here uses only family (inet labeled-unicast) in inet.0. yet it must include “bgp-igp-both-ribs” on ASBRs and “resolve-vpn” on PEs to make sure labeled routes exist in inet.0 on ASBRs and in inet.3 on PEs.

    Another method is to use two families (inet unicast) and (inet labeled-unicast) with BGP peers on all routers, but both families can’t work on the same rib (inet.0).. that’s why “rib inet.3” knob is needed under family (inet labeled-unicast) in this case.

  • Shiva Shankar says:

    I believe one more elegant way to exchange the label-route from edge to PEs especially when you have RRs in each ASs is
    1. Establish multi-hop eBGP session between RR to exchange VPN routes
    2. Redistribute labeled-routes from BGP onto OSPF/LDP on Edge routers.

    This will remove the need to have a third label in the label stack while observed on the PE.
    Thanks

  • MOHIT says:

    Why you have drawn as1-p in AS2 network?

  • guy says:

    I am trying to simulate this using 4 x SRX220 and it doesn’t work, can you please confirm, SRX-220 version 12.1X44-D35.5 support such configuration.

    In your sample, I see “swap”:
    inetzero@as1-edge> show route table mpls.0 label 301296
    mpls.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
    + = Active Route, – = Last Active, * = Both
    301296 *[VPN/170] 00:03:02
    > to 192.168.200.1 via xe-0/3/0.1505, Swap 299792

    But at my setup, I see “POP” can you tell what I do wrong?
    i needed I send my configuration.
    thanks
    guy

Leave a Reply

Your email address will not be published. Required fields are marked *