What if, for whatever reason, a customer L3VPN would need to span multiple service providers? RFC 4364 describes three options of Interprovider VPNs that do this. This article is about option C, the most scalable of the three.
The RFC describes option C as follows: ‘Multi-hop EBGP redistribution of labeled VPN-IPv4 routes between source and destination ASes, with EBGP redistribution of labeled IPv4 routes from AS to neighboring AS’.
That is a bit of a mouthful, but if you’re familiar with regular L3VPNs, the Interprovider VPN option C is really not that hard to understand. Let’s walkthrough the following topology:
In this topology, the setup of AS1 and AS2 is practically identical. Both ASes contain an edge, a P and a PE router. Inside the AS, these routers are running OSPF and LDP. Both PE routers connect to a CPE. The connection to this CPE is placed inside an L3VPN that will be made active on as1-pe and as2-pe. This L3VPN will span two ASes, making it an Interprovider layer 3 VPN option C.
When comparing the Interprovider VPN option C, to a ‘normal’ L3VPN setup, there are two differences that stand out the most. The first difference is that an LSP needs to be established between PE routers located in different ASes. This is achieved by using BGP to distribute the label for the loopback IP addresses of the PE routers. The second difference is that the PE routers will use an EBGP multihop peering session to exchange routing information for routes inside the L3VPN.
This second difference, the exchange of routes between the PE routers, is what makes this option of Interprovider VPN the most scalable one. None of the routers in between the two PE routers need to be aware of any VPN related routing information. The only extra routing information that the other routers are burdened with are the labeled routes towards the PE routers located in the other AS.
Let’s start off by briefly touching the situation inside the AS and then continue with the establishment of the LSPs between the two PE routers. After that, we’ll look into the EBGP peering session between the PE routers as well as the L3VPN configuration on the PE routers.
The situation inside the ASes.
To keep things simple, both ASes are running the exact same setup. As stated before, the establishment of LSPs between the routers inside the ASes is achieved by activating OSPF, LDP and MPLS on the relevant interfaces.
Since this configuration is (almost) the same on all routers, showing the configuration of the as1-p router should give you a clear indication on how things are configured:
as1-p:
set interfaces xe-0/2/0 unit 1505 description as1-edge set interfaces xe-0/2/0 unit 1505 vlan-id 1505 set interfaces xe-0/2/0 unit 1505 family inet address 192.168.200.1/30 set interfaces xe-0/2/0 unit 1505 family mpls set interfaces ae1 unit 4001 description as1-pe set interfaces ae1 unit 4001 vlan-id 4001 set interfaces ae1 unit 4001 family inet address 192.168.200.6/30 set interfaces ae1 unit 4001 family mpls set interfaces lo0 unit 3 family inet address 192.168.0.3/32 set protocols mpls interface xe-0/2/0.1505 set protocols mpls interface ae1.4001 set protocols ospf area 0.0.0.0 interface xe-0/2/0.1505 interface-type p2p set protocols ospf area 0.0.0.0 interface lo0.3 set protocols ospf area 0.0.0.0 interface ae1.4001 interface-type p2p set protocols ldp interface xe-0/2/0.1505 set protocols ldp interface ae1.4001
By applying this configuration to routers in both ASes, the connectivity between the routers inside each AS is taken care of.
Establishing an LSP between the PE-routers.
We need to provide the as1-pe with an LSP towards the as2-pe, and vice versa:
To make this happen, we will first need to configure BGP with the ‘inet labeled-unicast’ address family. By doing so, we enable BGP to advertise both IP addresses and MPLS label information. We first configure this for the edge routers of both AS1 and AS2. These routers will need to be configured to exchange a prefix and a label for the PE located in their own AS. To accomplish this, we can configure something like the following:
as1-edge:
set interfaces ae1 unit 4000 description as2-edge set interfaces ae1 unit 4000 vlan-id 4000 set interfaces ae1 unit 4000 family inet address 10.0.0.1/30 set interfaces ae1 unit 4000 family mpls set protocols mpls traffic-engineering bgp-igp-both-ribs set protocols bgp group ebgp type external set protocols bgp group ebgp export export-label set protocols bgp group ebgp neighbor 10.0.0.2 description as2-edge set protocols bgp group ebgp neighbor 10.0.0.2 family inet labeled-unicast set protocols bgp group ebgp neighbor 10.0.0.2 peer-as 2 set policy-options policy-statement export-label term as1-pe from route-filter 10.1.0.1/32 exact set policy-options policy-statement export-label term as1-pe then accept set policy-options policy-statement export-label term reject then reject set routing-options autonomous-system 1
as2-edge:
set interfaces ae1 unit 4000 description as1-edge set interfaces ae1 unit 4000 vlan-id 4000 set interfaces ae1 unit 4000 family inet address 10.0.0.2/30 set interfaces ae1 unit 4000 family mpls set protocols mpls traffic-engineering bgp-igp-both-ribs set protocols bgp group ebgp type external set protocols bgp group ebgp export export-label set protocols bgp group ebgp neighbor 10.0.0.1 description as1-edge set protocols bgp group ebgp neighbor 10.0.0.1 family inet labeled-unicast set protocols bgp group ebgp neighbor 10.0.0.1 peer-as 1 set policy-options policy-statement export-label term as2-pe from route-filter 192.168.0.2/32 exact set policy-options policy-statement export-label term as2-pe then accept set policy-options policy-statement export-label term reject then reject set routing-options autonomous-system 2
The ‘inet labeled-unicast’ will make the routers not only advertise an IP address using BGP, but a corresponding label as well. The export policy will make sure that the only address advertised is the one belonging to the PE in the originating AS. To see if this is really the case, we can look at the following:
inetzero@as1-edge> show route advertising-protocol bgp 10.0.0.2 detail inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden) * 10.1.0.1/32 (2 entries, 2 announced) BGP group ebgp type External Route Label: 301168 Nexthop: Self Flags: Nexthop Change MED: 1 AS path: [1] I
Another important aspect of the configuration that was applied to both edge routers happened inside the [ protocols mpls traffic-engineering ] stanza. The default value here is ‘bgp’. By configuring either ‘bgp-igp-both-ribs’ or ‘bgp-igp’, we can make the edge routers use the LSP towards the PE router located in their own AS. Since the necessity and explanation of this command was something I needed to read several times, I’d like to try and elaborate a little further on why this is necessary.
The as1-edge has an LDP route towards the as1-pe. This route is placed in the inet.3 table. However, as soon as the as2-edge forwards packets to the as1-pe, the as1-edge router will not use the mpls path information from table inet.3 by default. This is because by default, only BGP can use the inet.3 table recursively.
If the as2-edge sends a labeled packet towards the as1-pe, the as1-edge will ‘pop’ the label and forward the packet on to as1-pe using the OSPF route in the inet.0 table. When building an Interprovider VPN, this is a problem. This is a problem because after that label, there is a VPN label. When the as1-edge pops the top label, it will confront the P router with a VPN label. And that P router will not know what to do with it.
Let’s examine the situation before enabling ‘bgp-igp-both-ribs’:
The as1-egde has the following OSPF route towards the as1-pe:
inetzero@as1-edge> show route 10.1.0.1 table inet.0
inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.1.0.1/32 *[OSPF/10] 00:00:05, metric 2 > to 192.168.200.1 via xe-0/3/0.1505
Let’s verify what the as1-edge router does with labeled traffic by examining the mpls.0 table. First, we need to know what label the as1-edge advertised to the as2-edge for the as1-pe:
inetzero@as1-edge> show route advertising-protocol bgp 10.0.0.2 detail
inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
* 10.1.0.1/32 (1 entry, 1 announced)
BGP group ebgp type External
Route Label: 301312
Nexthop: Self
Flags: Nexthop Change
MED: 2
AS path: [1] I
So the as1-edge told the as2-edge that it can use label 301312 to reach the as1-pe. When we consult the mpls table on the as1-edge, we can find out what the as1-edge will do with this label:
inetzero@as1-edge> show route table mpls.0 label 301312 mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 301312 *[VPN/170] 00:00:17 > to 192.168.200.1 via xe-0/3/0.1505, Pop 301312(S=0) *[VPN/170] 00:00:17 > to 192.168.200.1 via xe-0/3/0.1505, Pop
As stated before, since we want to exchange VPN traffic, this behavior needs to change. We can do this by configuring the router with ‘set protocols mpls traffic-engineering bgp-igp-both-ribs’:
Let’s examine the changes on the router by issuing the same commands:
inetzero@as1-edge> show route 10.1.0.1 table inet.0 inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 10.1.0.1/32 *[LDP/9] 00:02:55, metric 1 > to 192.168.200.1 via xe-0/3/0.1505, Push 299792 [OSPF/10] 00:02:55, metric 2 > to 192.168.200.1 via xe-0/3/0.1505
The ‘bgp-igp-both-ribs’ configuration command copied the route from the inet.3 table into the inet.0 table. So now, the route towards the as1-pe has changed from OSPF into LDP. After using the configuration command ‘set protocols mpls traffic-engineering bgp-igp-both-ribs’, the advertised label changed. So before looking at the MPLS table, we need to check what label the as1-edge router is currently advertising:
inetzero@as1-edge> show route advertising-protocol bgp 10.0.0.2 detail
inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden)
* 10.1.0.1/32 (2 entries, 2 announced)
BGP group ebgp type External
Route Label: 301296
Nexthop: Self
Flags: Nexthop Change
MED: 1
AS path: [1] I
So now, the as1-edge instructed the as2-edge to use label 301296 when it wants to reach the as1-pe. This time, when we consult the mpls table, we should see a ‘swap’ operation:
inetzero@as1-edge> show route table mpls.0 label 301296
mpls.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
301296 *[VPN/170] 00:03:02
> to 192.168.200.1 via xe-0/3/0.1505, Swap 299792
Even though it might look so now, the end-to-end LSP is not complete yet. The PE routers don’t have the information necessary to reach the other PE router. Let’s look at the configuration needed to accomplish this in AS1:
as1-edge:
set protocols bgp group ibgp type internal set protocols bgp group ibgp local-address 192.168.0.1 set protocols bgp group ibgp family inet labeled-unicast set protocols bgp group ibgp export ibgp set policy-options policy-statement ibgp term remote-pe then next-hop self set protocols bgp group ibgp neighbor 10.1.0.1
as1-pe:
set protocols bgp group ibgp type internal
set protocols bgp group ibgp local-address 10.1.0.1
set protocols bgp group ibgp family inet labeled-unicast resolve-vpn
set protocols bgp group ibgp neighbor 192.168.0.1
The ‘family inet labeled-unicast’ is enough to make BGP advertise the necessary labels, but the PE requires an additional knob: ‘resolve-vpn’. This command will allow the PE router to place the route in the inet.3 table. Without this configuration command, the PE routers would not be able to resolve any of the VPN routes we intend to make them exchange.
So, after configuring IBGP for the routers in both ASes, a usable LSP exists between the PE routers:
On to the multihop EBGP exchange of routing information for the L3VPN.
The multihop EBGP exchange of routing information for the L3VPN.
After establishing an LSP between the two PE routers, there are two things left to configure. These are the L3VPN itself and the EBGP multihop session between the PE routers.
Let’s have a look at the L3VPN first. The configuration is kept the exact same on both PE routers:
set interfaces xe-2/0/1 unit 2 description cpe-as1 set interfaces xe-2/0/1 unit 2 vlan-id 2 set interfaces xe-2/0/1 unit 2 family inet mtu 1500 set interfaces xe-2/0/1 unit 2 family inet address 172.16.10.2/30 set interfaces lo0 unit 4000 family inet address 172.16.0.1/32 set policy-options policy-statement to-cpe term bgp from protocol bgp set policy-options policy-statement to-cpe term bgp then accept set policy-options policy-statement to-cpe term direct from protocol direct set policy-options policy-statement to-cpe term direct then accept set routing-instances ipvpn instance-type vrf set routing-instances ipvpn interface xe-2/0/1.2 set routing-instances ipvpn interface lo0.4000 set routing-instances ipvpn route-distinguisher 1:1 set routing-instances ipvpn vrf-target target:1:1 set routing-instances ipvpn vrf-table-label set routing-instances ipvpn protocols ospf export to-cpe set routing-instances ipvpn protocols ospf area 0.0.0.0 interface xe-2/0/1.2 interface-type p2p
OSPF is used to exchange routes between the PE and the CPE. But to make the as1-pe and the as2-pe exchange routes, an EBGP session between the two PE routers is needed:
This configuration could be done in the following way:
as1-pe:
set protocols bgp group pe-as-1 type external set protocols bgp group pe-as-1 multihop ttl 10 set protocols bgp group pe-as-1 local-address 192.168.0.2 set protocols bgp group pe-as-1 family inet-vpn unicast set protocols bgp group pe-as-1 peer-as 1 set protocols bgp group pe-as-1 neighbor 10.1.0.1 description as1-pe
as2-pe:
set protocols bgp group pe-as-2 type external set protocols bgp group pe-as-2 multihop ttl 10 set protocols bgp group pe-as-2 local-address 10.1.0.1 set protocols bgp group pe-as-2 family inet-vpn unicast set protocols bgp group pe-as-2 peer-as 2 set protocols bgp group pe-as-2 neighbor 192.168.0.2 description as2-pe
Other than the multihop configuration statement, this peering session contains nothing really out of the ordinary.
Let’s verify things from the as1-pe by checking the BGP session with the as2-pe router:
inetzero@as1-pe> show bgp summary group pe-as-2
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.l3vpn.0
3 3 0 0 0 0
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.0.2 2 23 23 0 0 8:21 Establ
bgp.l3vpn.0: 3/3/3/0
ipvpn.inet.0: 3/3/3/0
This shows us that the session is up and that there are three active, received and accepted routes. Since there is only one L3VPN active on this PE, the number of routes received in the ‘bgp.l3vpn.0’ table is equal to the number of routes received in the ‘ipvpn.inet.0’ table.
If all is well, the CPEs should be able to send traffic to each other’s loopback IP address:
inetzero@cpe-as1> ping 192.168.250.2 source 192.168.250.1 count 5 rapid
PING 192.168.250.2 (192.168.250.2): 56 data bytes !!!!! --- 192.168.250.2 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.792/0.987/1.706/0.360 ms
And the Interprovider Layer 3 VPN option C is up!
In closing, let’s look at the complete setup once more and examine how the as1-pe was able to forward those packets:
Routers in both AS1 and AS2 are all running OSPF and LDP. This way, inside their own AS, all routers have LDP routes to each other.
The as1-edge and as2-edge routers are exchanging labeled routes for the PE routers located in their own AS. These labeled routes are exchanged by using an EBGP session.
Both as1-edge and as2-edge routers are also running IGBP with the PE routers located inside their own AS. The edge routers use this IBGP session to advertise the labeled route for the PE located in the other AS. This way, both as1-pe and as2-pe learn of a labeled unicast route towards the other routers’ loopback address.
Both as1-pe and as2-pe are engaged in an EBGP session with each other to exchange the routes for the L3VPN.
Let’s examine the forwarding table entry on the as1-pe router and look at the loopback address of cpe-as2:
inetzero@as1-pe> show route forwarding-table vpn ipvpn destination 192.168.250.2
Logical system: as1-pe
Routing table: ipvpn.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
192.168.250.2/32 user 0 indr 1048575 4
192.168.200.6 Push 16, Push 301488, Push 299776(top) 630 2 ae1.4001
This is a forwarding entry with three labels; a VPN label, a label to reach the other PE and a label to reach the as1-edge router. Let’s briefly examine how these labels were learned again and start with the VPN label:
inetzero@as1-pe> show route receive-protocol bgp 192.168.0.2 table ipvpn.inet.0 192.168.250.2/32 detail
ipvpn.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
* 192.168.250.2/32 (1 entry, 1 announced)
Import Accepted
Route Distinguisher: 1:1
VPN Label: 16
Nexthop: 192.168.0.2
MED: 1
AS path: 2 I
Communities: target:1:1 rte-type:0.0.0.0:1:0
The next-hop address is the loopback of the as2-pe. The labeled route towards this PE was learned from the as1-edge router:
inetzero@as1-pe> show route receive-protocol bgp 192.168.0.1 detail inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden) * 192.168.0.2/32 (1 entry, 1 announced) Accepted Route Label: 301488 Nexthop: 192.168.0.1 MED: 1 Localpref: 100 AS path: 2 I inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) * 192.168.0.2/32 (1 entry, 1 announced) Accepted Route Label: 301488 Nexthop: 192.168.0.1 MED: 1 Localpref: 100 AS path: 2 I
Because of the ‘next-hop self’ policy that was applied to the as1-edge router, the next-hop for this route is the as1-edge routers’ own loopback IP address. To be able to reach the as1-edge loopback IP address, the top label that the as1-pe uses corresponds with the route towards the as1-edge loopback IP address. We can see how this label was learned simply by consulting the inet.3 table, asking for the route towards this loopback IP address:
inetzero@as1-pe> show route 192.168.0.1 table inet.3
inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
192.168.0.1/32 *[LDP/9] 00:50:18, metric 1
> to 192.168.200.6 via ae1.4001, Push 299776
After analyzing that, let’s follow the packet up until the as1-edge router by examining what the as1-p router does with this top label. The as1-p router will be confronted with the top label. To find out what that router will do, the easiest thing is to consult the mpls.0 tabel:
inetzero@as1-p> show route table mpls.0 label 299776
mpls.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
299776 *[LDP/9] 00:54:29, metric 1
> to 192.168.200.2 via xe-0/2/0.1505, Pop
299776(S=0) *[LDP/9] 00:54:29, metric 1
> to 192.168.200.2 via xe-0/2/0.1505, Pop
The as1-p router POPs the label. This makes sense, since the next hop is the destination. This leaves the as1-edge with the two remaining labels: 16 and 301488.
inetzero@as1-edge> show route table mpls label 301488
mpls.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
301488 *[VPN/170] 00:56:31
> to 10.0.0.2 via ae1.4000, Swap 301056
The label is swapped to 301056. This label, used to reach the PE in the other AS, was learned from the as2-edge:
inetzero@as1-edge> show route receive-protocol bgp 10.0.0.2 detail
inet.0: 11 destinations, 13 routes (11 active, 0 holddown, 0 hidden)
* 192.168.0.2/32 (1 entry, 1 announced)
Accepted
Route Label: 301056
Nexthop: 10.0.0.2
MED: 1
AS path: 2 I
Since AS2 is sort of a mirror image of the setup in AS1, I’ll stop the walkthrough right here.
Hopefully, I have been able to shed some light on how the Interprovider VPN option C works and how you can configure it.
In closing, I wanted to mention two more things. First of all, I configured a Layer 3 VPN example. The Interprovider VPN option C can also be used for a Layer 2 VPNs. Second, I configured the PE routers to exchange the routes directly with each other because this was easiest. To further enhance the scalability, you could choose to have the multi-hop EBGP session exist between route-reflectors.
More interprovider L3VPN scenario's can be found in iNETZERO's JNCIE-SP lab workbooks
