Demo – BGP troubleshooting
(For this lab, ensure you need to load the initial configuration named “lab19-initial” for all the devices before going through the task lists. The final configuration/solution for this lab (lab19-final) can also be loaded to the devices. For more information about accessing our labs and/or loading the configuration to the devices, please refer to the lab instructions section in the course introduction chapter.)
Detailed Explanation:The above design has four BGP AS numbers where R4 and R3 are multi-homed with AS numbers 12214 and 11113. R11 and R12 only run IGP and they are not BGP-aware devices. Even though these devices are fully configured, there are 5-6 BGP-related faults in this design that need to be diagnosed and sorted out as below.
- R3 should have reachability to both the prefixes originated by R4 via both upstream AS numbers.
- R14 should be able to reach out to R1 originated routes with its loopback0 as a source.
Identifying issues and solutions might look simpler if we just consider the tasks as we have a couple of requirements to correct ongoing issues that exist within this BGP network design. However, this can be difficult if the right approach to troubleshooting is not applied in such a design with a number of devices in operation. We could have checked the running-config of all the devices and quickly figure out the issues but this might take way longer than checking for the appropriate outputs by using BGP-specific verification commands. The very first thing is to check on the neighbor status and look into the BGP table for the received and advertised routes status. We can start from the R4 all the way down to the R3 since we need to check reachability from R3.
R4#sh ip bgp summary
BGP router identifier 44.44.44.44, local AS number 4
BGP table version is 54, main routing table version 54
5 network entries using 1240 bytes of memory
5 path entries using 680 bytes of memory
3/3 BGP path/bestpath attribute entries using 840 bytes of memory
2 BGP AS-PATH entries using 84 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2844 total bytes of memory
BGP activity 21/16 prefixes, 37/32 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.14.1 4 11113 2497 2499 54 0 0 1d13h 0
10.1.24.2 4 12214 2476 2487 54 0 0 1d13h 3
R4#show ip bgp neighbors 10.1.24.2 advertised-routes
BGP table version is 54, local router ID is 44.44.44.44
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 4.4.4.4/32 0.0.0.0 0 32768 i
*> 44.44.44.44/32 0.0.0.0 0 32768 i
Total number of prefixes 2
R4#sh ip bgp neighbors 10.1.14.1 advertised-routes
BGP table version is 54, local router ID is 44.44.44.44
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.3/32 10.1.24.2 0 12214 3 3 3 3 3 i
*> 4.4.4.4/32 0.0.0.0 0 32768 i
*> 14.14.14.14/32 10.1.24.2 0 12214 i
*> 33.33.33.33/32 10.1.24.2 0 12214 3 3 3 3 3 i
*> 44.44.44.44/32 0.0.0.0 0 32768 i
Total number of prefixes 5
R1#sh ip bgp 4.4.4.4
BGP routing table entry for 4.4.4.4/32, version 24
Paths: (2 available, best #2, table default)
Advertised to update-groups:
11
Refresh Epoch 1
3 12214 4
13.13.13.13 (metric 12) from 13.13.13.13 (13.13.13.13)
Origin IGP, localpref 100, valid, internal
rx pathid: 0, tx pathid: 0
Refresh Epoch 9
4 3
10.1.14.4 from 10.1.14.4 (44.44.44.44)
Origin IGP, metric 0, localpref 100, valid, external, best
rx pathid: 0, tx pathid: 0x0
R1#show ip bgp 44.44.44.44
BGP routing table entry for 44.44.44.44/32, version 25
Paths: (2 available, best #2, table default)
Advertised to update-groups:
11
Refresh Epoch 1
3 12214 4
13.13.13.13 (metric 12) from 13.13.13.13 (13.13.13.13)
Origin IGP, localpref 100, valid, internal
rx pathid: 0, tx pathid: 0
Refresh Epoch 9
4 3
10.1.14.4 from 10.1.14.4 (44.44.44.44)
Origin IGP, metric 0, localpref 100, valid, external, best
rx pathid: 0, tx pathid: 0x0
One thing to note here is that R1 is receiving AS path information of 3 within AS path attribute field. It is obvious that the R3 won’t be able to accept this update since it will find its own AS in the BGP update. We will check it later during this troubleshooting process.
R2#sh ip bgp 4.4.4.4
BGP routing table entry for 4.4.4.4/32, version 59
Paths: (1 available, best #1, table default)
Advertised to update-groups:
6
Refresh Epoch 4
4
10.1.24.4 from 10.1.24.4 (44.44.44.44)
Origin IGP, metric 0, localpref 100, valid, external, best
rx pathid: 0, tx pathid: 0x0
R2#sh ip bgp 44.44.44.44
BGP routing table entry for 44.44.44.44/32, version 58
Paths: (1 available, best #1, table default, not advertised to any peer)
Not advertised to any peer
Refresh Epoch 4
4
10.1.24.4 from 10.1.24.4 (44.44.44.44)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: no-advertise
rx pathid: 0, tx pathid: 0x0
The first prefix received from R4 has regular BGP attributes but the second prefix has been tagged with a well-known community value “no-advertise” which indicates that R14 shouldn’t have this prefix in its BGP table since R2 does only advertise 4.4.4.4/32 prefix to its iBGP peer.
R2#sh ip bgp neighbors 14.14.14.14 advertised-routes
BGP table version is 59, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 4.4.4.4/32 10.1.24.4 0 0 4 i
Total number of prefixes 1
Note: We can use the override option when we are working on the limitation since the attribute might be coming from the remote AS and we do not have control over it.
R2(config)#access-list 4 permit 44.44.44.44
R2(config)#route-map CL_COM permit 10
R2(config-route-map)# match ip address 4
R2(config-route-map)# set community internet
R2(config-route-map)#exit
R2(config)#route-map CL_COM permit 20
R2(config-route-map)#end
R2#sh ip bgp 44.44.44.44
BGP routing table entry for 44.44.44.44/32, version 6
Paths: (1 available, best #1, table default)
Advertised to update-groups:
10
Refresh Epoch 4
4
10.1.24.4 from 10.1.24.4 (44.44.44.44)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: internet
rx pathid: 0, tx pathid: 0x0
R2#sh ip bgp neighbors 14.14.14.14 advertised-routes
BGP table version is 24, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.3/32 10.1.24.4 0 4 11113 3 i
*> 4.4.4.4/32 10.1.24.4 0 0 4 i
*> 33.33.33.33/32 10.1.24.4 0 4 11113 3 i
*> 44.44.44.44/32 10.1.24.4 0 0 4 i
*> 101.101.101.101/32
10.1.24.4 0 4 11113 i
Total number of prefixes 5
Up to this point, we have concluded that the route advertisement till the transit AS numbers are fine. Now, let’s verify if the border routers R13 and R14 are advertising these prefixes to R3.
RP/0/RP0/CPU0:R13#sh bgp ipv4 all neighbors 10.1.133.3 advertised-routes
Wed Jul 28 09:38:06.340 UTC
Address Family: IPv4 Unicast
----------------------------
Network Next Hop From AS Path
101.101.101.101/32 10.1.133.13 1.1.1.1 11113i
Processed 1 prefixes, 1 paths
No advertisement from R13 to AS3 for AS4 prefixes, let’s observe the BGP table and check for the received prefixes in detail.
RP/0/RP0/CPU0:R13#sh bgp ipv4 unicast neighbors 10.1.133.3 advertised-routes
Wed Jul 28 10:16:54.327 UTC
Network Next Hop From AS Path
101.101.101.101/32 10.1.133.13 1.1.1.1 11113i
Processed 1 prefixes, 1 paths
RP/0/RP0/CPU0:R13#sh ip bgp
Wed Jul 28 10:17:00.202 UTC
BGP router identifier 13.13.13.13, local AS number 11113
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000 RD version: 213
BGP main routing table version 213
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.3/32 10.1.133.3 0 0 3 i
* i4.4.4.4/32 10.1.14.4 0 100 0 4 3 i
*> 10.1.133.3 0 3 12214 4 i
*> 14.14.14.14/32 10.1.133.3 0 3 12214 i
*> 33.33.33.33/32 10.1.133.3 0 0 3 i
* i44.44.44.44/32 10.1.14.4 0 100 0 4 3 i
*> 10.1.133.3 0 3 12214 4 i
*>i101.101.101.101/32 1.1.1.1 0 100 0 i
Processed 6 prefixes, 8 paths
Even though routing updates received from BGP peer for AS 4 prefixes are installed in the BGP table with the best AS path information but not marked as the best route. Let’s look into the prefix detail.
RP/0/RP0/CPU0:R13#sh bgp ipv4 all 4.4.4.4
Wed Jul 28 10:37:45.098 UTC
Address Family: IPv4 Unicast
----------------------------
BGP routing table entry for 4.4.4.4/32
Versions:
Process bRIB/RIB SendTblVer
Speaker 212 212
Last Modified: Jul 28 09:51:53.990 for 00:45:51
Paths: (2 available, best #2)
Advertised IPv4 Unicast paths to peers (in unique update groups):
1.1.1.1
Path #1: Received by speaker 0
Not advertised to any peer
4 3
10.1.14.4 (inaccessible) from 1.1.1.1 (101.101.101.101)
Origin IGP, metric 0, localpref 100, valid, internal
Received Path ID 0, Local Path ID 0, version 0
Path #2: Received by speaker 0
Advertised IPv4 Unicast paths to peers (in unique update groups):
1.1.1.1
3 12214 4
10.1.133.3 from 10.1.133.3 (33.33.33.33)
Origin IGP, localpref 100, valid, external, best, group-best
Received Path ID 0, Local Path ID 1, version 212
Origin-AS validity: (disabled)
When we talk about next-hop in BGP, the route learned from eBGP peer is advertised to iBGP peer with original next-hop information, i.e. remote end eBGP peer address. In order to solve these inaccessible next-hop issues, we have a non-BGP solution where we can make the point-to-point connectivity accessible by redistributing into running IGP within local AS. Also, we can advertise the network into IGP as well but the peering AS policy might not prefer this solution. So, the easiest way is to change the next-hop with the accessible address.
Problem: R1 is advertising inaccessible next-hop to R13. Solution: Change the next-hop information and set it as a loopback address of R1.R1(config)#router bgp 11113
R1(config-router)#neighbor 13.13.13.13 next-hop-self
R1(config-router)#end
RP/0/RP0/CPU0:R13#clear ip bgp * sof in
RP/0/RP0/CPU0:R13#sh ip bgp
Wed Jul 28 10:46:50.887 UTC
BGP router identifier 13.13.13.13, local AS number 11113
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000 RD version: 215
BGP main routing table version 215
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.3/32 10.1.133.3 0 0 3 i
*>i4.4.4.4/32 1.1.1.1 0 100 0 4 3 i
* 10.1.133.3 0 3 12214 4 i
*> 14.14.14.14/32 10.1.133.3 0 3 12214 i
*> 33.33.33.33/32 10.1.133.3 0 0 3 i
*>i44.44.44.44/32 1.1.1.1 0 100 0 4 3 i
* 10.1.133.3 0 3 12214 4 i
*>i101.101.101.101/32 1.1.1.1 0 100 0 i
Processed 6 prefixes, 8 paths
Now the next-hop has changed with R1’s loopback address which is accessible via IGP, the BGP path has now changed for AS4 originated prefixes. Now, check for the advertised route to R3.
RP/0/RP0/CPU0:R13#sh bgp ipv4 unicast neighbors 10.1.133.3 advertised-routes
Wed Jul 28 10:50:54.042 UTC
Network Next Hop From AS Path
4.4.4.4/32 10.1.133.13 1.1.1.1 11113 4 3i
44.44.44.44/32 10.1.133.13 1.1.1.1 11113 4 3i
101.101.101.101/32 10.1.133.13 1.1.1.1 11113i
Processed 3 prefixes, 3 paths
RP/0/RP0/CPU0:R14#sh bgp ipv4 all neighbors 10.1.143.3 advertised-routes
Wed Jul 28 10:52:31.731 UTC
Address Family: IPv4 Unicast
----------------------------
Network Next Hop From AS Path
3.3.3.3/32 10.1.143.14 2.2.2.2 12214 4 11113 3i
4.4.4.4/32 10.1.143.14 2.2.2.2 12214 4i
14.14.14.14/32 10.1.143.14 Local 12214i
33.33.33.33/32 10.1.143.14 2.2.2.2 12214 4 11113 3i
44.44.44.44/32 10.1.143.14 2.2.2.2 12214 4i
101.101.101.101/32 10.1.143.14 2.2.2.2 12214 4 11113i
Processed 6 prefixes, 6 paths
The border routers are advertising AS4 prefixes down to the R3, let’s see how R3 calculates the path to reach the destinations.
R3#sh ip bgp
BGP table version is 80, local router ID is 33.33.33.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.3/32 0.0.0.0 0 32768 i
*> 4.4.4.4/32 10.1.143.14 0 12214 4 i
*> 14.14.14.14/32 10.1.143.14 0 0 12214 i
*> 33.33.33.33/32 0.0.0.0 0 32768 i
*> 44.44.44.44/32 10.1.143.14 0 12214 4 i
* 101.101.101.101/32
10.1.143.14 0 12214 4 11113 i
*> 10.1.133.13 0 11113 i
BGP update from R13 is not shown on the BGP table which is probably caused by the AS path information prepended by R4. Let’s enable debugging for BGP updates in the inbound direction and clear BGP updates.
R3#debug ip bgp updates in
R3#clear ip bgp * soft in
*Jul 28 11:20:59.746: BGP(0): 10.1.133.13 rcv UPDATE about 44.44.44.44/32 -- DENIED due to: AS-PATH contains our own AS;
*Jul 28 11:20:59.746: BGP(0): 10.1.133.13 rcv UPDATE about 4.4.4.4/32 -- DENIED due to: AS-PATH contains our own AS;
R3(config-router)#neighbor 10.1.133.13 allowas-in 1
Since the occurrence of AS number is 1, we have set it as 1. This value can be used based on the number of the same AS path information in the update.
R3(config-router)#do sh ip bgp
BGP table version is 80, local router ID is 33.33.33.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.3/32 0.0.0.0 0 32768 i
* 4.4.4.4/32 10.1.133.13 0 11113 4 3 i
*> 10.1.143.14 0 12214 4 i
*> 14.14.14.14/32 10.1.143.14 0 0 12214 i
*> 33.33.33.33/32 0.0.0.0 0 32768 i
* 44.44.44.44/32 10.1.133.13 0 11113 4 3 i
*> 10.1.143.14 0 12214 4 i
* 101.101.101.101/32
10.1.143.14 0 12214 4 11113 i
*> 10.1.133.13 0 11113 i
R3(config-router)#no neighbor 10.1.133.13 allowas-in 1
R4(config)#route-map To-AS11113 permit 10
R4(config-route-map)#no set as-path prepend 3
R3#clear ip bgp * soft in
R3#sh ip bgp
BGP table version is 80, local router ID is 33.33.33.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 3.3.3.3/32 0.0.0.0 0 32768 i
* 4.4.4.4/32 10.1.133.13 0 11113 4 i
*> 10.1.143.14 0 12214 4 i
*> 14.14.14.14/32 10.1.143.14 0 0 12214 i
*> 33.33.33.33/32 0.0.0.0 0 32768 i
* 44.44.44.44/32 10.1.133.13 0 11113 4 i
*> 10.1.143.14 0 12214 4 i
* 101.101.101.101/32
10.1.143.14 0 12214 4 11113 i
*> 10.1.133.13 0 11113 i
R3#sh ip route bgp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is not set
4.0.0.0/32 is subnetted, 1 subnets
B 4.4.4.4 [20/0] via 10.1.143.14, 01:43:05
14.0.0.0/32 is subnetted, 1 subnets
B 14.14.14.14 [20/0] via 10.1.143.14, 17:40:30
44.0.0.0/32 is subnetted, 1 subnets
B 44.44.44.44 [20/0] via 10.1.143.14, 01:43:05
101.0.0.0/32 is subnetted, 1 subnets
B 101.101.101.101 [20/0] via 10.1.133.13, 02:25:06
R3#traceroute 4.4.4.4 source l0
Type escape sequence to abort.
Tracing the route to 4.4.4.4
VRF info: (vrf in name/id, vrf out name/id)
1 * * *
2 * * *
R3(config)#inter gi4
R3(config-if)#shut
R3#traceroute 4.4.4.4 source l0
Type escape sequence to abort.
Tracing the route to 4.4.4.4
VRF info: (vrf in name/id, vrf out name/id)
1 10.1.133.13 19 msec 12 msec 7 msec
2 10.1.131.11 7 msec 19 msec 16 msec
3 10.1.111.1 2 msec 1 msec 2 msec
4 10.1.14.4 2 msec * 2 msec
R3#traceroute 44.44.44.44 source l0
Type escape sequence to abort.
Tracing the route to 44.44.44.44
VRF info: (vrf in name/id, vrf out name/id)
1 10.1.133.13 10 msec 1 msec 7 msec
2 10.1.131.11 11 msec 10 msec 9 msec
3 10.1.111.1 1 msec 1 msec 1 msec
4 10.1.14.4 1 msec * 2 msec
So the routing path via AS 12214 is not working correctly.
DEMO END ….