Free sample content / demo version
Course Introduction
Interior Gateway Protocol
Border Gateway Protocol
MPLS LDP
MPLS L3VPN
MPLS Traffic Engineering
Segment Routing
MPLS L2VPN
MPLS QoS
Security and Services
Automation and Assurance
Final Lab1
Final Lab 2

Demo – BGP troubleshooting

Diagram:

(For this lab, ensure you need to load the initial configuration named “lab19-initial” for all the devices before going through the task lists. The final configuration/solution for this lab (lab19-final) can also be loaded to the devices. For more information about accessing our labs and/or loading the configuration to the devices, please refer to the lab instructions section in the course introduction chapter.)

Detailed Explanation:

The above design has four BGP AS numbers where R4 and R3 are multi-homed with AS numbers 12214 and 11113. R11 and R12 only run IGP and they are not BGP-aware devices. Even though these devices are fully configured, there are 5-6 BGP-related faults in this design that need to be diagnosed and sorted out as below.

  • R3 should have reachability to both the prefixes originated by R4 via both upstream AS numbers.
  • R14 should be able to reach out to R1 originated routes with its loopback0 as a source.

Identifying issues and solutions might look simpler if we just consider the tasks as we have a couple of requirements to correct ongoing issues that exist within this  BGP network design. However, this can be difficult if the right approach to troubleshooting is not applied in such a design with a number of devices in operation. We could have checked the running-config of all the devices and quickly figure out the issues but this might take way longer than checking for the appropriate outputs by using BGP-specific verification commands. The very first thing is to check on the neighbor status and look into the BGP table for the received and advertised routes status. We can start from the R4 all the way down to the R3 since we need to check reachability from R3.

R4#sh ip bgp summary 
BGP router identifier 44.44.44.44, local AS number 4
BGP table version is 54, main routing table version 54
5 network entries using 1240 bytes of memory
5 path entries using 680 bytes of memory
3/3 BGP path/bestpath attribute entries using 840 bytes of memory
2 BGP AS-PATH entries using 84 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2844 total bytes of memory
BGP activity 21/16 prefixes, 37/32 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.14.1       4        11113    2497    2499       54    0    0 1d13h           0
10.1.24.2       4        12214    2476    2487       54    0    0 1d13h           3

R4#show ip bgp neighbors 10.1.24.2 advertised-routes 
BGP table version is 54, local router ID is 44.44.44.44
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   4.4.4.4/32       0.0.0.0                  0         32768 i
 *>   44.44.44.44/32   0.0.0.0                  0         32768 i

Total number of prefixes 2 

R4#sh ip bgp neighbors 10.1.14.1 advertised-routes 
BGP table version is 54, local router ID is 44.44.44.44
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   3.3.3.3/32       10.1.24.2                              0 12214 3 3 3 3 3 i
 *>   4.4.4.4/32       0.0.0.0                  0         32768 i
 *>   14.14.14.14/32   10.1.24.2                              0 12214 i
 *>   33.33.33.33/32   10.1.24.2                              0 12214 3 3 3 3 3 i
 *>   44.44.44.44/32   0.0.0.0                  0         32768 i

Total number of prefixes 5 

At this point we can assume that the neighbor status and BGP table look fine since we can see the originating route followed by transiting routes in the BGP table. Let’s go to the downstream AS devices and check similar things.
R1#sh ip bgp 4.4.4.4
BGP routing table entry for 4.4.4.4/32, version 24
Paths: (2 available, best #2, table default)
  Advertised to update-groups:
     11        
  Refresh Epoch 1
  3 12214 4
    13.13.13.13 (metric 12) from 13.13.13.13 (13.13.13.13)
      Origin IGP, localpref 100, valid, internal
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 9
  4 3
    10.1.14.4 from 10.1.14.4 (44.44.44.44)
      Origin IGP, metric 0, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0

R1#show ip bgp 44.44.44.44
BGP routing table entry for 44.44.44.44/32, version 25
Paths: (2 available, best #2, table default)
  Advertised to update-groups:
     11        
  Refresh Epoch 1
  3 12214 4
    13.13.13.13 (metric 12) from 13.13.13.13 (13.13.13.13)
      Origin IGP, localpref 100, valid, internal
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 9
  4 3
    10.1.14.4 from 10.1.14.4 (44.44.44.44)
      Origin IGP, metric 0, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0
For both the prefixes originated by R4, R1 is choosing best path via direct eBGP session where it received BGP update with shortest AS path information and AS-path attribute is taken into consideration for best path selection process.

One thing to note here is that R1 is receiving AS path information of 3 within AS path attribute field. It is obvious that the R3 won’t be able to accept this update since it will find its own AS in the BGP update. We will check it later during this troubleshooting process.

R2#sh ip bgp 4.4.4.4
BGP routing table entry for 4.4.4.4/32, version 59
Paths: (1 available, best #1, table default)
  Advertised to update-groups:
     6         
  Refresh Epoch 4
  4
    10.1.24.4 from 10.1.24.4 (44.44.44.44)
      Origin IGP, metric 0, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0

R2#sh ip bgp 44.44.44.44
BGP routing table entry for 44.44.44.44/32, version 58
Paths: (1 available, best #1, table default, not advertised to any peer)
  Not advertised to any peer
  Refresh Epoch 4
  4
    10.1.24.4 from 10.1.24.4 (44.44.44.44)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Community: no-advertise
      rx pathid: 0, tx pathid: 0x0

The first prefix received from R4 has regular BGP attributes but the second prefix has been tagged with a well-known community value “no-advertise” which indicates that R14 shouldn’t have this prefix in its BGP table since R2 does only advertise 4.4.4.4/32 prefix to its iBGP peer.

R2#sh ip bgp neighbors 14.14.14.14 advertised-routes 
BGP table version is 59, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   4.4.4.4/32       10.1.24.4                0             0 4 i

Total number of prefixes 1 
Problem: BGP community ‘no-advertise’ has been tagged with the BGP update for 44.44.44.44/32 Solution: Remove community value from R4 BGP advertisement or override it with ‘internet’ well-known community on R2 itself.

Note: We can use the override option when we are working on the limitation since the attribute might be coming from the remote AS and we do not have control over it.

R2(config)#access-list 4 permit 44.44.44.44
R2(config)#route-map CL_COM permit 10             
R2(config-route-map)# match ip address 4
R2(config-route-map)# set community internet
R2(config-route-map)#exit
R2(config)#route-map CL_COM permit 20 
R2(config-route-map)#end

R2#sh ip bgp 44.44.44.44
BGP routing table entry for 44.44.44.44/32, version 6
Paths: (1 available, best #1, table default)
  Advertised to update-groups:
     10        
  Refresh Epoch 4
  4
    10.1.24.4 from 10.1.24.4 (44.44.44.44)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Community: internet
      rx pathid: 0, tx pathid: 0x0

R2#sh ip bgp neighbors 14.14.14.14 advertised-routes 
BGP table version is 24, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   3.3.3.3/32       10.1.24.4                              0 4 11113 3 i
 *>   4.4.4.4/32       10.1.24.4                0             0 4 i
 *>   33.33.33.33/32   10.1.24.4                              0 4 11113 3 i
 *>   44.44.44.44/32   10.1.24.4                0             0 4 i
 *>   101.101.101.101/32
                      10.1.24.4                              0 4 11113 i

Total number of prefixes 5 

Up to this point, we have concluded that the route advertisement till the transit AS numbers are fine. Now, let’s verify if the border routers R13 and R14 are advertising these prefixes to R3.

RP/0/RP0/CPU0:R13#sh bgp ipv4 all neighbors 10.1.133.3 advertised-routes 
Wed Jul 28 09:38:06.340 UTC

Address Family: IPv4 Unicast
----------------------------

Network            Next Hop        From            AS Path
101.101.101.101/32 10.1.133.13     1.1.1.1         11113i

Processed 1 prefixes, 1 paths

No advertisement from R13 to AS3 for AS4 prefixes, let’s observe the BGP table and check for the received prefixes in detail.

RP/0/RP0/CPU0:R13#sh bgp ipv4 unicast neighbors 10.1.133.3 advertised-routes 
Wed Jul 28 10:16:54.327 UTC
Network            Next Hop        From            AS Path
101.101.101.101/32 10.1.133.13     1.1.1.1         11113i

Processed 1 prefixes, 1 paths
RP/0/RP0/CPU0:R13#sh ip bgp 
Wed Jul 28 10:17:00.202 UTC
BGP router identifier 13.13.13.13, local AS number 11113
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 213
BGP main routing table version 213
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 3.3.3.3/32         10.1.133.3               0             0 3 i
* i4.4.4.4/32         10.1.14.4                0    100      0 4 3 i
*>                    10.1.133.3                             0 3 12214 4 i
*> 14.14.14.14/32     10.1.133.3                             0 3 12214 i
*> 33.33.33.33/32     10.1.133.3               0             0 3 i
* i44.44.44.44/32     10.1.14.4                0    100      0 4 3 i
*>                    10.1.133.3                             0 3 12214 4 i
*>i101.101.101.101/32 1.1.1.1                  0    100      0 i

Processed 6 prefixes, 8 paths

Even though routing updates received from BGP peer for AS 4 prefixes are installed in the BGP table with the best AS path information but not marked as the best route. Let’s look into the prefix detail.

RP/0/RP0/CPU0:R13#sh bgp ipv4 all 4.4.4.4
Wed Jul 28 10:37:45.098 UTC

Address Family: IPv4 Unicast
----------------------------

BGP routing table entry for 4.4.4.4/32
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker                212         212
Last Modified: Jul 28 09:51:53.990 for 00:45:51
Paths: (2 available, best #2)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    1.1.1.1         
  Path #1: Received by speaker 0
  Not advertised to any peer
  4 3
    10.1.14.4 (inaccessible) from 1.1.1.1 (101.101.101.101)
      Origin IGP, metric 0, localpref 100, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
  Path #2: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    1.1.1.1         
  3 12214 4
    10.1.133.3 from 10.1.133.3 (33.33.33.33)
      Origin IGP, localpref 100, valid, external, best, group-best
      Received Path ID 0, Local Path ID 1, version 212
      Origin-AS validity: (disabled)
The next-hop information for AS 4 prefix given by R1 is inaccessible. It indicates that the R1 is missing the next-hop-self keyword for neighbor 13.13.13.13.

When we talk about next-hop in BGP, the route learned from eBGP peer is advertised to iBGP peer with original next-hop information, i.e. remote end eBGP peer address. In order to solve these inaccessible next-hop issues, we have a non-BGP solution where we can make the point-to-point connectivity accessible by redistributing into running IGP within local AS. Also, we can advertise the network into IGP as well but the peering AS policy might not prefer this solution. So, the easiest way is to change the next-hop with the accessible address.

Problem: R1 is advertising inaccessible next-hop to R13. Solution: Change the next-hop information and set it as a loopback address of R1.
R1(config)#router bgp 11113
R1(config-router)#neighbor 13.13.13.13 next-hop-self
R1(config-router)#end

RP/0/RP0/CPU0:R13#clear ip bgp * sof in 

RP/0/RP0/CPU0:R13#sh ip bgp 
Wed Jul 28 10:46:50.887 UTC
BGP router identifier 13.13.13.13, local AS number 11113
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 215
BGP main routing table version 215
BGP NSR Initial initsync version 2 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 3.3.3.3/32         10.1.133.3               0             0 3 i
*>i4.4.4.4/32         1.1.1.1                  0    100      0 4 3 i
*                     10.1.133.3                             0 3 12214 4 i
*> 14.14.14.14/32     10.1.133.3                             0 3 12214 i
*> 33.33.33.33/32     10.1.133.3               0             0 3 i
*>i44.44.44.44/32     1.1.1.1                  0    100      0 4 3 i
*                     10.1.133.3                             0 3 12214 4 i
*>i101.101.101.101/32 1.1.1.1                  0    100      0 i

Processed 6 prefixes, 8 paths

Now the next-hop has changed with R1’s loopback address which is accessible via IGP, the BGP path has now changed for AS4 originated prefixes. Now, check for the advertised route to R3.

RP/0/RP0/CPU0:R13#sh bgp ipv4 unicast neighbors 10.1.133.3 advertised-routes 
Wed Jul 28 10:50:54.042 UTC
Network            Next Hop        From            AS Path
4.4.4.4/32         10.1.133.13     1.1.1.1         11113 4 3i
44.44.44.44/32     10.1.133.13     1.1.1.1         11113 4 3i
101.101.101.101/32 10.1.133.13     1.1.1.1         11113i

Processed 3 prefixes, 3 paths

RP/0/RP0/CPU0:R14#sh bgp ipv4 all neighbors 10.1.143.3 advertised-routes 
Wed Jul 28 10:52:31.731 UTC

Address Family: IPv4 Unicast
----------------------------

Network            Next Hop        From            AS Path
3.3.3.3/32         10.1.143.14     2.2.2.2         12214 4 11113 3i
4.4.4.4/32         10.1.143.14     2.2.2.2         12214 4i
14.14.14.14/32     10.1.143.14     Local           12214i
33.33.33.33/32     10.1.143.14     2.2.2.2         12214 4 11113 3i
44.44.44.44/32     10.1.143.14     2.2.2.2         12214 4i
101.101.101.101/32 10.1.143.14     2.2.2.2         12214 4 11113i

Processed 6 prefixes, 6 paths

The border routers are advertising AS4 prefixes down to the R3, let’s see how R3 calculates the path to reach the destinations.

R3#sh ip bgp 
BGP table version is 80, local router ID is 33.33.33.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   3.3.3.3/32       0.0.0.0                  0         32768 i
 *>   4.4.4.4/32       10.1.143.14                            0 12214 4 i
 *>   14.14.14.14/32   10.1.143.14              0             0 12214 i
 *>   33.33.33.33/32   0.0.0.0                  0         32768 i
 *>   44.44.44.44/32   10.1.143.14                            0 12214 4 i
 *    101.101.101.101/32
                      10.1.143.14                            0 12214 4 11113 i
 *>                    10.1.133.13                            0 11113 i

BGP update from R13 is not shown on the BGP table which is probably caused by the AS path information prepended by R4. Let’s enable debugging for BGP updates in the inbound direction and clear BGP updates.

R3#debug ip bgp updates in 

R3#clear ip bgp * soft in

*Jul 28 11:20:59.746: BGP(0): 10.1.133.13 rcv UPDATE about 44.44.44.44/32 -- DENIED due to: AS-PATH contains our own AS;
*Jul 28 11:20:59.746: BGP(0): 10.1.133.13 rcv UPDATE about 4.4.4.4/32 -- DENIED due to: AS-PATH contains our own AS;
Problem: Wrong AS path information has been received. Solution: We can remove the prepended AS information on R4 or tell R3 NOT to reject prefixes with looped AS path information using the “allowas-in” command under the BGP process. However, the ‘allowas-in” command might be risky where there are multi-homed scenarios. We can utilize this feature in two VPN single-homed site that is interconnected via MPLS L3VPN. However, as part of the test process, we can check the usage of this command.
R3(config-router)#neighbor 10.1.133.13 allowas-in 1

Since the occurrence of AS number is 1, we have set it as 1. This value can be used based on the number of the same AS path information in the update.

R3(config-router)#do sh ip bgp 
BGP table version is 80, local router ID is 33.33.33.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   3.3.3.3/32       0.0.0.0                  0         32768 i
 *    4.4.4.4/32       10.1.133.13                            0 11113 4 3 i
 *>                    10.1.143.14                            0 12214 4 i
 *>   14.14.14.14/32   10.1.143.14              0             0 12214 i
 *>   33.33.33.33/32   0.0.0.0                  0         32768 i
 *    44.44.44.44/32   10.1.133.13                            0 11113 4 3 i
 *>                    10.1.143.14                            0 12214 4 i
 *    101.101.101.101/32
                      10.1.143.14                            0 12214 4 11113 i
 *>                    10.1.133.13                            0 11113 i
Let’s remove this feature from R3 and delete the wrong AS path information from the R4 advertisement.
R3(config-router)#no neighbor 10.1.133.13 allowas-in 1

R4(config)#route-map To-AS11113 permit 10 
R4(config-route-map)#no  set as-path prepend 3

R3#clear ip bgp * soft in 

R3#sh ip bgp 
BGP table version is 80, local router ID is 33.33.33.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   3.3.3.3/32       0.0.0.0                  0         32768 i
 *    4.4.4.4/32       10.1.133.13                            0 11113 4 i
 *>                    10.1.143.14                            0 12214 4 i
 *>   14.14.14.14/32   10.1.143.14              0             0 12214 i
 *>   33.33.33.33/32   0.0.0.0                  0         32768 i
 *    44.44.44.44/32   10.1.133.13                            0 11113 4 i
 *>                    10.1.143.14                            0 12214 4 i
 *    101.101.101.101/32
                      10.1.143.14                            0 12214 4 11113 i
 *>                    10.1.133.13                            0 11113 i

R3#sh ip route bgp 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 10.1.143.14, 01:43:05
      14.0.0.0/32 is subnetted, 1 subnets
B        14.14.14.14 [20/0] via 10.1.143.14, 17:40:30
      44.0.0.0/32 is subnetted, 1 subnets
B        44.44.44.44 [20/0] via 10.1.143.14, 01:43:05
      101.0.0.0/32 is subnetted, 1 subnets
B        101.101.101.101 [20/0] via 10.1.133.13, 02:25:06
Let’s check routing path with trace route for both the paths.
R3#traceroute 4.4.4.4 source l0
Type escape sequence to abort.
Tracing the route to 4.4.4.4
VRF info: (vrf in name/id, vrf out name/id)
  1  *  *  * 
  2  *  *  * 
With the current best path the trace is not successful, let’s check the alternate path by disabling the Gi4 interface on R3.
R3(config)#inter gi4
R3(config-if)#shut


R3#traceroute 4.4.4.4 source l0
Type escape sequence to abort.
Tracing the route to 4.4.4.4
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.133.13 19 msec 12 msec 7 msec
  2 10.1.131.11 7 msec 19 msec 16 msec
  3 10.1.111.1 2 msec 1 msec 2 msec
  4 10.1.14.4 2 msec *  2 msec

R3#traceroute 44.44.44.44 source l0
Type escape sequence to abort.
Tracing the route to 44.44.44.44
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.133.13 10 msec 1 msec 7 msec
  2 10.1.131.11 11 msec 10 msec 9 msec
  3 10.1.111.1 1 msec 1 msec 1 msec
  4 10.1.14.4 1 msec *  2 msec

So the routing path via AS 12214 is not working correctly.

DEMO END ….