cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4634
Views
40
Helpful
1
Comments
Vinit Jain
Cisco Employee
Cisco Employee

MPLS VPN Services are one of the most common services provided by Service Providers. Because of its wide and huge deployments, there is a need to have deep troubleshooting knowledge to fix the issues that occur in MPLS VPN deployment.

Today, we shall see how to troubleshoot Basic Layer-3 MPLS VPN Services. For this, lets consider the below topology:

 

Configuration

Config on PE1:
==============
ip vrf ABC
 rd 1:1
 route-target export 1:1
 route-target import 1:1
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface Ethernet0/0
 ip vrf forwarding ABC
 ip address 192.168.10.1 255.255.255.252
! 
interface Ethernet1/0
 ip address 12.12.12.1 255.255.255.252
 ip ospf network point-to-point
 mpls ip
!
router ospf 100
 router-id 1.1.1.1
 network 1.1.1.1 0.0.0.0 area 0
 network 12.12.12.1 0.0.0.0 area 0
!
router bgp 100
 bgp router-id 1.1.1.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 3.3.3.3 remote-as 100
 neighbor 3.3.3.3 update-source Loopback0
 !        
 address-family ipv4
 exit-address-family
 !        
 address-family vpnv4
  neighbor 3.3.3.3 activate
  neighbor 3.3.3.3 send-community both
 exit-address-family
 !        
 address-family ipv4 vrf ABC
  redistribute connected
  neighbor 192.168.10.2 remote-as 65535
  neighbor 192.168.10.2 activate
 exit-address-family
!

Config on P-1:
==============
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface Ethernet0/0
 ip address 23.23.23.1 255.255.255.252
 ip ospf network point-to-point
 mpls ip
!
interface Ethernet1/0
 ip address 12.12.12.2 255.255.255.252
 ip ospf network point-to-point
 mpls ip
!
router ospf 100
 router-id 2.2.2.2
 network 2.2.2.2 0.0.0.0 area 0
 network 12.12.12.2 0.0.0.0 area 0
 network 23.23.23.1 0.0.0.0 area 0
! 

Config PE2:
===========
ip vrf ABC
 rd 100:1
 route-target export 1:1
 route-target import 1:1
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
!
interface Ethernet0/0
 ip address 23.23.23.2 255.255.255.252
 ip ospf network point-to-point
 mpls ip
!
interface Ethernet1/0
 ip vrf forwarding ABC
 ip address 192.168.20.1 255.255.255.252
!
router ospf 100
 router-id 3.3.3.3
 network 3.3.3.3 0.0.0.0 area 0
 network 23.23.23.2 0.0.0.0 area 0
!
router bgp 100
 bgp router-id 3.3.3.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 remote-as 100
 neighbor 1.1.1.1 update-source Loopback0
 !        
 address-family ipv4
 exit-address-family
 !        
 address-family vpnv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 send-community both
 exit-address-family
 !        
 address-family ipv4 vrf ABC
  redistribute connected
  neighbor 192.168.20.2 remote-as 65000
  neighbor 192.168.20.2 activate
 exit-address-family
!

Config on CE1:
=============​
interface Loopback0
 ip address 100.1.1.1 255.255.255.255
!
interface Ethernet0/0
 ip address 192.168.10.2 255.255.255.252
!
router bgp 65535
 bgp router-id 100.1.1.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 192.168.10.1 remote-as 100
 !
 address-family ipv4
  network 100.1.1.1 mask 255.255.255.255
  neighbor 192.168.10.1 activate
 exit-address-family
!

Config on CE2:
==============
interface Loopback0
 ip address 100.2.2.1 255.255.255.255
!
interface Ethernet1/0
 ip address 192.168.20.2 255.255.255.252
!
router bgp 65000
 bgp router-id 100.2.2.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 192.168.20.1 remote-as 100
 !
 address-family ipv4
  network 100.2.2.1 mask 255.255.255.255
  neighbor 192.168.20.1 activate
 exit-address-family
!

 

Verification

Output on CE1:
==============
CE1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      100.0.0.0/32 is subnetted, 2 subnets
C        100.1.1.1 is directly connected, Loopback0
B        100.2.2.1 [20/0] via 192.168.10.1, 00:42:16
      192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.10.0/30 is directly connected, Ethernet0/0
L        192.168.10.2/32 is directly connected, Ethernet0/0
      192.168.20.0/30 is subnetted, 1 subnets
B        192.168.20.0 [20/0] via 192.168.10.1, 00:23:50

CE1#ping 100.2.2.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.2.2.1, timeout is 2 seconds:
Packet sent with a source address of 100.1.1.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
CE1#

The above output shows that the routing table on CE1 has the routes from remote CE router CE2. Also, we can see that there is end to end reachability between CE1 and CE2 router loopbacks. Further looking at the output on PE1 (below), the VRF routing table of PE1 (VRF ABC) is populated with necessary prefixes.

PE1#show ip route vrf ABC

Routing Table: ABC
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      100.0.0.0/32 is subnetted, 2 subnets
B        100.1.1.1 [20/0] via 192.168.10.2, 01:06:49
B        100.2.2.1 [200/0] via 3.3.3.3, 00:44:36
      192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.10.0/30 is directly connected, Ethernet0/0
L        192.168.10.1/32 is directly connected, Ethernet0/0
      192.168.20.0/30 is subnetted, 1 subnets
B        192.168.20.0 [200/0] via 3.3.3.3, 00:26:15
PE1#

 

Troubleshooting

Step1: The first step in troubleshooting MPLS VPN setup is to verify the LSP path between PE to PE. This can be done using two methods

1. MPLS PING

If the MPLS ping goes through from PE to PE loopback, then it would confirm that the LSP (Label Switched Path) is complete and there is no problem with it.

PE1# ping mpls ipv4 3.3.3.3 255.255.255.255        
Sending 5, 72-byte MPLS Echos to 3.3.3.3/32, 
     timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface, 
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 
  'P' - no rx intf label prot, 'p' - premature termination of LSP, 
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
 Total Time Elapsed 3 ms

PE1#

2. PE to PE VRF PING

You can also perform a PE to PE to the VRF's IP Address:

PE1# ping vrf ABC 192.168.20.1 source 192.168.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
PE1#

This would again confirm the same thing that the LSP is complete in the Service Provider Core.

Step2: Verifying the VPN and IGP label along the path. For this, the first step is check the VPN label. 

1. Verifying VPN Label:

PE1#show mpls forwarding vrf ABC
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
19         No Label   100.1.1.1/32[V]  2136          Et0/0      192.168.10.2
20         No Label   192.168.10.0/30[V]   \
                                       570           aggregate/ABC 
PE1#

The local label value shown above i.e. 19 and 20 are nothing but the VPN labels. show mpls forwarding vrf command helps with checking the VPN label. There is another way to check the VPN labels and that is using BGP. Since BGP allocates the VPN label, you can verify the VPN labels for the prefixes (local as well as remote).

PE1# show ip bgp vpnv4 all 100.1.1.1
BGP routing table entry for 1:1:100.1.1.1/32, version 2
Paths: (1 available, best #1, table ABC)
  Advertised to update-groups:
     2         
  Refresh Epoch 1
  65535
    192.168.10.2 (via vrf ABC) from 192.168.10.2 (100.1.1.1)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Extended Community: RT:1:1
      mpls labels in/out 19/nolabel
      rx pathid: 0, tx pathid: 0x0
PE1#

All the local as well as remove VPN labels can be verified using the above command. There is another easy way to look at the VPN labels:

PE1# show ip bgp vpnv4 all labels
   Network          Next Hop      In label/Out label
Route Distinguisher: 1:1 (ABC)
   100.1.1.1/32     192.168.10.2    19/nolabel
   100.2.2.1/32     3.3.3.3         nolabel/19
   192.168.10.0/30  0.0.0.0         20/nolabel(ABC)
   192.168.20.0/30  3.3.3.3         nolabel/20
Route Distinguisher: 100:1
   100.2.2.1/32     3.3.3.3         nolabel/19
   192.168.20.0/30  3.3.3.3         nolabel/20

PE1#

In the above output, the label values highlighted on the left are the local labels (In Label) where as label values highlighted in the right (Out Label) are the remote prefix VPN label. You can filter the labels based on the RD value as well using "show ip bgp vpnv4 rd <rd_value> labels".

2. Verifying IGP labels in the path

For for the traffic destined to the remote prefix learnt from remote PE - PE2, we can check the vrf Routing table and then check the IGP labels for the same along the path:

PE1# show ip route vrf ABC 100.2.2.1 

Routing Table: ABC
Routing entry for 100.2.2.1/32
  Known via "bgp 100", distance 200, metric 0
  Tag 65000, type internal
  Last update from 3.3.3.3 01:40:38 ago
  Routing Descriptor Blocks:
  * 3.3.3.3 (default), from 3.3.3.3, 01:40:38 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 65000
      MPLS label: 19
      MPLS Flags: MPLS Required
PE1#

PE1# show mpls forwarding 3.3.3.3
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
16         17         3.3.3.3/32       0             Et1/0      12.12.12.2  
PE1#

In above output, the next-hop for prefix learnt (100.2.2.1/32) from remote CE router CE2 is 3.3.3.3 which is the loopback IP of PE2 router. The LFIB (Label Forwarding Information Base) for 3.3.3.3 shows that outgoing label as 17. This label 17 is learnt from the P-1 router where it should be the local label.

P-1# show mpls forwarding 3.3.3.3
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
17         Pop Label  3.3.3.3/32       476193        Et0/0      23.23.23.2  
P-1#

Since the prefix is present on PE2 router itself, it will send a Pop label (implicit-null label) to P-1 router which is seen as the outgoing label on P-1 router for the PE2 loopback.

Step 3. Verifying CEF

On Cisco platform, CEF takes care of forwarding. Thus, its important to verify that the CEF is programmed correctly.

PE1# show ip cef vrf ABC 100.2.2.1 det
100.2.2.1/32, epoch 0, flags [rib defined all labels]
  recursive via 3.3.3.3 label 19
    nexthop 12.12.12.2 Ethernet1/0 label 17
PE1#

In the cef output for the remote prefix, we can see that label 19 is the VPN label where as 17 is the IGP label and we can also see the respective next-hop value. We can also check the Service Provider core for the IGP labels and forwarding information using the above method.

Step 4. Traceroute

Traceroute can be used for checking the labelling and figure where a possibly problem could be.

PE1#traceroute vrf ABC 100.2.2.1 numeric 
Type escape sequence to abort.
Tracing the route to 100.2.2.1
VRF info: (vrf in name/id, vrf out name/id)
  1 12.12.12.2 [MPLS: Labels 17/19 Exp 0] 1 msec 1 msec 0 msec
  2 192.168.20.1 [MPLS: Label 19 Exp 0] 0 msec 0 msec 1 msec
  3 192.168.20.2 0 msec *  1 msec
PE1#

We can check the LSP path as well using the trace from PE to PE:

PE1# trace mpls ipv4 3.3.3.3 255.255.255.255
Tracing MPLS Label Switched Path to 3.3.3.3/32, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface, 
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 
  'P' - no rx intf label prot, 'p' - premature termination of LSP, 
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 12.12.12.1 MRU 1500 [Labels: 17 Exp: 0]
L 1 12.12.12.2 MRU 1500 [Labels: implicit-null Exp: 0] 1 ms
! 2 23.23.23.2 1 ms
PE1#

Hope the above steps helps you troubleshoot Basic MPLS VPN deployments. Even for complex MPLS VPN deployments like Inter-AS MPLS VPN or CSC (Carrier Supporting Carrier), these steps can serve useful.

1 Comment
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: