High CPU with error "%ADJ-3-RESOLVE_REQ:" in Catalyst 4500-X VSS after making L3 function (static r...

lotfi.bouhaddad · ‎05-15-2014

We have a VSS based on 2x WS-C4500X-16., The VSS is used as Layer 2 Switch for diffrents Vlan in our DC.

After making the VSS as a Layer 3 gateway for our production VLAN and added 2 routes for routing purposes, we encountered a network down time with high CPU in the VSS and a huges log messages :

.May 14 12:11:25.947: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.7.22 Vlan100

.May 14 12:11:34.516: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.7.22 Vlan100

.May 14 12:11:40.072: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.1.1.254 Vlan100

.May 14 12:11:49.682: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.253 Vlan100

.May 14 12:11:55.079: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.1.1.254 Vlan100

.May 14 12:12:00.926: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.7.40 Vlan100

.May 14 12:12:06.701: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.8.32 Vlan100

.May 14 12:12:12.624: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.7.40 Vlan100

.May 14 12:12:21.627: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.7.40 Vlan100

.May 14 12:12:32.261: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.8.32 Vlan100

.May 14 12:12:41.801: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.2.105 Vlan100

.May 14 12:12:49.633: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.253 Vlan100

.May 14 12:12:54.831: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.1.1.254 Vlan100

.May 14 12:12:59.960: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.1.1.254 Vlan100

.May 14 12:13:08.745: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.253 Vlan100

.May 14 12:13:16.138: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.253 Vlan100

.May 14 12:13:22.393: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.253 Vlan100

.May 14 12:13:31.415: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.141 Vlan100

.May 14 12:13:38.944: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.2.215 Vlan100

.May 14 12:13:45.972: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.253 Vlan100

Bellow are the show version of our VSS,

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.00.SG RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 05-Dec-12 04:38 by prod_rel_team

ROM: 15.0(1r)SG10

S_C4500X_01 uptime is 33 weeks, 1 day, 14 minutes

Uptime for this control processor is 33 weeks, 1 day, 16 minutes

System returned to ROM by power-on

System restarted at 11:59:10 UTC Tue Sep 24 2013

Running default software

Jawa Revision 2, Winter Revision 0x0.0x40

Last reload reason: power-on

License Information for 'WS-C4500X-16'

License Level: ipbase Type: Permanent

Next reboot license Level: ipbase

cisco WS-C4500X-16 (MPC8572) processor (revision 9) with 4194304K/20480K bytes of memory.

Processor board ID JAE173303CF

MPC8572 CPU at 1.5GHz, Cisco Catalyst 4500X

Last reset from PowerUp

4 Virtual Ethernet interfaces

32 Ten Gigabit Ethernet interfaces

511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

Can you help please,

Naveen Venkateshaiah · ‎05-15-2014

these message are seen may be because for following reason,
the adjacency table might not have updated with the updated ARP entry when the CEF trying to locate valid adjacency for destination prefix/ip address. This might seems to be the timing issue, which is temporary.

if this continues check for the hsrp/glbp interfaces.

Regards,

Naveen

***rate if it is helpful***

lotfi.bouhaddad · ‎05-15-2014

Hi,

thanks for your reply, but there is no hsrp configured, just an interface vlan. with 2 static routes and the problem was there for more than an hour before we decided to rollback.

Is there a BugId with this problem in Cisco DataBase.

here is a show ip route

S_C4500X_01# show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is 10.2.1.253 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 10.2.1.253
10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.0.0/8 is directly connected, Vlan100
L 10.1.1.250/32 is directly connected, Vlan100
172.31.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.31.0.0/16 is directly connected, Vlan120
L 172.31.0.1/32 is directly connected, Vlan120
S 192.1.0.0/16 [1/0] via 10.1.1.254

and the show ip cef:

_C4500X_01# show ip cef
.May 14 12:13:57.859: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 10.2.1.158 Vlan100 f
Prefix Next Hop Interface
0.0.0.0/0 10.2.1.253 Vlan100
0.0.0.0/8 drop
0.0.0.0/32 receive
10.0.0.0/8 attached Vlan100
10.0.0.0/32 receive Vlan100
10.1.1.6/32 attached Vlan100
10.1.1.17/32 attached Vlan100
10.1.1.40/32 attached Vlan100
10.1.1.41/32 attached Vlan100
10.1.1.50/32 attached Vlan100
10.1.1.60/32 attached Vlan100
10.1.1.99/32 attached Vlan100
10.1.1.121/32 attached Vlan100
10.1.1.122/32 attached Vlan100
10.1.1.124/32 attached Vlan100
10.1.1.125/32 attached Vlan100
10.1.1.126/32 attached Vlan100
10.1.1.225/32 attached Vlan100
10.1.1.227/32 attached Vlan100
10.1.1.250/32 receive Vlan100
10.1.1.254/32 10.1.1.254 Vlan100
10.2.1.3/32 attached Vlan100
10.2.1.4/32 attached Vlan100
10.2.1.6/32 attached Vlan100
10.2.1.8/32 attached Vlan100
10.2.1.9/32 attached Vlan100
10.2.1.18/32 attached Vlan100
10.2.1.23/32 attached Vlan100
10.2.1.24/32 attached Vlan100
Prefix Next Hop Interface
10.2.1.26/32 attached Vlan100
10.2.1.28/32 attached Vlan100
10.2.1.29/32 attached Vlan100
10.2.1.31/32 attached Vlan100
10.2.1.103/32 attached Vlan100
10.2.1.108/32 attached Vlan100
10.2.1.109/32 attached Vlan100
10.2.1.124/32 attached Vlan100
10.2.1.129/32 attached Vlan100
10.2.1.137/32 attached Vlan100
10.2.1.139/32 attached Vlan100
10.2.1.143/32 attached Vlan100
10.2.1.144/32 attached Vlan100
10.2.1.159/32 attached Vlan100
10.2.1.167/32 attached Vlan100
10.2.1.174/32 attached Vlan100
10.2.1.175/32 attached Vlan100
10.2.1.176/32 attached Vlan100
10.2.1.181/32 attached Vlan100

10.2.4.38/32 attached Vlan100
10.2.4.39/32 attached Vlan100
10.2.4.43/32 attached Vlan100
10.2.4.47/32 attached Vlan100
10.2.4.51/32 attached Vlan100
10.2.4.63/32 attached Vlan100
10.2.4.65/32 attached Vlan100
10.2.4.69/32 attached Vlan100
10.2.4.71/32 attached Vlan100
10.2.4.73/32 attached Vlan100
10.2.4.102/32 attached Vlan100
10.2.4.106/32 attached Vlan100
10.2.4.107/32 attached Vlan100
10.2.4.113/32 attached Vlan100
10.2.4.116/32 attached Vlan100
10.2.4.119/32 attached Vlan100
10.2.4.120/32 attached Vlan100
10.2.4.122/32 attached Vlan100
10.2.4.141/32 attached Vlan100
10.2.4.148/32 attached Vlan100
10.2.6.7/32 attached Vlan100
Prefix Next Hop Interface
10.2.6.16/32 attached Vlan100
10.2.6.31/32 attached Vlan100
10.2.7.14/32 attached Vlan100
10.2.7.22/32 attached Vlan100
10.2.7.24/32 attached Vlan100
10.2.7.34/32 attached Vlan100
10.2.7.37/32 attached Vlan100
10.2.7.41/32 attached Vlan100
10.2.7.48/32 attached Vlan100
10.2.8.18/32 attached Vlan100
10.2.8.32/32 attached Vlan100
10.2.8.59/32 attached Vlan100
10.2.8.70/32 attached Vlan100
10.2.8.85/32 attached Vlan100
10.2.8.88/32 attached Vlan100
10.2.8.104/32 attached Vlan100
10.2.8.135/32 attached Vlan100
10.2.99.10/32 attached Vlan100
10.2.99.54/32 attached Vlan100
10.255.255.255/32 receive Vlan100
127.0.0.0/8 drop
172.31.0.0/16 attached Vlan120
172.31.0.0/32 receive Vlan120
172.31.0.1/32 receive Vlan120
172.31.0.5/32 attached Vlan120
172.31.0.29/32 attached Vlan120
172.31.255.255/32 receive Vlan120
192.1.0.0/16 10.1.1.254 Vlan100
224.0.0.0/4 drop
224.0.0.0/24 receive
Prefix Next Hop Interface
240.0.0.0/4 drop

and show ip arp

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.1.1.1 0 aa00.0400.c286 ARPA Vlan100
Internet 10.1.1.6 0 0050.5689.24b8 ARPA Vlan100
Internet 10.1.1.10 0 0050.5694.7d20 ARPA Vlan100
Internet 10.1.1.11 0 0050.5694.7d20 ARPA Vlan100
Internet 10.1.1.12 0 0050.5694.6ae7 ARPA Vlan100
Internet 10.1.1.13 0 0050.5694.6ae7 ARPA Vlan100
Internet 10.1.1.14 0 0050.568a.6321 ARPA Vlan100
Internet 10.1.1.16 0 0050.5694.0ab5 ARPA Vlan100
Internet 10.1.1.17 0 0050.5694.493d ARPA Vlan100
Internet 10.1.1.40 0 0013.19b0.9c40 ARPA Vlan100
Internet 10.1.1.41 0 1c17.d35a.c840 ARPA Vlan100
Internet 10.1.1.50 0 0002.b9b4.a5c0 ARPA Vlan100
Internet 10.1.1.60 0 000a.410f.e500 ARPA Vlan100
Internet 10.1.1.71 - 0008.e3ff.fc28 ARPA Vlan100
Internet 10.1.1.96 0 e02f.6d12.4df3 ARPA Vlan100
Internet 10.1.1.98 0 0050.5696.6d86 ARPA Vlan100
Internet 10.1.1.99 0 0050.5696.6d88 ARPA Vlan100
Internet 10.1.1.121 0 e02f.6d12.4dea ARPA Vlan100
Internet 10.1.1.122 0 e02f.6d12.4e61 ARPA Vlan100
Internet 10.1.1.123 0 e02f.6d5b.c10e ARPA Vlan100
Internet 10.1.1.124 0 e02f.6d17.c869 ARPA Vlan100
Internet 10.1.1.125 0 e02f.6d5b.c217 ARPA Vlan100
Internet 10.1.1.126 0 e02f.6d17.c8ec ARPA Vlan100
Internet 10.1.1.127 0 e02f.6d17.c876 ARPA Vlan100
Internet 10.1.1.128 0 e02f.6d5b.bef3 ARPA Vlan100
Internet 10.1.1.202 0 0000.85b7.9031 ARPA Vlan100
Internet 10.1.1.222 0 000f.f84d.2ca9 ARPA Vlan100
Internet 10.1.1.225 0 000f.f84d.3963 ARPA Vlan100
Internet 10.1.1.227 0 00c0.ee26.9367 ARPA Vlan100
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.1.1.250 - 0008.e3ff.fc28 ARPA Vlan100
Internet 10.1.1.254 0 0000.0c07.ac07 ARPA Vlan100
Internet 10.2.1.2 0 0011.4333.bcda ARPA Vlan100
Internet 10.2.1.3 0 0050.5689.5d38 ARPA Vlan100
Internet 10.2.1.4 0 0050.5689.0404 ARPA Vlan100
Internet 10.2.1.6 0 0050.5689.6d3b ARPA Vlan100
Internet 10.2.1.7 0 1cc1.def4.6940 ARPA Vlan100
Internet 10.2.1.8 0 0050.5689.330e ARPA Vlan100
Internet 10.2.1.9 0 0012.793a.3ccc ARPA Vlan100
Internet 10.2.1.10 0 0012.7990.e5d3 ARPA Vlan100
Internet 10.2.1.13 0 0050.568a.6dcf ARPA Vlan100
Internet 10.2.1.15 0 0050.568a.60ff ARPA Vlan100
Internet 10.2.1.18 0 0050.5689.091b ARPA Vlan100
Internet 10.2.1.20 0 0050.5689.451c ARPA Vlan100
Internet 10.2.1.21 0 0050.568a.0cf4 ARPA Vlan100
Internet 10.2.1.22 0 0050.5689.6c59 ARPA Vlan100
Internet 10.2.1.23 0 0050.5696.6d9e ARPA Vlan100
Internet 10.2.1.24 0 0050.5689.76c4 ARPA Vlan100
Internet 10.2.1.26 0 0050.5689.2f4e ARPA Vlan100
Internet 10.2.1.27 0 0050.5689.0632 ARPA Vlan100
Internet 10.2.1.28 0 0050.5689.1ce9 ARPA Vlan100
Internet 10.2.1.29 0 0050.5689.6aaa ARPA Vlan100
Internet 10.2.1.31 0 0050.5689.0d1a ARPA Vlan100
Internet 10.2.1.37 0 0050.5696.6d81 ARPA Vlan100
Internet 10.2.1.103 0 d4be.d9be.8eef ARPA Vlan100
Internet 10.2.1.106 0 14fe.b5e1.c595 ARPA Vlan100
Internet 10.2.1.107 0 0023.ae7d.a966 ARPA Vlan100
Internet 10.2.1.108 0 d4be.d9c8.6770 ARPA Vlan100
Internet 10.2.1.109 0 14fe.b5e9.c5b5 ARPA Vlan100
Internet 10.2.1.110 0 14fe.b5ea.5f9d ARPA Vlan100
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.2.1.111 0 001e.c959.d4f0 ARPA Vlan100
Internet 10.2.1.114 0 b8ac.6f48.4538 ARPA Vlan100
Internet 10.2.1.115 0 14fe.b5e1.ed89 ARPA Vlan100
Internet 10.2.1.116 0 7845.c409.1959 ARPA Vlan100

Thanks

Lotfi

High CPU with error "%ADJ-3-RESOLVE_REQ:" in Catalyst 4500-X VSS after making L3 function (static routing)