High CPU load on 6500, NBAR problem

ashirshov · ‎06-27-2008

After apply policy-map with class test "match protocol icmp" CPU load up to 40-60%. After remove this policy from interface CPU load 40-50% saved.

In docs says, match protocol, match length is soft triggers. Why CPU continue high load? Only rebooting help me?

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(33)SXH, RELEASE SOFTWARE (fc5)

sh proc cpu sorted 5sec

CPU utilization for five seconds: 41%/35%; one minute: 45%; five minutes: 64%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

6 70916580 3969191 17866 4.00% 0.68% 0.57% 0 Check heaps

151 128531380 912061034 140 0.63% 0.92% 1.06% 0 IP Input

211 25497384 25547561 998 0.39% 0.33% 0.39% 0 CEF: IPv4 proces

36 22724516 87274634 260 0.07% 0.12% 0.11% 0 ARP Snoop

9 56675336 114761987 493 0.07% 0.34% 0.37% 0 ARP Input

147 9624 1788 5382 0.07% 0.58% 0.33% 3 Virtual Exec

4 3308 484 6834 0.00% 0.05% 0.09% 0 Exec

8 0 2 0 0.00% 0.00% 0.00% 0 Timers

sh proc cpu sorted 1min

CPU utilization for five seconds: 36%/34%; one minute: 41%; five minutes: 51%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

151 128535692 912082220 140 0.55% 0.81% 0.97% 0 IP Input

6 70918992 3969313 17866 0.00% 0.52% 0.54% 0 Check heaps

211 25499128 25547986 998 0.39% 0.37% 0.37% 0 CEF: IPv4 proces

9 56676888 114763634 493 0.07% 0.31% 0.36% 0 ARP Input

346 33609300 240855846 139 0.00% 0.14% 0.16% 0 Port manager per

36 22724984 87275991 260 0.07% 0.11% 0.11% 0 ARP Snoop

203 15344736 4053426 3785 0.15% 0.09% 0.10% 0 IPC LC Message H

21 27432904 386882045 70 0.00% 0.08% 0.06% 0 IPC Seat Manager

385 26949452 35944691 749 0.07% 0.05% 0.17% 0 SNMP ENGINE

221 6649952 8217903 809 0.00% 0.03% 0.01% 0 HIDDEN VLAN Proc

44 8087132 329081 24574 0.00% 0.03% 0.00% 0 Per-minute Jobs

394 34330576 1273894 26949 0.31% 0.02% 0.00% 0 BGP Scanner

sh proc cpu sorted 5min

CPU utilization for five seconds: 42%/39%; one minute: 42%; five minutes: 50%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

151 128536144 912084187 140 0.79% 0.86% 0.97% 0 IP Input

6 70919224 3969327 17866 0.00% 0.52% 0.54% 0 Check heaps

9 56677148 114763800 493 0.39% 0.43% 0.38% 0 ARP Input

211 25499300 25548026 998 0.39% 0.37% 0.37% 0 CEF: IPv4 proces

346 33609372 240856156 139 0.47% 0.15% 0.16% 0 Port manager per

385 26949472 35944703 749 0.07% 0.04% 0.16% 0 SNMP ENGINE

147 9736 1826 5331 0.07% 0.04% 0.12% 3 Virtual Exec

36 22725044 87276123 260 0.15% 0.14% 0.12% 0 ARP Snoop

203 15344792 4053434 3785 0.07% 0.09% 0.10% 0 IPC LC Message H

21 27433056 386883032 70 0.00% 0.11% 0.07% 0 IPC Seat Manager

343 9936724 70132288 141 0.00% 0.01% 0.04% 0 IP SNMP

4 3308 484 6834 0.00% 0.00% 0.02% 0 Exec

Giuseppe Larosa · ‎06-29-2008

Hello,

if I understand correctly even after the policy-map has been removed your main cpu usage was high.

I would try to clear the CEF process and then to verify that most of traffic is processed by CEF. Try a clear ip route * to trigger a CEF table recalculation.

It looks like that main cpu is processing a lot of IPv4 packets (IP input process). So verify the CEF and multilayer switching status on your c6500.

By the way you don't need NBAR to classify icmp traffic an extended ACL is enough for that !

NBAR should be used for traffic flows that can use dynamic ports and in your system it may be unsupported at all.

looking at the following

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/qos.html#wp1581702

it says that:

QoS in Cisco IOS Software Release 12.2SX (PFC QoS) uses some Cisco IOS modular QoS CLI (MQC). Because PFC QoS is implemented in hardware, it supports only a subset of the MQC syntax.

â€¢With Cisco IOS Software Release 12.2SX, the PFC3 does not support Network-Based Application Recognition (NBAR).

I hope you have solved your issue without a device reload.

I think the IOS CLI should warn when trying to enable a feature that can cause a so high performance penalty because it is not supported in hardware in the PFC3.

hope to help

Giuseppe

ashirshov · ‎06-29-2008

Problem was fixed:

I'am just create class map with match protocol icmp

make policy map and apply this to interfaces.

After remove: 1) service policy, 2) remove policy-map from system config, and then 3)remove class map from config.

After CPU load is normal.

Thank you.

Giuseppe Larosa · ‎06-30-2008

Hello Andrey,

I'm happy that you didn't need to reload the 6509 switch and that removing all the objects related to the QoS service policy has solved the issue.

It is useful to provide feedback on solutions to a problem described in the forum because it can help somebody else in the future. It completes the case history.

Best Regards

Giuseppe