High CPU on a 3750

samuel.kling · ‎10-02-2007

Hello,

I have a 3750G which has got a high CPU (up to 40%), mostly at interrupt level (41%/28%).

I think the problem comes from IP packets that are processed instead of beeing hardware switched.

ROUTER#sh platform ip unicast failed route

Total of 0 covering fib entries

Entries covered by Actual default route(0.0.0.0/0)

10.56.102.0/24 Tbl:0 : Cover:0.0.0.0/0 Tbl:0

10.56.103.0/24 Tbl:0 : Cover:0.0.0.0/0 Tbl:0

(etc... : I've got 9 routes like this)

Total of 9 entries covered by 0.0.0.0/0 Tbl:0

If I understand well this command, it prints out the routes that could not be programmed in the hardware and are thus process-switched by the 3750.

But what can cause these routes not to be programmed in the hardware ? I've checked everything and I can't figure out where the problem comes from...

ROUTER#sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used

Masks/Values Masks/values

Unicast mac addresses: 672/5376 178/1355

IPv4 IGMP groups: 144/1152 6/26

IPv4 unicast directly-connected routes: 672/5376 178/1355

IPv4 unicast indirectly-connected routes: 144/1152 130/972

IPv6 Multicast groups: 672/5376 178/1355

IPv6 unicast directly-connected routes: 672/5376 178/1355

IPv6 unicast indirectly-connected routes: 128/1024 11/40

IPv4 policy based routing aces: 0/0 0/0

IPv4 qos aces: 512/512 6/6

IPv4 security aces: 1024/1024 22/22

IPv6 policy based routing aces: 0/0 0/0

IPv6 qos aces: 0/0 0/0

IPv6 security aces: 204/510 5/5

I am under the limits for all the groups.

Anyone has got an idea ?

dhabben · ‎10-02-2007

Take a look at what process is using up the processors time: This will show all the switches separately in the stack, the normal command just averages all of them.

remote command all show proc cpu history

remote command all show proc cpu sorted 1min

Our problem was the PM Callback process, it was going to 100% when multiple ports flapped at the same time. Bug ID CSCsj90406

Take a look at the output of:

show vtp status

Our problem was having VTP Pruning enabled trigering this bug. Disabling VTP Pruning worked around the problem until a fixed IOS is released.

samuel.kling · ‎10-03-2007

Thanks for your answer but the 3750 is not member of a stack. It is a standalone device.

And the diags I have already done allow me to be quite sure that the high CPU comes from IP packets that are not hardware switched.

By the way, I forgot to say that once I saw a strong CPU increase on this device just after adding a static route. When I removed this static route, the CPU went back to normal. And it increased again anormally when I added this route once more.

This route was a /25 prefix. And when I replaced it by the same route but configured as a /24, the CPU went back to normal again. All of those elements make me think that this is really a matter of programmation of the routes in the hardware of the switch. But I don't understand why since the command issued "show platform tcam utilization" does not highlight any threshold exceeding. I am within the hardware limits...

fdubus · ‎01-31-2008

Yes I had the same problem... 3750 with IOS IP Services 12.2(37)SE1.

Disabling VTP pruning worked perfectly

I did not try to upgrade yet

Thx for your post was very helpfull

dhabben · ‎01-31-2008

Upgrading to 12.2(44)SE will also resolve the issue without turning off VTP pruning.