High CPU on 3750, TCAM Full

v.c.bodenstab · ‎09-27-2008

Hi All,

I'm facing the following problem and before I contact Cisco TAC, I want to drop a line here.

I've got a 3750G-12S running IOS 12.2(37)SE1. It's got two L3 VLAN interfaces. Interface Gi1/0/1 is placed in one VLAN and int gi1/0/2 is placed in the other. I've got a few static routes (about 20) and 3 ACL's. The largest ACL is only 3 rules. I am routing between the two L3 interfaces.

Basically, the setup is a border router connected to our internetlink. Traffic is about 100Mbps sustained, 300Mbps peak.

I've got a second 3750 connected to the same ISP. It's doing the same. Both 3750 use HSRP for redundancy.

Now, when I apply a simple ACL (only 3 rules) to one of the VLAN interfaces, CPU usage shoots up to 70-80%. Furthermore, in the logging I can see the following message (i'm pasting command log as well):

Sep 27 17:05:10: %PARSER-5-CFGLOG_LOGGEDCMD: User:FOO logged command:interface VlanXX

Sep 27 17:05:26: %PARSER-5-CFGLOG_LOGGEDCMD: User:FOO logged command:ip access-group 150 in

Sep 27 17:05:26: %ACLMGR-4-UNLOADING: Unloading ACL input label 255 VLAN interfaces 24 IPv4/Mac feature

Sep 27 17:05:26: %ACLMGR-4-ACLTCAMFULL: ACL TCAM Full. Software Forwarding packets on Input label 255 on L3 L2

Sep 27 17:05:26: %ACLMGR-4-UNLOADING: Unloading ACL input label 255 VLAN interfaces 24 IPv6 feature

Sep 27 17:05:30: %SYS-5-CONFIG_I: Configured from console by FOO on vty0 (x.x.x.x)

I'm not using IPv6. What could be wrong?? This is the ooutput from "sh plat tcam uti"

CAM Utilization for ASIC# 0 Max Used

Masks/Values Masks/values

Unicast mac addresses: 784/6272 15/37

IPv4 IGMP groups + multicast routes: 144/1152 6/26

IPv4 unicast directly-connected routes: 784/6272 15/37

IPv4 unicast indirectly-connected routes: 784/6272 20/103

IPv4 policy based routing aces: 0/0 0/0

IPv4 qos aces: 896/896 18/18

IPv4 security aces: 1024/1024 27/27

Giuseppe Larosa · ‎09-27-2008

Hello Vincent,

cpu usage goes high because packets are process switched once the ACL TCAM is full.

May you post the config of acl 150 ?

Are you using any log options ?

The ACL TCAM entries look like to be used in a different manner then expected.

Hope to help

Giuseppe

v.c.bodenstab · ‎09-27-2008

Hi,

Thanks for the reply. This is the contents of ACL150.

access-list 150 permit ip any 145.x.x.0 0.0.255.255

access-list 150 permit ip any 224.x.x.0 15.255.255.255

access-list 150 deny ip any any

So, no logging option here. I already disabled logging because of the CPU usage :-)

Furthermore, this is an extra "sh plat tcam uti" at the moment ACL150 is applied to VLANXX.

mu-3750-1-surf#sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used

Masks/Values Masks/values

Unicast mac addresses: 784/6272 15/37

IPv4 IGMP groups + multicast routes: 144/1152 6/26

IPv4 unicast directly-connected routes: 784/6272 15/37

IPv4 unicast indirectly-connected routes: 784/6272 20/103

IPv4 policy based routing aces: 0/0 0/0

IPv4 qos aces: 896/896 18/18

IPv4 security aces: 1024/1024 27/27

Also some CPU info during an active ACL150:

mu-3750-1-surf#sh proc cpu sorted

CPU utilization for five seconds: 50%/45%; one minute: 48%; five minutes: 22%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

35 7854723 472057 16639 0.31% 0.03% 0.00% 0 Per-minute Jobs

167 1438 325 4424 0.31% 0.14% 0.20% 1 Virtual Exec

65 349131 836969194 0 0.15% 0.13% 0.04% 0 HLFM address lea

4 23751853 2856235 8315 0.00% 0.11% 0.07% 0 Check heaps

5 0 3936 0 0.00% 0.00% 0.00% 0 Pool Manager

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

7 0 1 0 0.00% 0.00% 0.00% 0 HRPC asic-stats

....

And when ACL150 is removed:

CPU utilization for five seconds: 5%/0%; one minute: 49%; five minutes: 31%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

167 1672 475 3520 0.47% 0.19% 0.18% 1 Virtual Exec

2 540 5610481 0 0.00% 0.00% 0.00% 0 Load Meter

1 336 7399 45 0.00% 0.00% 0.00% 0 Chunk Manager

4 23751970 2856247 8315 0.00% 0.08% 0.06% 0 Check heaps

3 50 1872225 0 0.00% 0.00% 0.00% 0 MDFS LC Download

5 0 3936 0 0.00% 0.00% 0.00% 0 Pool Manager

7 0 1 0 0.00% 0.00% 0.00% 0 HRPC asic-stats

...

Giuseppe Larosa · ‎09-27-2008

Hello Vincent,

I'm afraid the any keyword is not mapped to a single entry in the ACL TCAM and this may be the reason of your issue.

I agree this could be a bug.

line number 4 is useless for the implicit deny any any at the end of each ACL.

there is some difference for 3750 12S see the sdm prefer command

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_46_se/command/reference/cli2.html#wp9598134

they speak of an aggregator template for 3750 12S

if routing is enabled on the switch use:

sdm prefer routing desktop

then

reload

You must reload the switch for the configuration to take effect. If you enter the show sdm prefer command before you enter the reload privileged EXEC command, the show sdm prefer command shows the template currently in use and the template that will become active after a reload.

Desktop switches support only desktop templates; an aggregator switch (Catalyst 3750-12S) supports both desktop and aggregator templates. On an aggregator switch, if you do not enter the desktop keyword, the aggregator templates are selected.

From the internet traffic can arrive only to the IP subnets that you advertise.

Hope to help

Giuseppe

v.c.bodenstab · ‎09-29-2008

Hi Giuseppe,

How do you know 'any' isn't mapped to a TCAM entry? Is this by design?

I use the explicit deny for logging. Logging is disabled now because of the performance problem.

Currently both switches are set for 'aggregate default'. I've got another 3750, running the same IOS, the same sdm setting with no problems... While this 3750 has more VLAN interfaces and a far bigger ACL. Traffic is less though...

Giuseppe Larosa · ‎09-29-2008

Hello Vincent,

mine was only a guess because the ACL TCAM is full with a 4 entry ACL.

My impression is that traffic volume and even more important traffic variety of multiple possible ip flows from internet plays a role here.

the sdm scheme aggregate default could have a role in this: it is stated that is different from that of other 3750.

Well the name would suggest that aggregate scheme is more scalable I would suggest to schedule a maintanance window and to try on one of them to change the sdm strategy.

Or, you can open a service request complaining of this behaviour and get TAC feedback.

Hope to help

Giuseppe

v.c.bodenstab · ‎10-01-2008

Hi,

I've found a solution to this strange issue. The first step was to upgrade the IOS to 12.2(46)SE, IP services image. The second step was to change the SDM template to "aggregate routing".

If I apply the ACL's to the VLAN interfaces now, the TCAM error isn't displayed. Also, CPU usage is at an acceptable rate (< 8%).

OK, just wanted to let you guys know. Thanks for the effort :-)

Giuseppe Larosa · ‎10-01-2008

Hello Vincent,

it is good news you solved this issue and I rated it accordingly.

Providing feedback is a service to all people using the forum because it tells that the case has an happy end and a solution has been found.

If you like you can flag the solved symbol to show a solution was found.

Best Regards

Giuseppe

glen.grant · ‎10-01-2008

Something I didn't know about the 3750 series and SDM templates on these switches. For more info

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00801e7bb9.shtml