ACL order causes high IP-Input

[registrato]

Hello,

today I experienced a weird problem with a Catalyst 3550. One interface has two ACLs, B in ingress and A in egress. They've been in place for quite a long time without any problem. To sum it up, I trivially modify ACL A (add a routine "deny" for a host who isn't even transmitting) and IP-Input goes from 1% to 70%. I remove this last rule but to no avail. I remove the whole ACL A and everything is OK again, so the problem is with ACL A. Here comes the funny part: from this moment on, if I re-apply ACL A IP-Input goes thru the roof, no matter what ACL A contains. Even a dummy "permit ip any any" causes the problem. But if I apply ACL B before applying ACL A everything works just fine:

ip access-group A out +

ip access-group B in = high CPU

ip access-group B in +

ip access-group A = OK

I can't explain it.

[adrian.chadd]

catalyst 3550's are hardware-based routers, which mean you shouldn't ever see CPU use during IP forwarding. This, unfortunately, means you have to make sure you don't create configurations which stretch the hardware resources too far or you won't be able to forward all the packets in hardware.

The 3550 has limited TCAM entries for routing table entries, mac table entries, QoS/policing, ACLs and policy-based routing. You have to make sure you don't run out of any one of those or you'll start seeing packets punted to the CPU for handling.

'show sdm prefer' on the 3550 will tell you the basic TCAM resource allocations. You can then use 'show ip route summary' and 'show mac-address-table' to see if you've overflowed the MAC/IP table entries.

Do you have many ACLs on the 3550? Are they long and complicated?

Here's some debugging output available from the 3550:

blue-1#show tcam inacl 1 statistics

Ingress ACL TCAM#1: Number of active labels: 9

Ingress ACL TCAM#1: Number of masks allocated: 62, available: 146

Ingress ACL TCAM#1: Number of entries allocated: 157, available: 1507

blue-1#show tcam outacl 1 statistics

Egress ACL TCAM#1: Number of active labels: 6

Egress ACL TCAM#1: Number of masks allocated: 28, available: 180

Egress ACL TCAM#1: Number of entries allocated: 66, available: 1598

Make sure you haven't run out of TCAM entries. A notice should show up in 'show log' if/when you have and the hardware has to punt packets to the CPU for forwarding.

[registrato]

Thank you Adrian for your answer.

The problem seems to be that though there's enough space and the ACL is short, for some reason it's not loaded into the TCAM.

[adrian.chadd]

It might be that the ACL is too complicated or you've run out of a specific resource. For example, the TCAMv2 used on the 3550 uses one mask for a number of entries, which means you can run out of masks very quickly if your ACL is complicated. Newer hardware which uses TCAMv3 has one mask per entry, alleviating this problem.

Check the output of 'show log' as you load/unload the ACL and see what it says. Note that the device won't actually program the ACL into TCAM until its applied to an interface so you won't see anything logged until such happens. Make sure you've turned on 'logging notification debug' and 'logging buffered debug' before you do this too (and put it back to informational later!)

Don't forget to rate the post if it was helpful!

[registrato]

Issue resolved (for now). I suspect it was some kind of memory problem (maybe memory fragmentation? 3550 had been up for 1 yr+ , the ACL modified many times..).

The ACL was (is) field tested, short and very simple, I had plenty of masks and entries.

After a reload without applying any config changes everything went ok.

Thank you for you useful tips.