I'm trying to solve/identify a problem for one of our customers in an PIX535 <-> 3750-stack enviroment.
I think I may know the answer but I really hope someone can prove me wrong.
100% CPU load (average for several hours during backup/batchjob hours!) where IP Input claims allmost all cpu time.
And yes, I've read "Troublesooting High CPU Utilization in IP Input Process" http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml
ASCII version of network topology:
External network (our AS)
Redundant PIX 535 FW
Redundant 3750stacks (Internal networks)
Even more Internal networks...
The problem arises in the 3750 stack connected to the Active PIX 535, on the transit Vlan between the PIX and the 3750 stack.
All internal networks with Vlan Interfaces configured in the 3750 stacks seems to do well regarding fast switching, as in 90% (or so) of the packets are NOT Process switched/routed. Packets from 3750 to PIX FW included.
But... 100% of all packets from PIX FW to 3750 is process switched!
I have found figures stating that 3750 is able to process switch somewhere between 2500 to 3000 pps which gives around 45Mbit troughput in best case. (This is well below what our customer needs)
To the best of my knowledge, this is how it works:
Inbound traffic from 'outside' and DMZ's on the PIX FW is simply thrown on the (from the PIX perspective) "next hop", which happens to be the Interface vlanX in the 3750. PIX knows that the IP address of the next hop has mac-address "AA-BB-CC-DD-EE-FF" and forwards all packets there on level 2.
Arriving in the 3750 all packets from PIX is addressed to Interface vlanX mac-address (3750 still not aware of destination host IP address). 3750 now has to pick up the packet to level 3 (IP) to see where it's destined to. Okey, 3750 knows where to forward the packet... another trunk or an access switch port etc.
As far as I understand, this has to be done with each and every packet comming from the PIX since I don't see there is any chanse for the 3750 to "IP CEF" this since all packets are destined for Interface vlanX mac-address on level2.
Please, someone tell me that I'm missing something here or else I will have to slap the designer of our "refreshed" datacenter core switch/router environment in his face... hard!
(We used to have 7200VXR routers connected with GigaInterfaces to Catalyst 4000 chassis before, which ran smoothly without any problems. Nearly 100 servers is served by the 3750 today).