Introduction
This document disucsses on high CPU condition seen in Catalyst 6500 platforms due to IOS Server Load Balancing (SLB) feature.
Problem Description:
Catalyst 6500 reporting high CPU due to interrupts and "IP Input" process.
C6K-A#show process cpu sorted | exclude 0.00
CPU utilization for five seconds: 98%/57%; one minute: 99%; five minutes: 97%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
227 816331652 12632718 4266 32.10% 31.69% 31.07% 0 IP Input
Troubleshooting and Symptoms:
(1)
The IOS SLB is configured with "nat server""
ip slb serverfarm TEST-FARM1
nat server
nat client POOL1
<snip>
(2)
Switch do NOT have any sw-installed Netflow entries:
C6K-A#show mls netflow ip sw-installed
Displaying Netflow entries in Active Supervisor EARL in module 5
No Entries
C6K-A#show mls netflow ip sw-installed module 1 <<=== Ingress DFC module
Displaying Netflow entries in Active Supervisor EARL in module 1
No Entries
(3)
SLB has active connections:
C6K-A#show ip slb connections
vserver prot client real state nat
-------------------------------------------------------------------------------
VSERVER-NAME TCP <client-ip>:<port> <real-ip>:<port> <TCP-State> S,C
<snip>
Root-Cause and Resolution:
When the client sends the traffic to virtual IP address, the load-balancer (in this case, IOS SLB) will NAT the traffic, as the real/physical severs are NOT aware of the virtual IP address.
Cat6500 with "nat server" configuration, the switch is NOT capable of creating hardware shortcuts. As a result, the traffic will be process/software switched. This is done by punting the traffic to the CPU and it can be verified by: (Here, 10.50.50.2 is IP address of a virtual server)
C6K-A#show tcam int vlan <client-vlan> acl in ip | inc 10.50.50.2
punt udp any host 10.50.50.2
redirect tcp any host 10.50.50.2 fragments
policy-route tcp any host 10.50.50.2 eq <port#>
<snip>
To resolve this issue, it is recommended to configure IOS SLB in "Dispatched" mode.
http://www.cisco.com/en/US/products/hw/modules/ps2706/products_configuration_example09186a0080094066.shtml
http://www.cisco.com/en/US/products/hw/routers/ps341/products_tech_note09186a0080134735.shtml
Further Information:
To know more on IOS SLB feature, please refer the 12.2SX Feature Configuration Guide:
http://www.cisco.com/en/US/docs/ios/12_2sx/feature/guide/slbsxf7.html
Please be aware that starting from 12.2(33)SXJ IOS SLB is NOT supported:
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/release/notes/features.html#wp4798296
For feedback and comments, please contact Yogesh at yramdoss@cisco.com
