Cisco 3560 switch high utilization because of interrupt traffic

[jubair151]

Dear Team,

I am running with an IOS of 12.2(55)SE7 in my Cisco 3560 switches. I am observing high cpu utilization intermittently in my Cisco 3560 access switches.

I found that this high cpu utilization is because of interrupt traffic, but I am not able to what is the cause of this interrupt traffic. Kindly support me to find the interrupt traffic which is causing this high cpu utilization.

Please find the cpu sorted utilization and cpu controller cpu-interface as attached.

Regards,

Jubair.S

[Reza Sharifi]

There is no attachment

[InayathUlla Sharieff]

Jubair,

Interrupt means due to some reason packet were sent to CPU for process . Packet will be punted to cpu for below reason :

*              Packets with IP options, an expired Time to Live (TTL), or non-Advanced
Research Projects Agency (ARPA) encapsulation
*              Packets with special handling, such as tunneling
*              IP fragmentation
*              Packets that require Internet Control Message Protocol (ICMP) messages from
the RP or SP
*              Maximum transmission unit (MTU) check failure
*              Packets with IP errors, which include IP checksum and length errors
*              If the input packets return a bit error (such as the single-bit error
(SBE)), the packets are sent to the CPU for software processing and are corrected. The
system allocates a buffer for them and uses the CPU resource to correct it.
*              When PBR and reflexive access list are in the path of a traffic flow, the
packet is software switched, which requires an additional CPU cycle.
*              Adjacency same interface
*              Packets that fail the Reverse Path Forwarding (RPF) check-rpf-failure
*              Hardware resources full conditions :These resources include FIB,
content-addressable memory (CAM), and ternary CAM (TCAM).

Please capture the below commands at the time of high CPU:

?         Show proc cpu history

?         Show proc cpu sort | ex 0.0

?         Show platform port-asic stats drop

?         show spann det | i exec|of top|from

?         show interfaces switching

?         show sdm prefer

?         Show controllers cpu-interface

Link below can help you in doing the TS:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

HTH

Regards

Inayath

*Plz dont forget to rate all usefull posts.

[jubair151]

I noticed that I am getting this high CPU utilization after configuring flexconnect in my wireless network.  AP connected switch ports are configured with trunk for all the vlan. Does this make any hihg cpu utilization in the  swtitches??

[jubair151]

pls find  the CPU utilization attachment.

[InayathUlla Sharieff]

We cant say anything based on just 1 line that will this/that cause the high cpu.

I have given the reason of Interupt in my earlier reply please go through the same.

Incase you are seeing high cpu utlization capture us the requested outputs and we will see whats happening.

HTH

[jubair151]

Hi insharie,

Please find the requested logs.

Regards,

Jubair.S

[InayathUlla Sharieff]

Can we run some debugs here?

if yes kindly run the following commands:

show proc cpu sorted | ex 0.00 >> twice

Show platform port-asic stats drop >> I need this atleast 3-4 times in span of 2-3 mins 

show controllers cpu-interface   >> this same run this couple of times and watch out for the cpu interface which is increasing.

show platform tcam utilization 

debug platform cpu-queue <queue which is increasing >

HTH

[jubair151]

dear insharie ,

It may not be possible to run the debug command, as our network is a highly critical production network. You may please find the logs as attached.

Regards,

Jubair.S

[jubair151]

Dear insharie ,

One obsevation from my side is, I am getting huge packet output rate whrever my access points are connected which is in flexconnect mode. pls find the date rate as below.

JEDITBF8S03#show interfaces | include output rate
  5 minute output rate 3000 bits/sec, 1 packets/sec
  5 minute output rate 3577000 bits/sec, 4407 packets/sec
  5 minute output rate 3643000 bits/sec, 4420 packets/sec
  5 minute output rate 3850000 bits/sec, 4477 packets/sec
  5 minute output rate 3598000 bits/sec, 4429 packets/sec.

Will this be a problem.

[InayathUlla Sharieff]

Yes there could be possibilities.

What happens if you shut down those ports?

[jubair151]

dear insharie,

I got the real Culprit. One of the PC was highly sourced multicast traffic, i got this by checking the interface input traffic count. I generated the switch port utlization by using NSM tools and found that one of the switch port is sourcing high traffic. By this input, i found the source who has been generating high multicast traffic. After blocking this port, utlization has been come down.

Thanks for your suggestion.

Regards,

Jubair.S

[InayathUlla Sharieff]

Thanks Jubair.

Could you mark the thread as closed rather than leaving it open.

Thanks in advance.

[jubair151]

Dear Insharie,

I am checking the cause of CPU utilizaiton, by sniffing i came to know its because of IPv6 multicast traffic. Kindly find the attached sniffed screen shot. Kindly advise the practice to stop this in future.

Regards,

Jubair.S

[jubair151]

Hi,

I found the answer, pls refer the below link. HP EliteOne 800 system sourcing high ICMPv6 multicast traffic. Access switches will broadcast this traffic inturn, thereby switches CPU is getting highly Utilized.

http://social.technet.microsoft.com/Forums/windows/en-US/945125a7-035f-4f33-a9d2-c274717d7706/icmpv6-multicast-listener-report-messages-are-flooding-the-local-network?forum=w7itpronetworking.

Solution

~~~~~~

As of now we are running with old Network Adaptor driver software (12.6.47.0, released on 2013)  as attached. We have to upgrade the Network driver to 12.10.30.x version which is released on 2014 and disaable the IPv6 if not required. Kindly refer the below link.

Regards,

Jubair.S