Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
3106
Views
0
Helpful
5
Replies

high cpu on 6500

Muhammed AKYUZ
Level 1
Level 1

‎03-20-2012 01:18 AM - edited ‎03-07-2019 05:40 AM

Hi ,

what should be the problem?

SUP2T-D#sh proce cpu hist

    11111111111     1111     1111          1111    1111      

    0000000000099999000099999000099999999990000999900009999999

    0000000000099999000099999000099999999990000999900009999999

100 **********************************************************

90 **********************************************************

80 **********************************************************

70 **********************************************************

60 **********************************************************

50 **********************************************************

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5   

               CPU% per second (last 60 seconds)

SUP2T-D#sh proce cpu sorted

CPU utilization for five seconds: 100%/83%; one minute: 99%; five minutes: 99%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

416     3324188    879928       3777 13.05% 14.42% 14.45%   0 Spanning Tree   

633      104408      5091      20508  1.50%  0.53%  0.45%   0 Env Poll        

  75       22000       298      73825  1.10%  0.13%  0.07%   0 Per-minute Jobs 

168       69696    163563        426  0.39%  0.23%  0.22%   0 slcp process    

   2         532      1010        526  0.07%  0.00%  0.00%   0 Load Meter      

712         504      2467        204  0.07%  0.00%  0.00%   0 NF AM Main Proc 

352        9224      1410       6541  0.07%  0.01%  0.00%   0 Lif stats hw rea

413       82300    238061        345  0.07%  0.03%  0.00%   0 MOP Protocols   

  10       28304      1454      19466  0.07%  0.07%  0.07%   0 Check heaps     

   9           0         2          0  0.00%  0.00%  0.00%   0 LC Registry Expo

  11           0         1          0  0.00%  0.00%  0.00%   0 Pool Manager    

   8           0         2          0  0.00%  0.00%  0.00%   0 PF Redun ICC Req

  13           4       165         24  0.00%  0.00%  0.00%   0 WATCH_AFS       

  14           0         1          0  0.00%  0.00%  0.00%   0 License Client N

  15           0         1          0  0.00%  0.00%  0.00%   0 Inode Table Dest

   7           0         1          0  0.00%  0.00%  0.00%   0 PF Redun ICC Req

  12           0         2          0  0.00%  0.00%  0.00%   0 Timers          

SUP2T-D#show ibc                         

====================================

Inband counters and statistics

====================================

Interface information:

        Interface IBC0/0(idb 0x5172A510)

        5 minute  rx rate 132671000 bits/sec  215989 packets/sec

        5 minute  tx rate 5324000 bits/sec  6655 packets/sec

        841634035 Packets input, 61939066968 bytes

        674810428 broadcasts received

        31998854 Packets outputs, 3210901354 bytes

        0 broadcasts sent

        0 Inband input packet drops

        0 Inband output packet drops

        1 IBC resets

        *** Inband Generic Counters***

        841634104 Packets Input, 1809524824 Bytes

        674810466 Broadcasts Input

        31999239 Packets Output

        816543102 Total Drops

        0 software bridged paks

        0 Packets Fast switching paks

        Potential/Actual paks CEF switched 30197067/49692

        Potential/Actual paks tag CEF switched 0/0

        0 packets xconnect CEF switched

        0 Packets xconnect L2 switched

        0 packets xconnect dropped

        0 Packets immediately punted

        Potential/Actual paks copied to process level 826935947/10304396

        Potential/Actual paks copied to L2 process level 14645067/14645067

        14647635 Packets L2 processed

        2568 packets L2 switched

        0 Encapsulation corrections

        Internally intercepted packets:

         0 exc, 151275503 dindex (14647635 sp, 0 old index), 796790606 cap1, 0 cap2

        Earl8 shimmed paks rcvd:

         0 pseudoipv6 0 svchdr 30195034 shim

        *** INBAND PACKET DROPS & FAILURES ***:

          0 shim_processing 0 shim_error

          0 no idb 0 get_l2_idb failures 14644875 if_input changes

          0 intercept 88449 spd

          0 Number of times intercept function(s) returning incorrect values

          0 bridge loopback 0 diag

          0 fastsend 0 soutput 0 no mac descriptors 0 NULL if_outputs

          816543102 throttle 580 throttle sneaks

          0 encapsulation 24769 get_lif

          0 if_input corrections 0 uncorrectable if_inputs

          0 corr ecc errors

          0 null parse packet

          17360486 shim insertions attempted at tx, 24769 failed

          0 times les cef vector not set

          0 times les cef vector disabled

SUP2T-D#Show netdr captu                 

A total of 4096 packets have been captured

The capture buffer wrapped 0 times

Total capture capacity: 4096 packets

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  20820000 000A0000 00020000 51080000 06000002 8E000004 00000008 200A9360

destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8C6

              5F806400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  28820100 000A0000 00000000 51080000 06000052 8E000004 00000008 200AA3DA

destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 00141B30

              50006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  30820000 000A0000 00020000 51080000 06000002 0E000004 00000008 200A69C3

destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8C6

              5F806400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  40820100 000A0000 00000000 51080000 06000002 0E000004 00000008 200A3567

destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8C6

              5F806400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  48820000 000A0000 00020000 51080000 06000012 0E000004 00000008 200AD5C0

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8B2

              2B006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  58820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A2961

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8B2

              2B006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  60820000 000A0000 00020000 51080000 06000052 8E000004 00000008 200A577E

destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 00141B30

              50006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  68820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200AB961

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8B2

              2B006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  70820000 000A0000 00020000 51080000 06000002 8E000004 00000008 200AA365

destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8C6

              5F806400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  78820100 000A0000 00000000 51080000 06000052 8E000004 00000008 200A93DF

destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 00141B30

              50006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  80820000 000A0000 00020000 51080000 06000002 0E000004 00000008 200A79C9

destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8C6

              5F806400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  98820000 000A0000 00020000 51080000 06000012 0E000004 00000008 200A65CF

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8B2

              2B006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  A0820000 000A0000 00020000 51080000 06000012 0E000004 00000008 200ACDCF

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8B2

              2B006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  A8820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A796E

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8B2

              2B006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  B0820000 000A0000 00020000 51080000 06000052 8E000004 00000008 200AE771

destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 00141B30

              50006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  B8820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A096E

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 0021D8B2

              2B006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

  bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

  cap1 1, cap2 0

  C8820100 000A0000 00000000 51080000 06000052 8E000004 00000008 200A83D5

destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

              00000400 023C0005 0002D805 06000200 01070006 00141B30

              50006400 01799001 01019101 02EE0500 00000000 00000000

              00000000 00

Labels:
0 Helpful
5 Replies 5

nkarpysh
Cisco Employee
Cisco Employee

‎03-20-2012 01:37 AM

Hello,

CPU is high due to interrupts (traffic) and it is good you have taken nedtr capture.

Ethertype 6002 is MOP for incoming packets

1. It looks like your router may transmit DECnet MOP (DEC Maintenance Operation Protocol (MOP)) packets (ethertype 6002) to multicast

MAC address ab00.0002.0000

2. Those packets sourced from Vlan 10(MAC 0023.33C5.AD80) were massively punted to CPU -

a part of netdrive capture below:

------- dump of incoming inband packet -------

l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943

dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)

cap1 1, cap2 0

B8820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A096E

destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002

layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000

00000400 023C0005 0002D805 06000200 01070006 0021D8B2

2B006400 01799001 01019101 02EE0500 00000000 00000000

00000000 00

http://www.cisco.com/en/US/tech/tk870/tk136/tk885/technologies_tech_note09186a0080093cd1.s
html

3.You can jcheck if  MOP protocol and MOP sysid enabled on on Vlan4005 or inout interfaces Gi1/1 and Gi1/3 and consider if you want to remove.

4. You can also trace the source MAcs of coming packets and understand why MOP traffic is coming from it.

Hope this helps,

Nik

HTH,
Niko
0 Helpful

Muhammed AKYUZ
Level 1
Level 1
In response to nkarpysh

‎03-20-2012 08:59 AM

Hi,

thank you for your answer.

I have disable mop on vlan interfaces, but still same.

how can i completely disabe mop?

0 Helpful

nkarpysh
Cisco Employee
Cisco Employee
In response to Muhammed AKYUZ

‎03-20-2012 09:04 AM

Hey.

Did you check it on physical interfaces as well? It might be needed to be disabled there. Also you can see where those source MAcs 00.21.D8.B2.2B.00 and 00.14.1B.30.50.00 are coming from. I do see these two sending MOP packets.

check where those are learnt from with "show mac-addr addr 0014.1B30.5000" or similar (based on platform) command and go to the device connected on that interface to do same check until you find the end host with that MAC.

Hope that helps.

Nik

HTH,
Niko
0 Helpful

Muhammed AKYUZ
Level 1
Level 1
In response to nkarpysh

‎03-20-2012 09:07 AM

hi,

i have diasble mop on physical interfaces also.  g1/1 ang g1/3 are also 6500. I have disable mop on their phsical and vlan interfaces but netdr output still shows ether 6002..  can i police these packets via COPP?

0 Helpful

nkarpysh
Cisco Employee
Cisco Employee
In response to Muhammed AKYUZ

‎03-20-2012 09:16 AM

I guess CoPP wont help here as ethertype 6002 is not ip so IP acl wont help and Copp afaik does not support MAC acl.

You can try to create MAC ACL or VACL and apply that on port just to drop those MAC addresses as sources.

I guess now even with MOP disabled switch treat destination MAC of those packets like some mcast MAC and send copy ofr CPU analysis.

Nik

HTH,
Niko
0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card