Cat6500 - HSRP Failing / High CPU Load

[cisco_lite]

The 'show processes cpu' is gone upto 98% on utilization on both Cat switches in redundancy.

The process consuming highest CPU is 'IP Input'.

Is there any way I can identify the host that is causing broadcast.

show cdp nei shows other switches only.

Please assist.

[cisco_lite]

Hello Giuseppe,

The root id is the same on both devices for vlan12

I noticed something, not sure if its an issue.

show spanning-tree vlan 12 detail

gives ' VLAN0012 is executing the ieee compatible Spanning Tree protocol' on standby switch and gives 'VLAN0012 is executing the rstp compatible Spanning Tree protocol' on primary switch'. Is that ok ?

I have enabled ip accounting on the interface, but nothing is seen on 'show ip accouting'. ip accounting was enabled with

interface vlan 12

ip accounting

Another observation, I was trying to go one-by-one with SVIs. So I shutdown Vlan12 (to which front end servers are connected) and brought up another vlan13 which is connected to FWSM outside. On primary switch 'no shut' on vlan13 works fine. But on the standby switch it says

'Forcing SVI13 to stay shutdown (SVI 20 tied to line card in slot 1.)'

Any clues.

Thanks.

[cisco_lite]

Please ignore Forcing SVI to stay shutdown error. It occured due to another shut duplicate vlan shared with FWSM. I have removed the duplicate vlan.

With vlan12 down, I brought up vlan13 and did not see any hike in cpu. So the culprit is vlan12 to which front end servers are connected. Please let me know how can I identify the source of broadcast traffic in vlan12. Why wasn't I able to see the broadcast on debug ip packet detail on the Cat6500.

Please advise.

Thanks.

[cisco_lite]

Additional info,

vlan12 is assigned to the ACE module configured in bridged mode. vlan14 is the server vlan and vlan12 is the client vlan(SVI).

The physical ports on ethernet module are connected to vlan14 (i.e. Server vlan). Vlan12 which shows a high broadcast traffic does not have any physical ports in the vlan.

Thanks.

[Giuseppe Larosa]

Hello Cisco_lite,

gives ' VLAN0012 is executing the ieee compatible Spanning Tree protocol' on standby switch and gives 'VLAN0012 is executing the rstp compatible Spanning Tree protocol' on primary switch'. Is that ok ?

it looks like that the two switches are running two different modes of STP:

rPVTP should fall back to PVST on detection of a legacy neighbor.

I would suggest you to configure RPVST also on the standby

config t

spanning-tree mode rapid-pvst

Also if ACE is bridging you need to think of it as a bridge/switch too.

FWSM can convert PVST BPDUs from one vlan to the other.

I will look for ACE

what are the vlans bridged by ACE ?

are VL12 and Vl14 ?

Hope to help

Giuseppe

[Giuseppe Larosa]

Hello Cisco_Lite,

the ACE can convert PVST BPDUs from vlan x to vlan y

be aware that ACE by bridging joins two broadcast domains into a single one.

by default supervisor STP BPDUs are not allowed but if you want to do this you need:

NoteIf you use failover, you must permit BPDUs on both interfaces with an EtherType ACL to avoid bridging loops.

and

important note

NoteBefore allowing or blocking BPDUs on the ACE, you must disable the spanning-tree loopguard default IOS command (if configured) on the Catalyst 6500 supervisor. Otherwise, if you allow and then block BPDUs on the ACE, the ACE port enters the blocking state, resulting in a complete outage. To recover, you must reboot the ACE.

So there are chances that two ACE in two C6500 in bridging mode are creating a loop

C6500_1 --- ACE_1

VLx Vly

C6500_2 --- ACE_2

please give a look at the following chapter of ACE conf. guide

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/rtg_brdg/guide/bridge.html

Hope to help

Giuseppe

[cisco_lite]

Yes, the bridged vlans on ACE are 12 & 14.

Please find below the output of 'show int stats', show ip traffic, and show interfaces switching. I would like to identify which specific host in vlan 12 (all hosts are connected to vlan 14-bridged) is generating high traffic. On show interfaces switching I can't see anyone generating huge traffic. Please let me know the difference between Pkts Out and Pkts In in show interfaces switching output.

CORE-SW#sh int stats | b Vlan12

Vlan12

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 576571532 38111277318 563901 37537634

Route cache 2449 150205 25 2211

Distributed cache 17598303 9025600495 18650247 13836881113

Total 594172284 47137028018 19214173 13874420958

CORE-SW#sh ip traffic

IP statistics:

Rcvd: 705186107 total, 579197716 local destination

0 format errors, 0 checksum errors, 1544 bad hop count

0 unknown protocol, 0 not a gateway

0 security failures, 0 bad options, 11 with options

Opts: 0 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 0 record route

0 stream ID, 0 strict source route, 11 alert, 0 cipso, 0 ump

0 other

Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

0 fragmented, 0 couldn't fragment

Bcast: 440082 received, 0 sent

Mcast: 578301381 received, 2309157 sent

Sent: 2660882 generated, 125989596 forwarded

Drop: 28 encapsulation failed, 0 unresolved, 0 no adjacency

353 no route, 0 unicast RPF, 0 forced drop

0 options denied, 0 source IP address zero

CORE-SW#sh interfaces switching

GigabitEthernet4/3

Throttle count 0

Drops RP 0 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 0 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out

[Giuseppe Larosa]

Hello Cisco_lite,

I would suggest you to disable one ACE module and to see if one ACE disabled and the etherchannel trunk on we still see high cpu usage and broadcast.

for ip accounting

int vlan 12

ip accounting mac-address input

then after some minutes

do

sh ip accounting

Hope to help

Giuseppe

[cisco_lite]

Excellent!!!. Thanks Giuseppe for pointing in the right direction.

I shutdown the BVI on ACE module and the CPU usage came down. I recently added a new context to the ACE module and the ft group went out of sync. Hence, I could see 'Received ARP collision message' on both the ACE modules.

Could you please let me know how can I configure ft for a new context. The FT group for admin context was working perfectly fine. Can same FT group be used for non-admin context. Or is there a separate procedure to configure FT for non-admin context.

Regards.

[Giuseppe Larosa]

Hello Cisco_lite,

so something is wrong with the ACE

see

If you use failover, you must permit BPDUs on both interfaces with an EtherType ACL to avoid bridging loops.

The following is an example of an EtherType ACL that permits BDPUs:

host1/Admin(config)# access-list NONIP ethertype permit bdpu

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/rtg_brdg/guide/bridge.html

before thinking of FT link the two vlans bridged have to allow Supervisor's BPDU frames so that the redundant links can be blocked and the brigding loop doesn't appear anymore

notice that by default the ACE doesn't bridge the BPDUs and this create the problem.

Hope to help

Giuseppe

[Giuseppe Larosa]

Hello Cisco_lite,

for FT vlans groups and contexts you need to associate the context with an FT group

see

Associating a Context with an FT Group

An FT group consists of two members (contexts) with the same name, each residing on a different ACE. To associate a context with an FT group, use the associate-context command in FT group configuration mode. You need to make this association for both redundant contexts in an FT group. The syntax of this command is:

associate-context name

For the name argument, enter the unique identifier of the context that you want to associate with the FT group.

For example, enter:

host1/Admin(config-ft-group)# associate-context C1

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/administration/guide/redundcy.html#wp1008774

Hope to help

Giuseppe