3750 switch stack stopped working

Erick Bergquist · ‎03-11-2009

Hi,

I have a stack of 7 3750s, all on the same IOS, 12.2.25 IP base. Today the entire stack stopped passing traffic and there was one switch cycling power and not booting up all the way when it got to detecting the stackwise portion of the POST tests. We could not console into any switch either.

We bypassed the suspect problem switch and then could ping and get into the stack via telnet. We saw the following entries once a minute in the log.

Mar 11 09:32:52.389 CDT: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throt

tle mechanism: type 3, class 21, max_msg 8, total throttled 221

-Traceback= 8A4200 4B12F0 230BA0 230CE4 231740 231954 7599A8 753BEC

Mar 11 09:33:19.350 CDT: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throt

tle mechanism: type 37, class 14, max_msg 32, total throttled 1032

-Traceback= 8A4200 4B12F0 5E1064 5E5168 5D2C38 7599A8 753BEC

I found a few bugs matching above, so am thinking of upgrading to current 12.2.50 IOS version. I was able to get the entire stack back up by unplugging power to all switches and bringing them all up fresh. Then the one suspect switch came back fine also. So it looks like something in the IOS Software with the stack management was not functioning properly anymore? Anyone seen this behavior before? We had power cycled 2 of the bottom switches in stack connected to problem switch to try to see if that helped clear things up, and the 'show ver' output still showed those switches were up for months after they came back online which also points to something gone astray with the stack part of the IOS software.

Thanks

Sebastian Helmer · ‎03-12-2009

When too many outbound messages are in the queue for a message class, the Cisco Catalyst 3750 switch displays the %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type [int], class [int], max_msg [int], total throttled [int] error message. This error indicates that an RPC message is dropped. In the error message:

The first [int] represents the message type.

The second [int] represents the message class.

The third [int] indicates the maximum number of messages that the queue can accommodate before throttling occurs.

The last [int] is the total number of messages throttled.

The %PLATFORM_RPC-3-MSG_THROTTLED message appears while an attempt is made to create a routed interface when the snmp-server ifindex persist command is issued. Cisco bug ID CSCeg57839 identifies this issue.

In some scenarios, it causes the switch to report this error message every time these commands are issued:

write memory

copy running-config start-up config

These Cisco Catalyst switches do not support the snmp-server ifindex persist command:

3750

3560

2970

Remove the snmp-server ifindex persist command from the configuration.

Maybe this helps? I found it in the cisco wiki..

http://supportwiki.cisco.com/ViewWiki/index.php/The_%22PLATFORM_RPC-3-MSG_THROTTLED%22_error_message_appears_on_the_Cisco_Catalyst_3750_switch

best regards, Sebastian

net.administrator · ‎10-15-2009

I anm having the same/similar problem getting message:

Oct 15 09:22:35 BST: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 37, class 14, max_msg 32, total throttled 1260

I do not have this snmp command in my configuration.I have this snmp configuration:

snmp-server group SNMPc v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF7F

snmp-server group SNMPc v3 priv

snmp-server group SNMPcUser v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server system-shutdown

snmp-server host 192.168.245.20 version 3 auth SNMPcUser