Cisco3750Crash

Answered Question
Jan 12th, 2010
User Badges:

Hi,


I have been informed by the server team that, their server got rebooted due to heartbit fail. Those servers are connected to Cisco3750Stack Switch.


I have check the switch log and observed, the switches has been crashed at the same time when server got rebooted.


But I am not able to understand the reason for crash the switch.


anyone can help me out on this.  I have attached the log herewith.

Correct Answer by Jerry Ye about 7 years 4 months ago

I don't think he is hitting bug ID CSCsc59027, this bug is pretty old and should be resolved in the version that he is using.


However I do agree the author of the post should open a TAC and to troubleshoot further.


Regards,

jerry

Correct Answer by Reza Sharifi about 7 years 4 months ago

Leo,


Thanks for the rating!!


Goutam,


Here is the link to the Bug Toolkit

You have to have a CCO account to be able to access it.

The best course of action for you is to open a TAC case and send them the crash file to be analogized


http://www.cisco.com/pcgi-bin/Support/Bugtool/launch_bugtool.pl



HTH

Reza

Correct Answer by Reza Sharifi about 7 years 4 months ago

Hello Goutam,


On you 3750 do a "sh flash" and post the entire crash file.

From the log files you attached it looks as you are having memory leak in your 3750 stack.


You should also open a TAC case and send them the crash file for analysis.

Here is the bug idea and the workaround for it:



This problem has been observed only when switches are stacked
When "ip routing" is disabled and master switch has been changed (master failure or change master priority/reload)
slow constant memory leak will happen in CEF IPC Background.
This happen only if "ip routing" is disabled


Workaround:


Enable ip routng and reload switch stack.

C3750 crashed suddenly with debug exception.
SymptomCustomers using a Catalyst 3750 might experience an unexpected reload of the master switch of their stack of 3750's due to a Debug Exception (Could be NULL pointer dereference) Conditions: Conditions: So far this problem has been seen only on stacks of 3750 switches that have IP routing disabled. The up time of these switches has been around 46-52 weeks. Workaround:The root cause of this crash seems to be a slow memory leak resulting in the crash of the 3750 master switch when the memory has been exhausted. To verify this customer's are advised to monitor the memory usage on the switch. A possible cause for a memory leak when ip routing is disabled could be: CSCsc59027: mem leak in func. fib_memory_alloc_named_internal
C3750 reloaded unexpectedly with Debug Exception
Symptom: In the same customer's network, some C3750s crashed suddenly at almost the same time, around 43 weeks after installation. I got 2 types of decode results for these cases. There are a few similar defects in data base , but those results are not as same as this decode results . Conditions:- all crashed switches were masters in a stacked configuration that consisted of 1 master c3750 and a slave c3750 - all crashes occurred in a few days interval - other c3750s that were installed at the same time and run about the same coninguration are working properly - SNMP is enabled on all switches - HRPC process was w
Memory leak in CEF IPC Backgrou after Master switchover with L2-image
Symptom:Memory leak in CEF IPC Backgrou after Master switchover with L2-image. Conditions:[Topology] 3750G-24PS_A | stack | 3750G-24PS_B(Gig2/0/24)---Lab_backBone 3750G-24PS_A and _B is two box Stack. It is running c3750-ipbase-mz.122-25.SED1. The IP routing is disable. Workaround: The reload all Stack is stop memory leak.


HTH

Reza

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (4 ratings)
Loading.
Correct Answer
Reza Sharifi Tue, 01/12/2010 - 08:43
User Badges:
  • Super Bronze, 10000 points or more
  • Cisco Designated VIP,

    2017 LAN

Hello Goutam,


On you 3750 do a "sh flash" and post the entire crash file.

From the log files you attached it looks as you are having memory leak in your 3750 stack.


You should also open a TAC case and send them the crash file for analysis.

Here is the bug idea and the workaround for it:



This problem has been observed only when switches are stacked
When "ip routing" is disabled and master switch has been changed (master failure or change master priority/reload)
slow constant memory leak will happen in CEF IPC Background.
This happen only if "ip routing" is disabled


Workaround:


Enable ip routng and reload switch stack.

C3750 crashed suddenly with debug exception.
SymptomCustomers using a Catalyst 3750 might experience an unexpected reload of the master switch of their stack of 3750's due to a Debug Exception (Could be NULL pointer dereference) Conditions: Conditions: So far this problem has been seen only on stacks of 3750 switches that have IP routing disabled. The up time of these switches has been around 46-52 weeks. Workaround:The root cause of this crash seems to be a slow memory leak resulting in the crash of the 3750 master switch when the memory has been exhausted. To verify this customer's are advised to monitor the memory usage on the switch. A possible cause for a memory leak when ip routing is disabled could be: CSCsc59027: mem leak in func. fib_memory_alloc_named_internal
C3750 reloaded unexpectedly with Debug Exception
Symptom: In the same customer's network, some C3750s crashed suddenly at almost the same time, around 43 weeks after installation. I got 2 types of decode results for these cases. There are a few similar defects in data base , but those results are not as same as this decode results . Conditions:- all crashed switches were masters in a stacked configuration that consisted of 1 master c3750 and a slave c3750 - all crashes occurred in a few days interval - other c3750s that were installed at the same time and run about the same coninguration are working properly - SNMP is enabled on all switches - HRPC process was w
Memory leak in CEF IPC Backgrou after Master switchover with L2-image
Symptom:Memory leak in CEF IPC Backgrou after Master switchover with L2-image. Conditions:[Topology] 3750G-24PS_A | stack | 3750G-24PS_B(Gig2/0/24)---Lab_backBone 3750G-24PS_A and _B is two box Stack. It is running c3750-ipbase-mz.122-25.SED1. The IP routing is disable. Workaround: The reload all Stack is stop memory leak.


HTH

Reza

goutam_04 Tue, 01/12/2010 - 23:28
User Badges:

Hi,


This is sh flash output below:


ST-DC-SW-SVR-3750-6#sh flash

Directory of flash:/

    2  -rwx        3864  Jan 11 2010 22:30:35 +03:00  vlan.dat
    4  -rwx         109  Jan 12 2010 15:30:21 +03:00  private-config.text
    5  -rwx        2072  Jan 12 2010 15:30:21 +03:00  multiple-fs
    6  drwx         512   Mar 1 1993 03:31:08 +03:00  c3750e-universal-mz.122-50.SE3
  429  drwx         512  Jan 11 2010 21:22:49 +03:00  crashinfo_ext
  431  drwx         512   Mar 1 1993 03:00:08 +03:00  crashinfo
  433  -rwx       15424  Jan 12 2010 15:30:21 +03:00  config.text

57671680 bytes total (29364224 bytes free)


This  stack switch is running L2 mode and vlan routing is not required there, thats why ip routing is not enabled, do i need to enable that.  will it be solve if I enable ip routing.

Reza Sharifi Wed, 01/13/2010 - 07:34
User Badges:
  • Super Bronze, 10000 points or more
  • Cisco Designated VIP,

    2017 LAN

Hi

Can you download the crash file (431  drwx  512   Mar 1 1993 03:00:08 +03:00  crashinfo) and post it?


According the bug tool kit you should enable IP routing and see if that helps mitigate the crash issue.


Reza

goutam_04 Wed, 01/13/2010 - 09:35
User Badges:

Hi,


Please find the crashinfo file attached.  Can you please tell, what is the relation of this command to crash the system.  How it will solve this problem.


Do you have the bug tool kit.

Leo Laohoo Wed, 01/13/2010 - 13:15
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Reza is right (+5).  You may have a memory leak issue.

Correct Answer
Reza Sharifi Wed, 01/13/2010 - 13:56
User Badges:
  • Super Bronze, 10000 points or more
  • Cisco Designated VIP,

    2017 LAN

Leo,


Thanks for the rating!!


Goutam,


Here is the link to the Bug Toolkit

You have to have a CCO account to be able to access it.

The best course of action for you is to open a TAC case and send them the crash file to be analogized


http://www.cisco.com/pcgi-bin/Support/Bugtool/launch_bugtool.pl



HTH

Reza

Correct Answer
Jerry Ye Wed, 01/13/2010 - 17:31
User Badges:
  • Cisco Employee,

I don't think he is hitting bug ID CSCsc59027, this bug is pretty old and should be resolved in the version that he is using.


However I do agree the author of the post should open a TAC and to troubleshoot further.


Regards,

jerry

Leo Laohoo Thu, 08/29/2013 - 19:12
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

12.2(58)SE2, this bug still exists

I wouldn't come near 12.2(58)SE as it's very buggy.

Actions

This Discussion