cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3685
Views
5
Helpful
9
Replies

Cisco3750Crash

goutam_04
Level 1
Level 1

Hi,

I have been informed by the server team that, their server got rebooted due to heartbit fail. Those servers are connected to Cisco3750Stack Switch.

I have check the switch log and observed, the switches has been crashed at the same time when server got rebooted.

But I am not able to understand the reason for crash the switch.

anyone can help me out on this.  I have attached the log herewith.

3 Accepted Solutions

Accepted Solutions

Reza Sharifi
Hall of Fame
Hall of Fame

Hello Goutam,

On you 3750 do a "sh flash" and post the entire crash file.

From the log files you attached it looks as you are having memory leak in your 3750 stack.

You should also open a TAC case and send them the crash file for analysis.

Here is the bug idea and the workaround for it:

This problem has been observed only when switches are stacked
When "ip routing" is disabled and master switch has been changed (master failure or change master priority/reload)
slow constant memory leak will happen in CEF IPC Background.
This happen only if "ip routing" is disabled

Workaround:

Enable ip routng and reload switch stack.

C3750 crashed suddenly with debug exception.
SymptomCustomers using a Catalyst 3750 might experience an unexpected reload of the master switch of their stack of 3750's due to a Debug Exception (Could be NULL pointer dereference) Conditions: Conditions: So far this problem has been seen only on stacks of 3750 switches that have IP routing disabled. The up time of these switches has been around 46-52 weeks. Workaround:The root cause of this crash seems to be a slow memory leak resulting in the crash of the 3750 master switch when the memory has been exhausted. To verify this customer's are advised to monitor the memory usage on the switch. A possible cause for a memory leak when ip routing is disabled could be: CSCsc59027: mem leak in func. fib_memory_alloc_named_internal
C3750 reloaded unexpectedly with Debug Exception
Symptom: In the same customer's network, some C3750s crashed suddenly at almost the same time, around 43 weeks after installation. I got 2 types of decode results for these cases. There are a few similar defects in data base , but those results are not as same as this decode results . Conditions:- all crashed switches were masters in a stacked configuration that consisted of 1 master c3750 and a slave c3750 - all crashes occurred in a few days interval - other c3750s that were installed at the same time and run about the same coninguration are working properly - SNMP is enabled on all switches - HRPC process was w
Memory leak in CEF IPC Backgrou after Master switchover with L2-image
Symptom:Memory leak in CEF IPC Backgrou after Master switchover with L2-image. Conditions:[Topology] 3750G-24PS_A | stack | 3750G-24PS_B(Gig2/0/24)---Lab_backBone 3750G-24PS_A and _B is two box Stack. It is running c3750-ipbase-mz.122-25.SED1. The IP routing is disable. Workaround: The reload all Stack is stop memory leak.


HTH

Reza

View solution in original post

Leo,

Thanks for the rating!!

Goutam,

Here is the link to the Bug Toolkit

You have to have a CCO account to be able to access it.

The best course of action for you is to open a TAC case and send them the crash file to be analogized

http://www.cisco.com/pcgi-bin/Support/Bugtool/launch_bugtool.pl

HTH

Reza

View solution in original post

I don't think he is hitting bug ID CSCsc59027, this bug is pretty old and should be resolved in the version that he is using.

However I do agree the author of the post should open a TAC and to troubleshoot further.

Regards,

jerry

View solution in original post

9 Replies 9

Reza Sharifi
Hall of Fame
Hall of Fame

Hello Goutam,

On you 3750 do a "sh flash" and post the entire crash file.

From the log files you attached it looks as you are having memory leak in your 3750 stack.

You should also open a TAC case and send them the crash file for analysis.

Here is the bug idea and the workaround for it:

This problem has been observed only when switches are stacked
When "ip routing" is disabled and master switch has been changed (master failure or change master priority/reload)
slow constant memory leak will happen in CEF IPC Background.
This happen only if "ip routing" is disabled

Workaround:

Enable ip routng and reload switch stack.

C3750 crashed suddenly with debug exception.
SymptomCustomers using a Catalyst 3750 might experience an unexpected reload of the master switch of their stack of 3750's due to a Debug Exception (Could be NULL pointer dereference) Conditions: Conditions: So far this problem has been seen only on stacks of 3750 switches that have IP routing disabled. The up time of these switches has been around 46-52 weeks. Workaround:The root cause of this crash seems to be a slow memory leak resulting in the crash of the 3750 master switch when the memory has been exhausted. To verify this customer's are advised to monitor the memory usage on the switch. A possible cause for a memory leak when ip routing is disabled could be: CSCsc59027: mem leak in func. fib_memory_alloc_named_internal
C3750 reloaded unexpectedly with Debug Exception
Symptom: In the same customer's network, some C3750s crashed suddenly at almost the same time, around 43 weeks after installation. I got 2 types of decode results for these cases. There are a few similar defects in data base , but those results are not as same as this decode results . Conditions:- all crashed switches were masters in a stacked configuration that consisted of 1 master c3750 and a slave c3750 - all crashes occurred in a few days interval - other c3750s that were installed at the same time and run about the same coninguration are working properly - SNMP is enabled on all switches - HRPC process was w
Memory leak in CEF IPC Backgrou after Master switchover with L2-image
Symptom:Memory leak in CEF IPC Backgrou after Master switchover with L2-image. Conditions:[Topology] 3750G-24PS_A | stack | 3750G-24PS_B(Gig2/0/24)---Lab_backBone 3750G-24PS_A and _B is two box Stack. It is running c3750-ipbase-mz.122-25.SED1. The IP routing is disable. Workaround: The reload all Stack is stop memory leak.


HTH

Reza

Hi,

This is sh flash output below:

ST-DC-SW-SVR-3750-6#sh flash

Directory of flash:/

    2  -rwx        3864  Jan 11 2010 22:30:35 +03:00  vlan.dat
    4  -rwx         109  Jan 12 2010 15:30:21 +03:00  private-config.text
    5  -rwx        2072  Jan 12 2010 15:30:21 +03:00  multiple-fs
    6  drwx         512   Mar 1 1993 03:31:08 +03:00  c3750e-universal-mz.122-50.SE3
  429  drwx         512  Jan 11 2010 21:22:49 +03:00  crashinfo_ext
  431  drwx         512   Mar 1 1993 03:00:08 +03:00  crashinfo
  433  -rwx       15424  Jan 12 2010 15:30:21 +03:00  config.text

57671680 bytes total (29364224 bytes free)

This  stack switch is running L2 mode and vlan routing is not required there, thats why ip routing is not enabled, do i need to enable that.  will it be solve if I enable ip routing.

Hi

Can you download the crash file (431  drwx  512   Mar 1 1993 03:00:08 +03:00  crashinfo) and post it?

According the bug tool kit you should enable IP routing and see if that helps mitigate the crash issue.

Reza

Hi,

Please find the crashinfo file attached.  Can you please tell, what is the relation of this command to crash the system.  How it will solve this problem.

Do you have the bug tool kit.

Reza is right (+5).  You may have a memory leak issue.

Leo,

Thanks for the rating!!

Goutam,

Here is the link to the Bug Toolkit

You have to have a CCO account to be able to access it.

The best course of action for you is to open a TAC case and send them the crash file to be analogized

http://www.cisco.com/pcgi-bin/Support/Bugtool/launch_bugtool.pl

HTH

Reza

I don't think he is hitting bug ID CSCsc59027, this bug is pretty old and should be resolved in the version that he is using.

However I do agree the author of the post should open a TAC and to troubleshoot further.

Regards,

jerry

Roman_Shilenko
Level 1
Level 1

12.2(58)SE2, this bug still exists

12.2(58)SE2, this bug still exists

I wouldn't come near 12.2(58)SE as it's very buggy.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: