01-12-2010 06:27 AM - edited 03-04-2019 07:11 AM
Hi,
I have been informed by the server team that, their server got rebooted due to heartbit fail. Those servers are connected to Cisco3750Stack Switch.
I have check the switch log and observed, the switches has been crashed at the same time when server got rebooted.
But I am not able to understand the reason for crash the switch.
anyone can help me out on this. I have attached the log herewith.
Solved! Go to Solution.
01-12-2010 08:43 AM
Hello Goutam,
On you 3750 do a "sh flash" and post the entire crash file.
From the log files you attached it looks as you are having memory leak in your 3750 stack.
You should also open a TAC case and send them the crash file for analysis.
Here is the bug idea and the workaround for it:
This problem has been observed only when switches are stacked
When "ip routing" is disabled and master switch has been changed (master failure or change master priority/reload)
slow constant memory leak will happen in CEF IPC Background.
This happen only if "ip routing" is disabled
Workaround:
Enable ip routng and reload switch stack.
C3750 crashed suddenly with debug exception. |
SymptomCustomers using a Catalyst 3750 might experience an unexpected reload of the master switch of their stack of 3750's due to a Debug Exception (Could be NULL pointer dereference) Conditions: Conditions: So far this problem has been seen only on stacks of 3750 switches that have IP routing disabled. The up time of these switches has been around 46-52 weeks. Workaround:The root cause of this crash seems to be a slow memory leak resulting in the crash of the 3750 master switch when the memory has been exhausted. To verify this customer's are advised to monitor the memory usage on the switch. A possible cause for a memory leak when ip routing is disabled could be: CSCsc59027: mem leak in func. fib_memory_alloc_named_internal |
C3750 reloaded unexpectedly with Debug Exception |
Symptom: In the same customer's network, some C3750s crashed suddenly at almost the same time, around 43 weeks after installation. I got 2 types of decode results for these cases. There are a few similar defects in data base , but those results are not as same as this decode results . Conditions:- all crashed switches were masters in a stacked configuration that consisted of 1 master c3750 and a slave c3750 - all crashes occurred in a few days interval - other c3750s that were installed at the same time and run about the same coninguration are working properly - SNMP is enabled on all switches - HRPC process was w |
Memory leak in CEF IPC Backgrou after Master switchover with L2-image |
Symptom:Memory leak in CEF IPC Backgrou after Master switchover with L2-image. Conditions:[Topology] 3750G-24PS_A | stack | 3750G-24PS_B(Gig2/0/24)---Lab_backBone 3750G-24PS_A and _B is two box Stack. It is running c3750-ipbase-mz.122-25.SED1. The IP routing is disable. Workaround: The reload all Stack is stop memory leak. |
HTH
Reza
01-13-2010 01:56 PM
Leo,
Thanks for the rating!!
Goutam,
Here is the link to the Bug Toolkit
You have to have a CCO account to be able to access it.
The best course of action for you is to open a TAC case and send them the crash file to be analogized
http://www.cisco.com/pcgi-bin/Support/Bugtool/launch_bugtool.pl
HTH
Reza
01-13-2010 05:31 PM
I don't think he is hitting bug ID CSCsc59027, this bug is pretty old and should be resolved in the version that he is using.
However I do agree the author of the post should open a TAC and to troubleshoot further.
Regards,
jerry
01-12-2010 08:43 AM
Hello Goutam,
On you 3750 do a "sh flash" and post the entire crash file.
From the log files you attached it looks as you are having memory leak in your 3750 stack.
You should also open a TAC case and send them the crash file for analysis.
Here is the bug idea and the workaround for it:
This problem has been observed only when switches are stacked
When "ip routing" is disabled and master switch has been changed (master failure or change master priority/reload)
slow constant memory leak will happen in CEF IPC Background.
This happen only if "ip routing" is disabled
Workaround:
Enable ip routng and reload switch stack.
C3750 crashed suddenly with debug exception. |
SymptomCustomers using a Catalyst 3750 might experience an unexpected reload of the master switch of their stack of 3750's due to a Debug Exception (Could be NULL pointer dereference) Conditions: Conditions: So far this problem has been seen only on stacks of 3750 switches that have IP routing disabled. The up time of these switches has been around 46-52 weeks. Workaround:The root cause of this crash seems to be a slow memory leak resulting in the crash of the 3750 master switch when the memory has been exhausted. To verify this customer's are advised to monitor the memory usage on the switch. A possible cause for a memory leak when ip routing is disabled could be: CSCsc59027: mem leak in func. fib_memory_alloc_named_internal |
C3750 reloaded unexpectedly with Debug Exception |
Symptom: In the same customer's network, some C3750s crashed suddenly at almost the same time, around 43 weeks after installation. I got 2 types of decode results for these cases. There are a few similar defects in data base , but those results are not as same as this decode results . Conditions:- all crashed switches were masters in a stacked configuration that consisted of 1 master c3750 and a slave c3750 - all crashes occurred in a few days interval - other c3750s that were installed at the same time and run about the same coninguration are working properly - SNMP is enabled on all switches - HRPC process was w |
Memory leak in CEF IPC Backgrou after Master switchover with L2-image |
Symptom:Memory leak in CEF IPC Backgrou after Master switchover with L2-image. Conditions:[Topology] 3750G-24PS_A | stack | 3750G-24PS_B(Gig2/0/24)---Lab_backBone 3750G-24PS_A and _B is two box Stack. It is running c3750-ipbase-mz.122-25.SED1. The IP routing is disable. Workaround: The reload all Stack is stop memory leak. |
HTH
Reza
01-12-2010 11:28 PM
Hi,
This is sh flash output below:
ST-DC-SW-SVR-3750-6#sh flash
Directory of flash:/
2 -rwx 3864 Jan 11 2010 22:30:35 +03:00 vlan.dat
4 -rwx 109 Jan 12 2010 15:30:21 +03:00 private-config.text
5 -rwx 2072 Jan 12 2010 15:30:21 +03:00 multiple-fs
6 drwx 512 Mar 1 1993 03:31:08 +03:00 c3750e-universal-mz.122-50.SE3
429 drwx 512 Jan 11 2010 21:22:49 +03:00 crashinfo_ext
431 drwx 512 Mar 1 1993 03:00:08 +03:00 crashinfo
433 -rwx 15424 Jan 12 2010 15:30:21 +03:00 config.text
57671680 bytes total (29364224 bytes free)
This stack switch is running L2 mode and vlan routing is not required there, thats why ip routing is not enabled, do i need to enable that. will it be solve if I enable ip routing.
01-13-2010 07:34 AM
Hi
Can you download the crash file (431 drwx 512 Mar 1 1993 03:00:08 +03:00 crashinfo) and post it?
According the bug tool kit you should enable IP routing and see if that helps mitigate the crash issue.
Reza
01-13-2010 09:35 AM
01-13-2010 01:15 PM
Reza is right (+5). You may have a memory leak issue.
01-13-2010 01:56 PM
Leo,
Thanks for the rating!!
Goutam,
Here is the link to the Bug Toolkit
You have to have a CCO account to be able to access it.
The best course of action for you is to open a TAC case and send them the crash file to be analogized
http://www.cisco.com/pcgi-bin/Support/Bugtool/launch_bugtool.pl
HTH
Reza
01-13-2010 05:31 PM
I don't think he is hitting bug ID CSCsc59027, this bug is pretty old and should be resolved in the version that he is using.
However I do agree the author of the post should open a TAC and to troubleshoot further.
Regards,
jerry
08-29-2013 11:54 AM
12.2(58)SE2, this bug still exists
08-29-2013 07:12 PM
12.2(58)SE2, this bug still exists
I wouldn't come near 12.2(58)SE as it's very buggy.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide