cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 
cancel
16100
Views
0
Helpful
17
Replies

Catalyst 3750 stack members removed unexpectedly

Hi all,

Today we had two 3750 switches remove from a core switch stack of eight 3750 switches. There was no loss of power or changes made. The other switches continued to work. The message was

040667: Jan  5 09:59:43.891 AEDT: %STACKMGR-6-SWITCH_REMOVED: Switch 7 has been REMOVED from the stack
040668: Jan  5 09:59:44.168 AEDT: %STACKMGR-6-SWITCH_REMOVED: Switch 8 has been REMOVED from the stack

The version is

Cisco IOS Software, C3750 Software (C3750-IPSERVICES-M), Version 12.2(25)SEC.

I have been unable to find any likely cause. Does anyone have an idea for a cause?

Thanks

Peter

17 Replies 17

Leo Laohoo
Hall of Fame
Hall of Fame

How long have the entire stack been up?  Have you tried rebooting the 2 switches?

The master had been up for over two years. The stack has been rebooted and is working normally.

Not really a big fan of something running a very old IOS to be running more than 1 year without a reboot.

sljacobs2015
Level 1
Level 1

Peter,

I had the same problem as well, only with a newer IOS version.  This occurred on Jan 2nd and I have not been able to find a cause as well.  I would welcome any thoughts on this as well.  This stack was installed 28 weeks ago and has had no problems until the 2nd of January.  Switch 1 restarted after this and has been fine since.

1103: 000037:   Switch 1 has been REMOVED from the stack
1102: 000024:   Switch 1 has been REMOVED from the stack
1118: 001015:   Interface GigabitEthernet1/0/12, changed state to down
1117: 001014:   Interface GigabitEthernet1/0/11, changed state to down
1116: 001013:   Interface GigabitEthernet1/0/10, changed state to down
1115: 001012:   Interface GigabitEthernet1/0/9, changed state to down
1114: 001011:   Interface GigabitEthernet1/0/8, changed state to down
1110: 001007:   Interface GigabitEthernet1/0/5, changed state to down
1099: 001000:   Switch 1 has been REMOVED from the stack
1098: 000999:   Stack Port 1 Switch 6 has changed to state DOWN
1097: 000998:   Stack Port 2 Switch 2 has changed to state DOWN

     1 12    WS-C3750G-12S      12.2(50)SE1           C3750-IPSERVICESK9-M
*    2 12    WS-C3750G-12S      12.2(50)SE1           C3750-IPSERVICESK9-M
     3 12    WS-C3750G-12S      12.2(50)SE1           C3750-IPSERVICESK9-M
     4 12    WS-C3750G-12S      12.2(50)SE1           C3750-IPSERVICESK9-M
     5 12    WS-C3750G-12S      12.2(50)SE1           C3750-IPSERVICESK9-M
     6 12    WS-C3750G-12S      12.2(50)SE1           C3750-IPSERVICESK9-M

Thank you!

Sara

francisco_1
Level 7
Level 7

Looks like a bug related issue.

There was a bug with a recent code release  where you could get random reboots if you did a "show cdp neighbors".
looking at Version 12.2(25)SE  is actually now deferred - boot issues and sh cdp neigh causes crash.

Version 12.2(25)SE Reason for Deferral:

CSCeh45368
Headline: Switch powered down/up in 3750 stack stuck at Initializing state


1st Found-In
12.2(25)SEA

Fixed-In
12.2(25)SEB1
12.2(25)SEC

CSCsa78000
Symptom:
A Catalyst switch running Cisco IOS release 12.2(25)SG, 12.2(25)SEA, 12.2(25)SEB, 12.2(25)EZ,
or 12.2(25)EY might reload when show cdp neighbor detail is entered.


1st Found-In
12.2(25)EY
12.2(25)SEA

Fixed-In
12.2(25)EY1
12.2(25)SEB1
12.2(25)SEC
12.2(31)SG

hope that helps

The IOS on the switches is 12.2(25)SEC which the bugs were fixed in.

No one was on the switches at the time, so no commands were being issued. The switches didn't reboot and weren't initialising. The switches just dropped out of the stack. The stacking configuration is such that the communications between switches 1,2,3 and 4,5,6 had to continue through the two removed switch stacking ports.

What did show version said, software reboot? Any crashdump file in the rebooted switch's flash?

Regards,

jerry

Hi,

The show version shows System return to ROM by power-on. We power off the entire stack to get it working again. There are crashdump files on the two switches that dropped out of the stack.

Peter,

I thought i saw 12.2(25)SE in your original post.

Are you using 12.2(25)SEC on your all switches in the stack?

Francisco.

Yes all switches are running Version 12.2(25)SEC

Switch   Ports  Model              SW Version              SW Image           
------   -----  -----              ----------              ----------         
*    1   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M 
     2   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M 
     3   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M 
     4   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M 
     5   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M 
     6   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M 
     7   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M 
     8   28     WS-C3750G-24TS     12.2(25)SEC             C3750-IPSERVICES-M

Hi Peter,

Prior to rebooting the switches did you happen to notice if the cpu was high on the switches that dropped out of the stack. Just another thought as quite often I have seen the CPU on 3750 switches increase for no apparent reason. Just out of curiosity what SDM profile are you running on these switches. Thx

HI Peter,

If you happen to have the crashdump files, can you post them.

Regards,

jerry

attached is the crashinfo file

I don't see anything funny here Peter.  My suspicion lies in the fact that the manager has been up for more than 2 years with a very ancient IOS.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: