3750X performance after one member reboots

[ieee1284c]

Scenario:

Pure 3750X-48PS ip base stack

One switch reboots from a power issue (possibly the master switch? Haven't confirmed yet in the 3 cases I've seen)

The rest of the stack stays running fine (as it should), the rebooted switch rejoins the stack just fine

What I've observed after this scenario takes place is that network throughput on any port in the entire stack is severely degraded.  Prior to the one switch rebooting, I can hit gigabit transfer rates going anywhere on any of the switches in the stack.  After one member reboots by itself, I typically hit a transfer wall around 5-20mbps in one direction and sometimes better going the other direction. I've only confirmed this on 3750X stacks and I can replicate it every time.  Newest code I've experienced it with is 12.2(58)SE1.  I have not seen confirmed reports of this on older 3750 (E, G, plain) stacks of which we have a ton of.

Resolution?  Rebooting the entire stack so that all 3750X members are coming up at the same time solves the transfer rate issue every time.  I would think there has to be a better way to get performance back to normal or prevent the performance hit in the first place.

[Leo Laohoo]

I believe this thread is very, very similar to the other one you've posted so I will refrain from responding to this one (to minimize confusion).

[ieee1284c]

I've experienced this with 3750X stacks as small as 2 switches.  Completely unrelated to my other post regarding large stacks of any 3750 flavor.

[ieee1284c]

Just ran into this issue again on a 3750X 12.2(55) SE3 stack where the local IT person accidentally power cycled one of the switches in the stack.  With one of the switches having booted at a different time from the others, cpu utilization during the day ran in the 80%+ range and traffic would not pass through the stack at anything greater than approx 15mbps.  sh proc cpu sorted did not indicate anything hogging resources, yet the cpu was always very high.  This was the case for almost a week and the problem was resolved by rebooting the stack in unison.  Now the cpu rides steady at 40% all day and all night.

I tried to recreate this in a lab environment a year ago, but failed.  I don't think my simulated traffic from a handful of sources was enough to recreate the problem.  I'm posting again in case anyone else experiencing this issue should happen to find this thread, please post your experience.  So far, I haven't been able to find any other discussions that line up with what I'm experiencing.