Today I was totally flustered and confused when I saw the state of my switched network. I'm administering a military network with approx 50 users; we run W2K3 server with DNS, AD, and Exchange local to our subnet. We draw services from a fiber pedestal here.
Today our distant end went down (they wouldn't tell me how or why) and our network went down subsequently. We had no local L3 connectivity from any node to any node on our network however, we soon figured out that no computer, server, router or switch could ping any other node. All of our link lights on our switches went amber, and as far as I know we did nothing to provoke this. Our users are on vlan 3, and our uplink was on vlan 5, see diagram.
I consoled into our switch (and later our other switches) and find that Vlans 3 and 5 are down; not administratively down, but down. Keep in mind our network configuration has worked for some time. After an hour of needless tinkering (sorry, troubleshooting), I entered vlan database and forced the vlans to active, which I had to do for every switch. Approximately an hour later, distant end came up and we were golden.
Finally, my question. What could have caused this? Note that remote administration is not an option on this network, and it's highly unlikely someone consoled into our switches. Network diagram attached.
The layer3 Vlan interfaces depend on the layer2 Vlan to be on the vlan database. If that information is missing, the layer3 Vlan will remain down.
If it occurs again, I recommend checking your vlan information with the show vlan command.
If you are running vtp modes in default mode, all switches are running vtp server. If you haven't set a vtp domain name, the name is null and the switch with the highest revision number will rule what vlan other switches must have (if they are inter-switch links).
I highly suggest you implement some kind of vtp configuration to prevent this in the future. Adding a switch with a higher revision can easily wipe out the vlan configuration on the remaining switches and perhaps that's exactly what happened.
Please take a moment and read this link: