On one of our Catalyst 4507R's, one of its VLAN interfaces went into shutdown mode apparently of its own accord over the weekend, which threw off a lot of IP routing. This switch is running IOS 12.2(25)EWA6. It has been up for about a year and a half, and has had no config changes in over a month. There is nothing in the logs that would indicate any errors that might have caused this.
Is this sort of thing very unusual? It's certainly the first time I've seen it. Does anyone have any ideas what might have caused this or whether there's anything I can do to further diagnose the issue?
Have never heard or seen anything like that where a SVI will go into a shutdown status . If it a single link in the vlan and you lost the link the SVI would go up/down I believe.
I have never seen anything that would put into a admin down state without manual intervention .
Ive never seen a vlan interface just shutdown for no reason. Do you mean it went administratively down? With no error messages, what makes you think that happened? How did you diagnose the problem?
Typically, a vlan interface will go "up,down" if there is no access port in that vlan that is in an "up" state.
Just to add to Glen's and Victor's posts.
A vlan interface goes down if there are no access ports in that vlan that are active on the switch or there are no trunk ports with that vlan active.
So if you don't have any servers/pcs etc. connected into your 4507R then do you have trunk links to other switches and is the vlan that shut itself down allowed on that trunk.
Three responses in 15 minutes, wow.
The problem became apparent when connectivity failed between devices in one of the VLANs and another. Both VLANs have multiple ports in that switch assigned to them, and both have a number of those ports containing devices that are always on.
We were chasing out tails for quite a while thinking the problem was in a PIX that sits in the VLAN-to-VLAN link, but when that proved to be innocent I looked more carefully at the switch. sh run int VLAN xxx had a "shutdown" in its config, so I guess that's "administratively down". I've seen VLANs come up in shutdown mode when they are first created, and I've seen L2 ports shut themselves after excessive errors, but I've never seen this with a VLAN interface.
I know for certain this was working correctly on Friday. I know for certain it was broken on Sunday morning. I'm trying to find out what could have happened in between.
Once again, I have to say that I have never seen or heard of a situtaion in which an interface will go into the "administratively down" state on its own. This has to be a result of manual intervention.
Are you sure that someone in your organization didnt shut it down? Are you running TACACS? If not, it can be hard to trace...
We are unfortunately not running TACACS. Before getting my baseball bat to go talk to people, I wanted to make sure there wasn't another possible answer.
Actually it does not have to be the result of manual intervention. I have seen an interface (a physical interface rather than a VLAN interface - but I believe that the principle is the same) go to shutdown because of errors on the interface. It was repeatable - I could no shut the interface and when the error occurred the interface would go back to shutdown/admin down.
It is very rare and I agree that I would ordinarily look for manual intervention in a situation like this. But I can say for sure that it is possible for certain types of interface errors to cause the interface to go to shutdown.
Perhaps so - and maybe justifiably so. If the switch has been running for a year and a half and then developed a problem it looks like something changed. And maybe the supervisor has some insight into what might have changed. I think the scope of questioning needs to be more broad than just "who entered the shutdown command on that switch interface".
I do not remember details of what our problem was (it was years ago) but I think we had a router on a connection and we brought up a new router on that connection and its interface was configured with something that very much conflicted with the interface configuration of the first router. And the router reacted to the error by putting its interface into shutdown.
I am certainly not saying that the switch just spontaneously put its VLAN interface into shutdown. I think that something changed and it is possible that the change was not on the local switch. And I could agree that someone manually entering the shutdown command is a more likely explanation of the problem. I just can not agree that it is the only explanation.
thanks again to all those who've taken the time to respond.
Simple question, which I'm not really in a position to test at the moment -- if someone were to enter a "shutdown" on an SVI, would that update the "Last configuration change" line that show up when you say sh run? Because on that particular switch, the date of the last config change when I started diagnosing this problem yesterday was Feb 6 2008.
From other logs I see that the problem began right around 8:00 am on Saturday, at a time when nobody with enable access to that switch was working. This, combined with the "last config change" date leads me to think this wasn't a manual screw-up. The only thing I see in this switch's logs from that time is the failure of the EIGRP neighbor relationships related to that SVI.
As far as I know if someone entered config mode and entered the shutdown command then I would certainly expect that the Last configuration change would reflect that time.
I guess that if someone changed the time on the switch before they made the config change then the Last configuration change time might not be correct. But that is getting to be a pretty elaborate set of circumstances.
As I indicate in my previous responses I believe that there is a realistic scenario in which the interface shutdown of your switch might be the result of some problem other than someone configuring your SVI with shutdown. Is it possible that there was some other type of change in the network at about 8 am on Saturday?
To shutdown the vlan interface, you would have to go into config mode.
If someone did that, it would be reflected in the last config change statement.
If they saved the config change, the NVRAM config last updated statement would reflect it.
By the way, you should really seriously look into a TACACS+ deployment. I don't know anything about your network, so you will have to discuss it with your management. But you should know that it is relatively cheap -- in terms of equipment and material -- to implement.
Meanwhile, you may look into configuring usernames and passwords for individual administrators. Again, I don't know anything about your network, so the feasibility or logic of this solution is something you would have to discuss with your management.
Please rate my posts if they have helped you
Turns out that at that time we switched over from the active to the standby Sup. Now I have to figure out why the active failed, and (more importantly) why it brought up this one VLAN in shutdown mode after the switchover.
BTW, we do have individual usernames and passwords, and a very limited set of people with 15-level access. No generic "enable" here. And TACACS+ is already a budgeted project this year, just not assigned at the moment.