01-05-2010 12:01 PM - edited 03-06-2019 09:09 AM
Hello,
has anyone ever had an issue where adding a vlan to a core 6509 (VTP server) has caused OSPF adjacencies to flap, the vlan created does not have any host associated to it, it was just created,on the core switch and this appears to have caused an OSPF neighbor to go down along with 10 vlans. About 2 mins later all adjacencies came back online. Prior to this, the network had been stable the only recent change was the addition of a vlan,
the core switch is the only VTP server, all other switches are all in client mode and part of the same VTP Domain/password.
it might be a IOS,bug current IOS is (s3223_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(33)SXH3a
FYI
loggin history shows error below, after OSPF came back up, vlans started to come back up as well, link on opposite side, only shows OSPF neighbor droping,
TC: %OSPF-5-ADJCHG: Process 1, Nbr 72.13.xx.xx on Vlan900 from FULL to DOWN, Neighbor Down: Dead timer expired
*Jan 2 13:07:39.847 UTC: %OSPF-5-ADJCHG: Process 1, Nbr 72.13.xx.xx on Vlan900 from LOADING to FULL, Loading Done
any ideas or thoughts would be helpful
thank you
01-05-2010 12:10 PM
Hi gataway
It shouldnt bring down ospf adjacency for sure ! How are your Layer 2 networks setup here ? and how is layer 3 adjacency formed ? do you have trunks running from switches, and having a layer 3 svi just for ospf purpose ?
it clearly says that the ospf dead timeer is expired, which means there was a break in layer 3 between the switches in question. did you notice your trunks flapping or any other message on the switch when the ospf went down ? what were the logs on the other switch which had the adjacency built to ?
Regards
Raj
01-05-2010 12:29 PM
Hello Raj,
thank you for your reply,
none of my trunk interfaces flapped, the initial interface that came down was VLAN 900 which caused the cascade of flapping vlans.
i have trunk links with most of the swithces in my network, none of the other swithces reported issues, ( it may be that i was caught up with the core and overlooked the others)I am ussing SVI's for routing purposed,(OSPF) i have two 6500 directly connected, on the primary unit, the OSPF adjancey went down due to the dead timer, on the second unit, the loggin just shows my OSPF interface flapping going up and down several times.
this message repeats several times until the adjacency is formed (second 6509)
%OSPF-5-ADJCHG: Process 1, Nbr 72.13.xx.xx on Vlan900 from LOADING to FULL, Loading Done
coudl it be that VTP messages overwhelemed the interface and suppresed/dropped OSPF hello messages?
Carlos
01-05-2010 12:50 PM
Hello Carlos,
you should verify if STP events happened in your campus, routing protocol messages are usually a symptom of problems at underlaying OSI layer2.
By the way take the following
sh proc cpu history
sh proc cpu sorted 1min
there is a remote but not zero probability that the system was too busy as you suspect, but if so you should see clearly peaks of 100% in sh proc cpu hist for the time you added the new vlan.
Hope to help
Giuseppe
01-05-2010 12:54 PM
HI carlos
is vlan 900 the only vlan having ospf neighbor to the switch ? are there other vlans which share the trunk whch are on ospf neighbor table ? if so, did those vlans flap too ? now, in your case, do you have two core switches, and a layer 2 trunk between them, with layer 3 svi for ospf ? I really dont know if vtp messages will flood so much as to bringing down ospf neighbors.. that too, till the dead time is expired (40 secs) ! i seriously think there this was just a co-incidence... there could have been other issues like spanning tree loops, or interface flaps which could have led to ospf flapping ! did you notice anything on your logs apart from the ospf messages ? anything on layer 2 ?
Raj
01-05-2010 02:34 PM
Hello
thank you guys for you responses
VLAN 900 was the only OSPF interface affected, the other vlans that bounced were also part of that trunk link.CPU utilization is at about 8%, not high by any means, I cant think of anything else that could have caused this issue. I'm running R-PVST, there has not been a new switch introduced which could have caused a SPT issue, no other interfaces, physical or logical were affected, other than the ones previously mentioned. the addition of the VLAN and OSPF bouncing are probably all coincidental, i just wanted a new set of eyes looking at this issue as and give their perspectiveim going to try to an IOS upgrade to rule out any bugs.
thank you for you help
Carlos
01-05-2010 05:20 PM
This sounds like it could be a hit of CSCsv21612.
Is VTP pruning enabled. Do you see high SP CPU utilization after the configuration in the PM Callback process? You can view SP CPU utilization with 'remote command switch show process cpu'.
This bug is fixed as of 12.2(33)SXI1, so if you can freely upgrade this box you may want to try that.
-Ryan
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: