cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
987
Views
0
Helpful
6
Replies

Weird VTP/VLAN issue

Gataway24
Level 1
Level 1

Hello,

has anyone ever had an issue where adding a vlan to a core 6509 (VTP server) has caused OSPF adjacencies to flap, the vlan created does not have any host associated to it, it was just created,on the core switch and this appears to have caused an OSPF neighbor to go down along with 10 vlans. About 2 mins later all adjacencies came back online. Prior to this, the network had been stable the only recent change was the addition of a vlan,

the core switch is the only VTP server, all other switches are all in client mode and part of the same VTP Domain/password.

it might be a IOS,bug current IOS is (s3223_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(33)SXH3a

FYI

loggin history shows error below, after OSPF came back up, vlans started to come back up as well, link on opposite side, only shows OSPF neighbor droping,

TC: %OSPF-5-ADJCHG: Process 1, Nbr 72.13.xx.xx on Vlan900 from FULL to DOWN, Neighbor Down: Dead timer expired
*Jan  2 13:07:39.847 UTC: %OSPF-5-ADJCHG: Process 1, Nbr 72.13.xx.xx on Vlan900 from LOADING to FULL, Loading Done

any ideas or thoughts would be helpful

thank you

6 Replies 6

sachinraja
Level 9
Level 9

Hi gataway

It shouldnt bring down ospf adjacency for sure ! How are your Layer 2 networks setup here ? and how is layer 3 adjacency formed ? do you have trunks running from switches, and having a layer 3 svi just for ospf purpose ?

it clearly says that the ospf dead timeer is expired, which means there was a break in layer 3 between the switches in question. did you notice your trunks flapping or any other message on the switch when the ospf went down ? what were the logs on the other switch which had the adjacency built to ?

Regards

Raj

Hello Raj,

thank you for your reply,

none of my trunk interfaces flapped, the initial interface that came down was VLAN 900 which caused the cascade of flapping vlans.

i have trunk links with most of the swithces in my network, none of the other swithces reported issues, ( it may be that i was caught up with the core and overlooked the others)I am ussing SVI's for routing purposed,(OSPF) i have two 6500 directly connected, on the primary unit, the OSPF adjancey went down due to the dead timer, on the second unit, the loggin just shows my OSPF interface flapping going up and down several times.

this message repeats several times until the adjacency is formed (second 6509)

%OSPF-5-ADJCHG: Process 1, Nbr 72.13.xx.xx on Vlan900 from LOADING to FULL, Loading Done

coudl it be that VTP messages overwhelemed the interface and suppresed/dropped OSPF hello messages?

Carlos

Hello Carlos,

you should verify if STP events happened in your campus, routing protocol messages are usually a symptom of problems at underlaying  OSI layer2.

By the way take the following

sh proc cpu history

sh proc cpu sorted 1min

there is a remote but not zero probability that the system was too busy as you suspect, but if so you should see clearly peaks of 100% in sh proc cpu hist for the time you added the new vlan.

Hope to help

Giuseppe

HI carlos

is vlan 900 the only vlan having ospf neighbor to the switch ? are there other vlans which share the trunk whch are on ospf neighbor table ? if so, did those vlans flap too ? now, in your case, do you have two core switches, and a layer 2 trunk between them, with layer 3 svi for ospf ? I really dont know if vtp messages will flood so much as to bringing down ospf neighbors.. that too, till the dead time is expired  (40 secs) ! i seriously think there this was just a co-incidence... there could have been other issues like spanning tree loops, or interface flaps which could have led to ospf flapping ! did you notice anything on your logs apart from the ospf messages ? anything on layer 2 ?

Raj

Hello

thank you guys for you responses

VLAN 900 was the only OSPF interface affected, the other vlans that bounced were also part of that trunk link.CPU utilization is at about 8%, not high by any means, I cant think of anything else that could have caused this issue. I'm running R-PVST, there has not been a new switch introduced which could have caused a SPT issue, no other interfaces, physical or logical were affected, other than the ones previously mentioned. the addition of the VLAN and OSPF bouncing are probably all coincidental, i just wanted a new set of eyes looking at this issue as and give their perspective

im going to try to an IOS upgrade to rule out any bugs.

thank you for you help

Carlos

Ryan Carretta
Cisco Employee
Cisco Employee

This sounds like it could be a hit of CSCsv21612.

Is VTP pruning enabled. Do you see high SP CPU utilization after the configuration in the PM Callback process? You can view SP CPU utilization with 'remote command switch show process cpu'.

This bug is fixed as of 12.2(33)SXI1, so if you can freely upgrade this box you may want to try that.

-Ryan

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco