OSPF stuck in Exchange/DR

Answered Question
Feb 23rd, 2009
User Badges:

We have a point to point 100MG connection between a 6509 gig interface and a 3750G-48TS Gig interface.


Its a basic point o point but the OSPF fails to load and gets stuck in Exchange/DR as below.


The circuit has been tested by the provider and they have said that there are no issues and MTU has been set correctly to 1500.


Has anyone come across this problem before and how we can over come this ?




Neighbor ID Pri State Dead Time Address Interface

10.137.243.1 1 EXCHANGE/DR 00:00:14 10.128.0.173




Correct Answer by Giuseppe Larosa about 8 years 3 months ago

Hello Jaya,


by setting a lower ip mtu on the C6509 side you can be sure that is not trying to build a packet that is bigger then 1500.


mtu-ignore disable a check but doesn't influence the size of packets that are sent out the interface


you may combine

ip ospf mtu-ignore on 3750 side

with

ip ospf mtu-ignore

ip mtu 1400

on c6509 and see what happens


Hope to help

Giuseppe


  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.5 (2 ratings)
Loading.
Giuseppe Larosa Mon, 02/23/2009 - 03:28
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,

try to use

debug ip ospf adj


on your side to see what is happening.


Being the C6509 a multi layer switch it can have different ip settings at physical interface level and at the SVI (interface vlan) if the port is not a routed port.


A MTU mismatch is probably the reason.


Also check the ospf network type with


sh ip ospf interface


have you used

ip ospf network point-to-point on your side or on the provider side they used it ?


feel to free to post debug outputs if public ip addresses are not involved


Hope to help

Giuseppe




Giuseppe Larosa Mon, 02/23/2009 - 05:47
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,

to be able to distingush from different possible problems I would suggest to remove OSPF authentication configuration.


From the debugs we see that the C3750 is elected DR and tries to send first Database Descriptor packet but it never receives an ack from the C6500.

So after some attempts 14 the neighbor state is restarted.


02.554 GMT: OSPF: Send DBD to 10.134.243.2 on GigabitEthernet1/0/2 seq 0x17DA opt 0x52 flag 0x7 len 32

001170: *Mar 1 18:22:02.554 GMT: OSPF: Send with youngest Key 1

001171: *Mar 1 18:22:02.554 GMT: OSPF: Retransmitting DBD to 10.134.243.2 on GigabitEthernet1/0/2 [11]

001172: *Mar 1 18:22:03.141 GMT: OSPF: Send with youngest Key 1

001173: *Mar 1 18:22:03.477 GMT: OSPF: rcv. v:2 t:1 l:48 rid:10.134.243.2

aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x4980537F from GigabitEthernet1/0/2



I think you need to verify if NTP in sync in the second device :

the second device thinks it March 1

the c6500 thinks it is Feb 23


this can make them to use a different key.


then also enable datetime in logging

on C3750 with

service timestamps log datetime msec

service timestamps deb datetime msec


So disable the authentication and see what happens


Hope to help

Giuseppe



mistryj Tue, 02/24/2009 - 08:41
User Badges:

Hi Giuseppe,


The clock was set manually on the 3750 and removed ip ospf authentication message-digest and message-digest key.


But still no change.


The ospf works if you put another 3750 instead of using the 6509, why is that ?





glen.grant Tue, 02/24/2009 - 15:42
User Badges:
  • Purple, 4500 points or more

Also check to make sure your network statements are correct on each side with the correct masking .

pkurdziel Wed, 02/25/2009 - 16:24
User Badges:

"The ospf works if you put another 3750 instead of using the 6509, why is that?"


What is the difference between the two configurations?




eric.follos Mon, 03/23/2015 - 14:00
User Badges:

We had two 6509s back to back with interfaces set at 9216 mtu.  Then replaced one with a 3600 and ospf wouldn't pass Exchange.  Used the ip ospf mtu-ignore on the 3600 as a temp workaround.  OSPF came up.  Found in our scenario the DWDM transport in the middle was the culprit.  Transport interfaces in the middle of the circuit were at the default, max mtu 1600.  I'll be honest, it took me a day to look there.

Initially 6500s on either end were set to MTU 9216, fat dumb and happy.  Perhaps the 3600 was doing something the 6500 wasnt, like sending DBD packets larger than 1600bytes and the 6509 wasn't getting them.  After adjusting the DWDM to support max frame size of 9600 I took the ip ospf mtu-ignore off the interface and it still worked fine.

pkurdziel Tue, 02/24/2009 - 19:43
User Badges:

Hi Jaja,


Did you remove authentication on both the 6500 and the 3750?


Looking at the show ip ospf int config both the 6500 and 3750 think its the DR.

On one of the interfaces add ip ospf priority 255 and on the other ip ospf priority 0.


ip ospf priority 255 - higher # is preferred. - Always become the DR.

ip ospf priority 0 - never become the DR


Let me know of this works for you.

CSCO11188854 Tue, 02/24/2009 - 17:29
User Badges:

Try on the 6509 interface facing the 3750


switch-if#ip ospf mtu-ignore


see how that goes.


mistryj Wed, 02/25/2009 - 02:46
User Badges:

Thank you all for you replies unfortunately no change tried all the following suggestions :-


1. The Network statements have correct subnet masks.


2. I have tried removing the following authentication but no change.


ip ospf authentication message-digest

ip ospf message-digest-key 1 md5 7 050F5501351C40


3. I set 6509 interface to priority to 255 and 3750 to priority to 0. Still no change its now stuck in Exchange/DROTHER.


4. I have 'ip ospf mtu-ignore' set on both interfaces but no change same results.





Giuseppe Larosa Wed, 02/25/2009 - 09:37
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,

after removing the authentication and changing the ip ospf priority the problem is still there.


I would suggest you to do the following:


Do

sh system mtu

on the C3750


a) remove ip ospf mtu-ignore on both sides

apply

ip mtu 1500


b) collect output of debug ip ospf adj on both sides


if you have changed the system MTU on c3750 there can be a side effect on SVIs and size of DBD packets.

I now remember I had problems with a C3550 to build OSPF adjacency with routers.


Hope to help

Giuseppe




pkurdziel Wed, 02/25/2009 - 16:22
User Badges:

Giuseppe,


With ip ospf mtu-ignore then the MTU is ignored so it does not matter if the MTU on one side is different. I've used this successfully where the switch MTU was 1504 and the router MTU was 1500.


What issues have you run into using ospf mtu-ignore?


Jaya for troubleshooting please remove it as Giuseppe suggests.

pkurdziel Wed, 02/25/2009 - 16:26
User Badges:

Please paste the output of the following commands from both ends.


show ip ospf


show ip ospf neighbor


debug ip ospf adj


debug ip ospf events




mistryj Thu, 02/26/2009 - 03:33
User Badges:

Hello all,


Please find attached debug files.


I have removed ip ospf mtu-ignore but left the ospf authentication statements in.


The outputs include :-


3750_CS01#sh system mtu


System MTU size is 1500 bytes

System Jumbo MTU size is 1500 bytes

Routing MTU size is 1500 bytes

3750_CS01#


show ip ospf

show ip ospf neighbor

debug ip ospf adj

debug ip ospf events





Attachment: 
pkurdziel Thu, 02/26/2009 - 14:28
User Badges:

Neighbor ID Pri State Dead Time Address Interface

10.137.243.1 1 EXCHANGE/DR 00:00:15 10.128.0.173


3750_CS01#sh ip ospf nei


Neighbor ID Pri State Dead Time Address Interface

10.134.243.2 1 INIT/DROTHER 00:00:13 10.128.0.174 GigabitEthernet1/0/47


OSPF: Nbr 10.134.243.2 10.128.0.174 GigabitEthernet1/0/47 is currently ignored


Common causes are

INIT - ACL blocking, corrupt database


EXSTART/EXCHANGE mtu mismatch- fix mtu or ip ospf mtu-ignore



Take a look at the acl's on the 6509.

Mohamad Qayoom Thu, 02/26/2009 - 14:41
User Badges:
  • Bronze, 100 points or more

Did you try removing CoS on the 6509 switch?

Giuseppe Larosa Thu, 02/26/2009 - 22:15
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,


I noticed that when the C6500 sends a bigger DBD with size 1412 bytes it is never received by c3750


example from your logs


255334: .Feb 26 11:04:46.296 GMT: OSPF: Send DBD to 10.137.243.1 on GigabitEthernet7/48 seq 0x99B opt 0x52 flag 0x2 len 1412

255335: .Feb 26 11:04:46.296 GMT: OSPF: Send with youngest Key 1


if you look in C3750 log and search for string Rcv DBD you can find only:


005726: .Feb 26 11:05:48.618: OSPF: Rcv DBD from 10.134.243.2 on GigabitEthernet1/0/47 seq 0x11BC opt 0x52 flag 0x7 len 32 mtu 1500 state EXSTART

005727: .Feb 26 11:05:48.618: OSPF: First DBD and we are not SLAVE


on the c6500 the same search gives much more matches


So the question can be at the physical layer check cables:

it looks like that small frames are received but when C6500 says it sending a DBD with size 1412 bytes it is not seen on the other side


check for errors in interfaces

with sh int g1/0/47


sh int g7/48


Hope to help

Giuseppe



mistryj Fri, 02/27/2009 - 05:48
User Badges:

Hi All,


I have tried removing access-lists and COS but no luck.


Giuseppe there are no errors on the interface and I do feel there is some configuration missing on the provider side. Do they need to have DF bit set on their equipment along with MTU 1500 ?


Hence the reason 6509 is not sending more than 1400 bytes ?



Giuseppe Larosa Sat, 02/28/2009 - 03:04
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,


I've never realized up to now there is another device in the middle or at least a L2 transport service of a service provider.


DF can be set on packets not on equipment. In our case DBD packets have the DF bit set in the IPv4 header.


Call them and explain that packets from one side (C6500) are not received on the other end (C3750) if their size is near to 1500.


the c6500 sends a DBD packet of 1412 bytes or with payload 1412 bytes but it is not seen on the C3750 side


You can also check yourself using extended ping and debug ip icmp if there are MTU issues on the link.

You can use the sweep option to explore a range of packet sizes while you have debug ip icmp on the other switch.



Hope to help

Giuseppe


mistryj Wed, 03/04/2009 - 13:42
User Badges:

Hi Giuseppe,


I have connected a laptop at one end and PC at the other end of the circuit and tested ping using different packet sizes and this was fine.


The provider configured a router and connected at the 6509 end with the 3750 at the other end and the connection worked for them. They are saying the circuit is fine and are not helping to resolve this any more.


What I cant understand is what different the 6509 IOS switch is doing compared to a Cisco 3750/Cisco Router. If I have a Cisco 3750 switch connected at both ends OSPF works fine.


Its getting to the point where we will have to put in a 3750 infront of the 6509 just to get this to work, its not a solution we want but we are running out of time.



Giuseppe Larosa Wed, 03/04/2009 - 14:01
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,

at this point I think there is a physical layer problem:


some years ago to have a correct working ethernet 10 Mbps link over SDH an older switch had to be inserted:


switch -- older switch -- sdh interface


the fact is that the DBD from the C6500 was not seen on the other side.


Only last attempt could be to reduce the ip mtu to something less then 1500 on both sides


Hope to help

Giuseppe

mistryj Wed, 03/04/2009 - 14:15
User Badges:

Hi Giuseppe,


I dont think the 3750 allows you to set the mtu. I can only set it on the 6509.


Also what affect will this have on performance if I set it to 1400 ?


Setting ip ospf mtu-ignore does this not have the same affect ?


Correct Answer
Giuseppe Larosa Wed, 03/04/2009 - 23:13
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,


by setting a lower ip mtu on the C6509 side you can be sure that is not trying to build a packet that is bigger then 1500.


mtu-ignore disable a check but doesn't influence the size of packets that are sent out the interface


you may combine

ip ospf mtu-ignore on 3750 side

with

ip ospf mtu-ignore

ip mtu 1400

on c6509 and see what happens


Hope to help

Giuseppe


mistryj Thu, 03/05/2009 - 02:58
User Badges:

Hi Giuseppe


This works !


I have set the 6509 with the lowest possible mtu which was 1456 and the OSPF has formed adjacency.


As I understand there are 2 methods of changing the MTU on the interface on the 6509 one is a L2 setting 'mtu 1500' and the other 'ip mtu' which is L3.


1. Is setting the mtu to 1456 on the packets likely to cause issues later on with other applications ?


2. Is there anything that the provider needs to do to resolve this issue ?









mistryj Thu, 03/19/2009 - 06:57
User Badges:

THE MTU issue finally resolved the provider upgraded the IOS on their network !!!

Giuseppe Larosa Thu, 03/19/2009 - 07:22
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jaya,

so at the end the problem was in the middle in the provider network


It has been kind of you to provide feedback this completes this case and makes it useful for other people


Best Regards

Giuseppe


mistryj Thu, 03/19/2009 - 15:13
User Badges:

Hi Giuseppe,


Yes I thought it best to. The provider upgraded the IOS one of their switches, but unfortunately they wont give us details on the upgrade and any configuration changes made on their network but we are glad its all working as it should now.


Many Thanks for your help.

Actions

This Discussion