OSPF Routing Table Refresh Issue!!

Chandra Dimri · ‎11-22-2013

Hello Guys,

I am working on below topology and I am facing issues of route refresh in ospf routing table

on both ABR's due to which connectivity to Internet is going down and up.

I am seeking your help to know the issue and help in troubleshooting the issue. Below is the output of routing table (showing route refresh) & negibor detail showing neigbor up since 2 days.

6506(ABR1)---ospf point-point-network----4500X (Area 2)

|

Back to Back connection

|

6506(ABR2)---ospf point-point-network----4500X (Area 2)

6506(ABR1)#sh ip route ospf

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.2.0/24 is variably subnetted, 4 subnets, 2 masks

O 192.168.2.4/30

[110/400] via 192.168.2.2, 00:06:26, GigabitEthernet1/2

O 192.168.2.8/30

[110/300] via 192.168.2.2, 00:06:26, GigabitEthernet1/2

192.168.4.0/24 is variably subnetted, 4 subnets, 2 masks

O 192.168.4.4/30

[110/400] via 192.168.4.2, 00:06:26, GigabitEthernet2/7

O 192.168.4.8/30

[110/300] via 192.168.4.2, 00:06:26, GigabitEthernet2/7

6506(ABR1)#

6506(ABR1)#sh ip ospf ne detail

Neighbor 192.168.2.5, interface address 192.8.178.4

In the area 0 via interface Vlan2

Neighbor priority is 1, State is FULL, 6 state changes

DR is 192.8.178.3 BDR is 192.8.178.4

Options is 0x12 in Hello (E-bit, L-bit)

Options is 0x52 in DBD (E-bit, L-bit, O-bit)

LLS Options is 0x1 (LR)

Dead timer due in 00:00:36

Neighbor is up for 2d04h

Index 1/2, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)

Last retransmission scan length is 0, maximum is 0

Last retransmission scan time is 0 msec, maximum is 0 msec

Neighbor 192.168.2.9, interface address 192.168.2.2

In the area 2 via interface GigabitEthernet1/2

Neighbor priority is 0, State is FULL, 6 state changes

DR is 0.0.0.0 BDR is 0.0.0.0

Options is 0x12 in Hello (E-bit, L-bit)

Options is 0x52 in DBD (E-bit, L-bit, O-bit)

LLS Options is 0x1 (LR)

Dead timer due in 00:00:39

Neighbor is up for 2d04h

Index 1/1, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)

Last retransmission scan length is 0, maximum is 0

Last retransmission scan time is 0 msec, maximum is 0 msec

Neighbor 192.168.4.2, interface address 192.168.4.2

In the area 4 via interface GigabitEthernet2/7

Neighbor priority is 0, State is FULL, 6 state changes

DR is 0.0.0.0 BDR is 0.0.0.0

Options is 0x12 in Hello (E-bit, L-bit)

Options is 0x52 in DBD (E-bit, L-bit, O-bit)

LLS Options is 0x1 (LR)

Dead timer due in 00:00:35

Neighbor is up for 2d01h

Index 1/3, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)

Last retransmission scan length is 0, maximum is 0

Last retransmission scan time is 0 msec, maximum is 0 msec

6506(ABR1)#

6506(ABR2)#sh ip ospf ne de

Neighbor 192.168.2.1, interface address 192.8.178.3

In the area 0 via interface Vlan2

Neighbor priority is 1, State is FULL, 6 state changes

DR is 192.8.178.3 BDR is 192.8.178.4

Options is 0x12 in Hello (E-bit, L-bit)

Options is 0x52 in DBD (E-bit, L-bit, O-bit)

LLS Options is 0x1 (LR)

Dead timer due in 00:00:32

Neighbor is up for 2d04h

Index 1/2, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)

Last retransmission scan length is 0, maximum is 0

Last retransmission scan time is 0 msec, maximum is 0 msec

Neighbor 192.168.2.10, interface address 192.168.2.6

In the area 2 via interface GigabitEthernet1/2

Neighbor priority is 0, State is FULL, 6 state changes

DR is 0.0.0.0 BDR is 0.0.0.0

Options is 0x12 in Hello (E-bit, L-bit)

Options is 0x52 in DBD (E-bit, L-bit, O-bit)

LLS Options is 0x1 (LR)

Dead timer due in 00:00:35

Neighbor is up for 2d04h

Index 1/1, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)

Last retransmission scan length is 0, maximum is 0

Last retransmission scan time is 0 msec, maximum is 0 msec

Neighbor 192.168.4.6, interface address 192.168.4.6

In the area 4 via interface GigabitEthernet1/8

Neighbor priority is 0, State is FULL, 6 state changes

DR is 0.0.0.0 BDR is 0.0.0.0

Options is 0x12 in Hello (E-bit, L-bit)

Options is 0x52 in DBD (E-bit, L-bit, O-bit)

LLS Options is 0x1 (LR)

Dead timer due in 00:00:31

Neighbor is up for 2d01h

Index 1/3, retransmission queue length 0, number of retransmission 0

First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)

Last retransmission scan length is 0, maximum is 0

Last retransmission scan time is 0 msec, maximum is 0 msec

6506(ABR2)#

6506(ABR2)#sh ip route ospf

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.2.0/24 is variably subnetted, 4 subnets, 2 masks

O 192.168.2.0/30

[110/400] via 192.168.2.6, 00:04:50, GigabitEthernet1/2

O 192.168.2.8/30

[110/300] via 192.168.2.6, 00:04:50, GigabitEthernet1/2

192.168.4.0/24 is variably subnetted, 4 subnets, 2 masks

O 192.168.4.0/30

[110/400] via 192.168.4.6, 00:04:50, GigabitEthernet1/8

O 192.168.4.8/30

[110/300] via 192.168.4.6, 00:04:50, GigabitEthernet1/8

6506(ABR2)#

Rolf Fischer · ‎11-22-2013

Hi,

there seems to be another (perhaps unwanted?) adjacency between the 4500X's via network 192.168.2.8/30.

The routes you've posted were leared via that connection (the intra-area path through area 2 is always prefered over the inter-area path through the backbone).

So maybe you should check if that adjacency looks stable as well.

I also noticed that the RIDs are the physical interface IP addresses. Did you configure the RIDs with the router-id command? If not, using IP-addresses of loopback interfaces as RID or using the router-id command would be a recommended improvement.

HTH

Rolf

Chandra Dimri · ‎11-22-2013

Hi Rolf,

4500X are also ospf neigbor back to back ove vlan 20 (192.168.2.8/30) & neighborship between them is stable up since 2 days. I haven't configure the router-d command till yet but will do the same, meanwhile I am just wandering what could be the issue which is causing route-refresh. Links are also stable since 2 days and Neigborship is all well stable since 2 days too.

Rolf Fischer · ‎11-22-2013

Good morning,

I guess the route refreshs are caused by SPF algorithm re-runs, probably because of a flapping link (this can be a passive interface as well).

In the area sections of "show ip ospf" you can see how often and when the SPFs were executed the last time.

The "Age" column of "show ip ospf database" can be useful to identify the flapping link.

HTH

Rolf

Chandra Dimri · ‎11-23-2013

Good Morning Rolf,

As per the logs on 6500 (ABR's) I don't see any logs of interface going down but yes its shows OSPF going down and loading for AREA 2 & AREA 4 Neighbors. Connectivity is on fiber (SM Mode Fiber) and for the links I dont see any links going down or up. Below is the output for SPF --

6500(ABR1)#sh ip ospf statistics detail

OSPF Router with ID (192.168.2.1) (Process ID 50)

Area 0: SPF algorithm executed 12 times

SPF 3 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:1 N:0 Stub:1 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.1(R)

SPF 4 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:3 SA:0 X7:0

Change record 0x0

LSIDs changed 4

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.5(R) 192.168.2.1(R) 192.8.178.3(N) 192.168.2.5(R)

SPF 5 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:4 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.5(R)

SPF 6 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:4 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.1(R)

SPF 7 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:4 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.5(R)

SPF 8 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:4 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.1(R)

SPF 9 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:4 SA:0 X7:0

Change record 0x0

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

SPF 10 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:4 SA:0 X7:0

Change record 0x0

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

SPF 11 executed 02:07:55 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:6 SA:0 X7:0

Change record SN, SA

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

SPF 12 executed 01:31:54 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:1 Stub:0 SN:6 SA:0 X7:0

Change record SN, SA

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

Area 2: SPF algorithm executed 15 times

SPF 6 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.5(R)

SPF 7 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

SPF 8 executed 2d05h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

SPF 9 executed 04:49:58 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:1 N:0 Stub:0 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 2

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.9(R) 192.168.2.1(R)

SPF 10 executed 04:44:27 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:1 N:0 Stub:1 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.1(R)

SPF 11 executed 04:44:18 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:0 Stub:2 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 3

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.9(R) 192.168.2.1(R) 192.168.2.9(R)

SPF 12 executed 04:39:13 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:2 N:0 Stub:3 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.9(R)

SPF 13 executed 04:39:04 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 3

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.10(R) 192.168.2.9(R) 192.168.2.10(R)

SPF 14 executed 02:07:59 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record SN, SA

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

SPF 15 executed 01:31:58 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record SN, SA

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

Area 4: SPF algorithm executed 22 times

SPF 13 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:1 N:0 Stub:1 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.1(R)

SPF 14 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 3

Changed LSAs. Recorded is LS ID and LS type:

192.168.4.2(R) 192.168.2.1(R) 192.168.4.2(R)

SPF 15 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:1 N:0 Stub:0 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.1(R)

SPF 16 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:1 N:0 Stub:1 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.1(R)

SPF 17 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 3

Changed LSAs. Recorded is LS ID and LS type:

192.168.4.2(R) 192.168.2.1(R) 192.168.4.2(R)

SPF 18 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:3 N:0 Stub:4 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.4.6(R)

SPF 19 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:3 N:0 Stub:5 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 1

Changed LSAs. Recorded is LS ID and LS type:

192.168.4.6(R)

SPF 20 executed 2d03h ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record 0x0

LSIDs changed 3

Changed LSAs. Recorded is LS ID and LS type:

192.168.2.5(R) 192.168.4.6(R) 192.168.2.5(R)

SPF 21 executed 02:08:04 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record SN, SA

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

SPF 22 executed 01:32:02 ago, SPF type Full

SPF calculation time (in msec):

SPT Intra D-Intr Summ D-Summ Ext7 D-Ext7 Total

0 0 0 0 0 0 0 0

LSIDs processed R:4 N:0 Stub:6 SN:0 SA:0 X7:0

Change record SN, SA

LSIDs changed 0

Changed LSAs. Recorded is LS ID and LS type:

6500(ABR1)#

Rolf Fischer · ‎11-23-2013

Thanks for reminding; the show ip ospf statistics detail provides even much better information.

So the age of the routing entries in question match the last SPF executions?

I think SPF runs 11 - 12 in Area 0, 14 - 15 in Area 2 and 21 - 22 in Area 4 are particulary interesting: To my best knowledge such SPF runs indicate that a router lost it's ABR status or just became an ABR (possibly both happend since there were two such occurrences).

[EDIT] :

I believe that the SPF runs are triggerd originally from Area 4.

Have a look at the timestamps:

Area 4:

SPF 21 executed 02:08:04 ago, SPF type Full

Area 2:

SPF 14 executed 02:07:59 ago, SPF type Full

=> 5 seconds later (initial delay to schedule a SPF by default)

Area 0:

SPF 11 executed 02:07:55 ago, SPF type Full

=> again 5 seconds later

The second occurrence shows exactly the same order.