OSPF problem: Neighbor Down 1-2 times per day

Leo Liu · ‎03-17-2014

Hi

We have 4 devices are running OSPF (3 cisco routers and 1 Juniper firewall as show in attachment file). In last few months, we got Neighbot Down message almost 1-2 times per day. Network between them interrupt for a short time and even monitoring mechanism does not aware the interruption(down time is too short). Could I have your advice of any possible root cause to this problem?

Timer intervals configured to all devices are same: Hello 10, Dead 40, Wait 40, Retransmit 5

------------------------------------------------------------------------------------------------------------

Neighbor A

Neighbor A#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
172.16.255.128 1 FULL/DROTHER 00:00:34 172.16.108.11 GigabitEthernet0/11
172.16.255.130 1 FULL/DR 00:00:33 172.16.108.2 GigabitEthernet0/11
172.16.255.64 1 FULL/DROTHER 00:00:34 172.16.107.1 GigabitEthernet0/12
172.16.255.128 1 FULL/DROTHER 00:00:38 172.16.107.11 GigabitEthernet0/12
172.16.255.130 1 FULL/DR 00:00:38 172.16.107.7 GigabitEthernet0/12

Neighbor A#show log

Mar 16 06:03:33.159: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.128 on GigabitEthernet0/12 from FULL to DOWN, Neighbor Down: Dead timer expired
Mar 16 06:03:50.137: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.128 on GigabitEthernet0/12 from LOADING to FULL, Loading Done
Mar 16 21:15:05.509: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.64 on GigabitEthernet0/12 from LOADING to FULL, Loading Done

------------------------------------------------------------------------------------------------------------

Neighbor B

Neighbor B#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
172.16.255.128 1 FULL/DROTHER 00:00:36 172.16.108.11 GigabitEthernet0/11
172.16.255.129 1 FULL/BDR 00:00:35 172.16.108.1 GigabitEthernet0/11
172.16.255.64 1 FULL/DROTHER 00:00:39 172.16.107.1 GigabitEthernet0/12
172.16.255.128 1 FULL/DROTHER 00:00:30 172.16.107.11 GigabitEthernet0/12
172.16.255.129 1 FULL/BDR 00:00:35 172.16.107.6 GigabitEthernet0/12

Neighbor B#show log

Mar 16 06:03:33.143: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.128 on GigabitEthernet0/12 from FULL to DOWN, Neighbor Down: Dead timer expired
Mar 16 06:03:50.122: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.128 on GigabitEthernet0/12 from LOADING to FULL, Loading Done
Mar 16 21:14:58.054: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.64 on GigabitEthernet0/12 from LOADING to FULL, Loading Done
Mar 16 21:15:03.800: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.128 on GigabitEthernet0/12 from LOADING to FULL, Loading Done

------------------------------------------------------------------------------------------------------------

Neighbor C

Neighbor C#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
172.16.255.129 1 FULL/BDR 00:00:35 172.16.108.1 FastEthernet0/1/0
172.16.255.130 1 FULL/DR 00:00:38 172.16.108.2 FastEthernet0/1/0
172.16.255.64 1 2WAY/DROTHER 00:00:30 172.16.107.1 FastEthernet0/0/1
172.16.255.129 1 FULL/BDR 00:00:35 172.16.107.6 FastEthernet0/0/1
172.16.255.130 1 FULL/DR 00:00:33 172.16.107.7 FastEthernet0/0/1

Neighbor C#show log

Mar 16 06:03:23.571: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.130 on FastEthernet0/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired
Mar 16 06:03:25.479: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.129 on FastEthernet0/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired
Mar 16 06:03:29.415: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.64 on FastEthernet0/0/1 from EXSTART to DOWN, Neighbor Down: Dead timer expired
Mar 16 06:03:50.112: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.129 on FastEthernet0/0/1 from LOADING to FULL, Loading Done
Mar 16 06:03:50.112: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.130 on FastEthernet0/0/1 from LOADING to FULL, Loading Done
Mar 16 21:14:53.740: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.130 on FastEthernet0/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired
Mar 16 21:15:03.793: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.130 on FastEthernet0/0/1 from LOADING to FULL, Loading Done

LA-Engineer · ‎03-17-2014

How's the cpu utilization on these routers during the flap?

Leo Liu · ‎03-17-2014

We were not monitor CPU utilization of these routers and I just put them in monitoring. The flap often happens in one minute and I am not sure if it will be monitored. I will provide monitor results after next flap.

Muhammad Thanveer · ‎03-18-2014

How did u configure the switch between Router C and Firewall D, since you are forming all the neighbors on F0/0/1 which is between Router C and Firewall

Leo Liu · ‎03-18-2014

On the switch between C and D, 3 ports (to C, to D and to A,B switch) were assigned with same vlan.

Following interfaces are all configured in subnet 172.16.107.0/24:

G0/12 on Router A
G0/12 on Router B
F0/0/1 on router C
Outside (to switch) on Firewall D

Leo Liu · ‎03-19-2014

We have interupt this morning and CPU utilization of these routers are quite low.

CPU utilization:

Router A: 5% Router B: 5% Router C: 2%

show log:

Router A

Mar 20 01:40:17.252: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.128 on GigabitEthernet0/12 from LOADING to FULL, Loading Done
Mar 20 01:40:26.203: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.64 on GigabitEthernet0/12 from LOADING to FULL, Loading Done

Router B

Mar 20 01:40:24.597: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.128 on GigabitEthernet0/12 from LOADING to FULL, Loading Done

Router C

Mar 20 01:40:16.163: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.129 on FastEthernet0/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired
Mar 20 01:40:16.179: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.64 on FastEthernet0/0/1 from LOADING to FULL, Loading Done
Mar 20 01:40:17.243: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.129 on FastEthernet0/0/1 from LOADING to FULL, Loading Done
Mar 20 01:40:24.559: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.130 on FastEthernet0/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired
Mar 20 01:40:24.591: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.255.130 on FastEthernet0/0/1 from LOADING to FULL, Loading Done

I am thinking to replace a switch between C and D, any advice? Thanks.

rkbala3560 · ‎03-17-2014

Hi

is there any L2 loops happening in network..

Leo Liu · ‎03-17-2014

I think no L2 loops in the network. We have spanning tree configured on switches connecting to router A & B.

VLAN0108
Spanning tree enabled protocol ieee
Root ID    Priority    32876
             Address     5c50.15a3.8480
             This bridge is the root
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority    32876 (priority 32768 sys-id-ext 108)
             Address     5c50.15a3.8480
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec
             Aging Time 300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/45            Desg FWD 4         128.45   P2p Edge
Gi1/0/47            Desg FWD 19        128.47   P2p Edge
Gi1/0/48            Desg FWD 4         128.48   P2p Edge

VLAN0107
Spanning tree enabled protocol ieee
Root ID    Priority    32875
             Address     5c50.15ec.f000
             This bridge is the root
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority    32875 (priority 32768 sys-id-ext 107)
             Address     5c50.15ec.f000
             Hello Time   2 sec Max Age 20 sec Forward Delay 15 sec
             Aging Time 300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/45            Desg FWD 4         128.45   P2p Edge
Gi1/0/47            Desg FWD 19        128.47   P2p Edge
Gi1/0/48            Desg FWD 4         128.48   P2p Edge

LA-Engineer · ‎03-19-2014

It looks like the adjacency on multiple interfaces are flapping. Are all these routers connected through a common switch?

Leo Liu · ‎03-19-2014

Please refer to topology as attach file I just updated. Actually we have 3 offices (site 1, 2 and 3) are connecting with 10M FTTB link between 2 sites.