cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4446
Views
0
Helpful
7
Replies

OSPF adjacency down due to BFD node down

rommel-peraza
Level 1
Level 1

Hi there,

I am having an issue with OSPF adjacency going down because of BFD triggers a recalculation, there is no indication (logs) of any other event as flapping interfaces, CPU or traffic spike.

Message:

OSPF-5-ADJCHG

1149: Jan  9 00:03:09 EST: %OSPF-5-ADJCHG: Process 200, Nbr x.x.x.x on TenGigabitEthernet0/0/0 from FULL to DOWN, Neighbor Down: BFD node down

------------------------------

OSPF-5-ADJCHG

1150: Jan  9 00:03:09 EST: %OSPF-5-ADJCHG: Process 200, Nbr x.x.x.x on TenGigabitEthernet0/0/0 from LOADING to FULL, Loading Done

As you can see, it lasted less than a second.

OSPF is between:

ASR 1002 running IOS-XE, Version 15.3(1)S, release (fc4)  and a

Nexus 7000 running NX-OS n7000 (n7000-s1-d​k9), Version 5.1(5), release (c)

BFD timers: 750 msec / Mult 3 (not so aggresive)

I have seen this issue before in other company, but I never got to see the resolution. I do remember that it took a lot of work with Cisco TAC to figure out the root cause because that network was really complex and the timers were more aggresive, this is not the case, but I am sure it can get a lot worse.

Any recommendations?

7 Replies 7

mtsb
Level 1
Level 1

Hi,

Is the BFD in hardware of software? How often is the flap? What are the configs of BFD looknig like? What message it throws on N7K side? Its always bit trickly with BFD that it does its job by taking the adjacency down for missing BFD messages but we need to find out why and how often it get dropped and is there anything disturbing BFD packet exchange. Better to protect control plane packets with features like CoPP so we don't miss anything.

Thanks,

Madhu

Hi Madhu,

Thank you for taking the time to review this. BFD is on software, it has hapenned only twice so far, last Wednesday night and again on Friday night (non usual high trafficc hours). No BFD or OSPF messages are shown on the N7K. We don't have any QoS nor CoPP configured on those devices.

Thanks, 

Madhu,

I was wrong before, on the N7K we do have CoPP and BFD is in hardware on that device.

Hi Rommel,

Similarly did you protect BFD packets on ASR1K as well?

So you see the BFD flapss only on ASR1K and OSPF is taken down and brought up quickly?

Did you see the OSPF messages on N7K that it is going down and what is the reason OSPF throws?

A "show log" from both the devices when the issue is seen will help.

Can you post sh bfd neighbors  detail from ASR1K?
If BFD is in software on this side can you add "no bfd echo" to make it offload to hardware?

I am trying to understand which direction the packet is dropping.

Thanks,

Madhu

Madjhu,

There is no CoPP configured on the ASR1K to protect bfd packets. What else could we use to protect them?

Yes, BFD flapps only occur on the ASR and then OSPF flapps too.

No OSPF messages are shown on the N7K (no logs at all on those days, actually)

What could be the impact of applying that command "no bfd echo"?

The outputs:

IL04RT-SM02#sh log | in Jan

Jan   9 00:03:09 EST: %OSPF-5-ADJCHG: Process 200, Nbr 172.21.25.10 on  TenGigabitEthernet0/2/0 from FULL to DOWN, Neighbor Down: BFD node down

Jan  9 00:03:09 EST: %OSPF-5-ADJCHG: Process 200, Nbr 172.21.25.10 on TenGigabitEthernet0/2/0 from LOADING to FULL, Loading Done

Jan  10 21:14:30 EST: %OSPF-5-ADJCHG: Process 200, Nbr 172.21.25.10 on  TenGigabitEthernet0/2/0 from FULL to DOWN, Neighbor Down: BFD node down

Jan 10 21:14:33 EST: %OSPF-5-ADJCHG: Process 200, Nbr 172.21.25.10 on TenGigabitEthernet0/2/0 from LOADING to FULL, Loading Done

IL04RT-SM02#

IL04RT-SM02#

IL04RT-SM02#

IL04RT-SM02#sh bfd neighbors det

IPv4 Sessions

NeighAddr                              LD/RD         RH/RS     State     Int

172.21.5.33                          2049/1107296279 Up        Up        Te0/2/0

Session state is UP and using echo function with 750 ms interval.

Session Host: Software

OurAddr: 172.21.5.34

Handle: 1

Local Diag: 0, Demand mode: 0, Poll bit: 0

MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3

Received MinRxInt: 2000000, Received Multiplier: 3

Holddown (hits): 0(0), Hello (hits): 2000(12952676)

Rx Count: 24277084, Rx Interval (ms) min/max/avg: 26/5490/990 last: 472 ms ago

Tx Count: 12952695, Tx Interval (ms) min/max/avg: 1/2051/1756 last: 1619 ms ago

Elapsed time watermarks: 0 0 (last: 0)

Registered protocols: CEF OSPF

Uptime: 4d12h

Last packet: Version: 1                  - Diagnostic: 0

             State bit: Up               - Demand bit: 0

             Poll bit: 0                 - Final bit: 0

             C bit: 0

             Multiplier: 3               - Length: 24

             My Discr.: 1107296279       - Your Discr.: 2049

             Min tx interval: 750000     - Min rx interval: 2000000

             Min Echo interval: 750000

After  discussing this case with my manager, we have decided to turn off bfd on  that link because we don't have an alternate path for OSPF so BFD is  not really helping there, however I would like to find the cause of this  issue in case I see it again.

Thank you.

Hi Madjhu,

Did you have a chance to check this out?

Thanks,

Hello, 

I have similar issue. The messages appear like this 

008147: May 12 2017 00:33:42.367 CDT: %OSPF-5-ADJCHG: Process 4, Nbr 10.16.8.9 on TenGigabitEthernet0/1/0.801 from FULL to DOWN, Neighbor Down: BFD node down
008148: May 12 2017 00:33:42.396 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.16.0.2 on TenGigabitEthernet2/0/2.800 from FULL to DOWN, Neighbor Down: BFD node down
008149: May 12 2017 00:33:42.476 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.16.0.2 on TenGigabitEthernet0/1/0.800 from FULL to DOWN, Neighbor Down: BFD node down
008150: May 12 2017 00:33:42.498 CDT: %OSPF-5-ADJCHG: Process 8, Nbr 10.16.16.9 on TenGigabitEthernet0/1/0.802 from FULL to DOWN, Neighbor Down: BFD node down
008151: May 12 2017 00:33:42.928 CDT: %OSPF-5-ADJCHG: Process 8, Nbr 10.16.16.9 on TenGigabitEthernet0/1/0.802 from LOADING to FULL, Loading Done
008152: May 12 2017 00:33:42.938 CDT: %OSPF-5-ADJCHG: Process 4, Nbr 10.16.8.9 on TenGigabitEthernet0/1/0.801 from LOADING to FULL, Loading Done
008153: May 12 2017 00:33:43.031 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.16.0.2 on TenGigabitEthernet0/1/0.800 from LOADING to FULL, Loading Done
008154: May 12 2017 00:33:49.818 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.16.0.2 on TenGigabitEthernet2/0/2.800 from LOADING to FULL, Loading Done
008155: May 12 2017 00:34:18.461 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.16.0.2 on TenGigabitEthernet2/0/2.800 from FULL to DOWN, Neighbor Down: BFD node down
008156: May 12 2017 00:34:18.556 CDT: %OSPF-5-ADJCHG: Process 8, Nbr 10.16.16.9 on TenGigabitEthernet0/1/0.802 from FULL to DOWN, Neighbor Down: BFD node down
008157: May 12 2017 00:34:18.591 CDT: %OSPF-5-ADJCHG: Process 4, Nbr 10.16.8.9 on TenGigabitEthernet0/1/0.801 from FULL to DOWN, Neighbor Down: BFD node down
008158: May 12 2017 00:34:18.660 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.16.0.2 on TenGigabitEthernet0/1/0.800 from FULL to DOWN, Neighbor Down: BFD node down

sho bfd neighbors details

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.1.182 4306/1107317085 Up Up Te0/0/0.800
Session state is UP and using echo function with 250 ms interval.
Session Host: Software
OurAddr: 10.16.1.177
Handle: 12
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(1), Hello (hits): 2000(5498187)
Rx Count: 11500741, Rx Interval (ms) min/max/avg: 1/8210/848 last: 197 ms ago
Tx Count: 5498458, Tx Interval (ms) min/max/avg: 1/2008/1753 last: 280 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF
Uptime: 18:35:00
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
--More--   My Discr.: 1107317085 - Your Discr.: 4306
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.1.190 4568/1107324499 Up Up Te0/1/0.800
Session state is UP and using echo function with 250 ms interval.
Session Host: Software
OurAddr: 10.16.1.185
Handle: 9
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 2000(376597)
Rx Count: 760925, Rx Interval (ms) min/max/avg: 1/1683/822 last: 200 ms ago
Tx Count: 376611, Tx Interval (ms) min/max/avg: 1/2005/1758 last: 1343 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF
Uptime: 2d05h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
--More--   Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1107324499 - Your Discr.: 4568
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.3.10 4412/0 Down Down Gi1/1/5.800
Session Host: Software
OurAddr: 10.16.3.9
Handle: 3
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 5
Received MinRxInt: 0, Received Multiplier: 0
Holddown (hits): 0(0), Hello (hits): 1000(6346007)
Rx Count: 0, Rx Interval (ms) min/max/avg: 0/0/0 last: 4294967295 ms ago
Tx Count: 6346007, Tx Interval (ms) min/max/avg: 754/1010/877 last: 155 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF
Last packet: Version: 1 - Diagnostic: 0
--More--   State bit: AdminDown - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 0 - Length: 0
My Discr.: 0 - Your Discr.: 0
Min tx interval: 0 - Min rx interval: 0
Min Echo interval: 0

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.3.46 4564/0 Down Down Te2/0/0.800
Session Host: Software
OurAddr: 10.16.3.41
Handle: 2
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 5
Received MinRxInt: 0, Received Multiplier: 0
Holddown (hits): 0(0), Hello (hits): 1000(752572)
Rx Count: 0, Rx Interval (ms) min/max/avg: 0/0/0 last: 4294967295 ms ago
Tx Count: 752572, Tx Interval (ms) min/max/avg: 754/1005/878 last: 568 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF
--More--  Last packet: Version: 1 - Diagnostic: 0
State bit: AdminDown - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 0 - Length: 0
My Discr.: 0 - Your Discr.: 0
Min tx interval: 0 - Min rx interval: 0
Min Echo interval: 0

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.3.66 4565/1107317104 Up Up Te2/0/1.800
Session state is UP and using echo function with 250 ms interval.
Session Host: Software
OurAddr: 10.16.3.65
Handle: 6
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 2000(376684)
Rx Count: 758158, Rx Interval (ms) min/max/avg: 1/1942/848 last: 0 ms ago
Tx Count: 376710, Tx Interval (ms) min/max/avg: 1/2005/1758 last: 358 ms ago
--More--  Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF
Uptime: 18:35:01
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1107317104 - Your Discr.: 4565
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.3.74 4498/1107324501 Up Up Te2/0/2.800
Session state is UP and using echo function with 250 ms interval.
Session Host: Software
OurAddr: 10.16.3.73
Handle: 15
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
--More--  Holddown (hits): 0(1), Hello (hits): 2000(1939070)
Rx Count: 3900157, Rx Interval (ms) min/max/avg: 1/6720/822 last: 467 ms ago
Tx Count: 1939661, Tx Interval (ms) min/max/avg: 1/2007/1757 last: 1123 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF
Uptime: 2d05h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1107324501 - Your Discr.: 4498
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.9.182 4128/1107317107 Up Up Te0/0/0.801
Session state is UP and using echo function with 250 ms interval.
Session Host: Software
OurAddr: 10.16.9.177
Handle: 5
--More--  Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(1), Hello (hits): 2000(30962360)
Rx Count: 64019066, Rx Interval (ms) min/max/avg: 1/645328/848 last: 585 ms ago
Tx Count: 30962966, Tx Interval (ms) min/max/avg: 1/2012/1756 last: 153 ms ago
Elapsed time watermarks: 2 2 (last: 2)
Registered protocols: CEF OSPF
Uptime: 18:34:58
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1107317107 - Your Discr.: 4128
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.9.190 4566/1107324487 Up Up Te0/1/0.801
Session state is UP and using echo function with 250 ms interval.
--More--  Session Host: Software
OurAddr: 10.16.9.185
Handle: 7
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 2000(376620)
Rx Count: 760929, Rx Interval (ms) min/max/avg: 1/1683/822 last: 18 ms ago
Tx Count: 376634, Tx Interval (ms) min/max/avg: 1/2005/1756 last: 1011 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF
Uptime: 2d05h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1107324487 - Your Discr.: 4566
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
--More--  10.16.17.182 4567/1107317109 Up Up Te0/0/0.802
Session state is UP and using echo function with 250 ms interval.
Session Host: Software
OurAddr: 10.16.17.177
Handle: 8
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 2000(376727)
Rx Count: 758136, Rx Interval (ms) min/max/avg: 1/4778/848 last: 132 ms ago
Tx Count: 376753, Tx Interval (ms) min/max/avg: 1/2005/1750 last: 213 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF
Uptime: 18:34:57
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
My Discr.: 1107317109 - Your Discr.: 4567
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000
--More--  
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.17.190 4560/1107324502 Up Up Te0/1/0.802
Session state is UP and using echo function with 250 ms interval.
Session Host: Software
OurAddr: 10.16.17.185
Handle: 4
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 2000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 2000(770996)
Rx Count: 1510680, Rx Interval (ms) min/max/avg: 1/7855/822 last: 437 ms ago
Tx Count: 771252, Tx Interval (ms) min/max/avg: 1/2005/1756 last: 766 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: CEF OSPF
Uptime: 2d05h
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3 - Length: 24
--More--   My Discr.: 1107324502 - Your Discr.: 4560
Min tx interval: 250000 - Min rx interval: 2000000
Min Echo interval: 50000

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.16.19.46 4411/4100 Up Up Gi1/1/5.802
Session state is UP and using echo function with 400 ms interval.
Session Host: Software
OurAddr: 10.16.19.41
Handle: 1
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 5
Received MinRxInt: 1000000, Received Multiplier: 5
Holddown (hits): 0(0), Hello (hits): 1000(6345690)
Rx Count: 6350593, Rx Interval (ms) min/max/avg: 1/1036/876 last: 876 ms ago
Tx Count: 6345692, Tx Interval (ms) min/max/avg: 17/1012/879 last: 465 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF CEF
Uptime: 9w1d
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
--More--   Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 5 - Length: 24
My Discr.: 4100 - Your Discr.: 4411
Min tx interval: 1000000 - Min rx interval: 1000000

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: