ARP table not populating mac address for previously reachable IP address
Router has been online and working fine with one BGP neighbor for almost 2 years and no downtime. 2 weeks ago, added a 2nd BGP peer. Everything worked fine for 2 weeks, then all of a sudden yesterday the 2nd BGP peer is disconnected and does not come back. ISP checks and sees everything looks fine on their end. We cannot even ping each other now.
Upon investigation, the ARP table is not even populating the MAC address for the BGP peer IP anymore (same local subnet). Stays "incomplete" in the table no matter what we do, including clearing arp table, changing IP address, etc.
Plug a laptop directly into the 2nd BGP peer FE port and replicate the IP addressing. Laptop cannot ping Router, but Router CAN ping laptop. Check ARP table, but STILL no mac address assigned and now not even the ARP table showing "incomplete".
Thinking it could be the FE interface, switch to the 2nd FE interface and perform same laptop test, this time with arbitrary IP addressing. Now cannot ping each other, no MAC in ARP table.
End up rebooting the router and lo-and-behold, everything is working normally again. 2nd BGP peer peers up instantly.
I should also mention that the 1st BGP peer worked flawlessly throughout, taking all the Internet load and having no issues throughout.
Also, the FE ports for the 2nd BGP peer are on an HWIC FE card plugged into the router. The 1st BGP peer is plugged into the built-in GE interface. 2901 running: c2900-universalk9-mz.SPA.151-4.M4.bin
Lastly, no router resource issues, no error messages, no logs. Just the BGP peer disconnecting.
I have never, in 20 years working with Cisco routers seen something like this before. This is the most fundamental aspect of IP and Ethernet that was not working.
Has anyone ever seen this behavior before??
Here is the router config (IP's changed):
version 15.1 service timestamps debug datetime msec localtime show-timezone service timestamps log datetime msec localtime show-timezone service password-encryption service internal service sequence-numbers ! boot-start-marker boot-end-marker ! ! logging buffered 150000 ! aaa new-model ! ! aaa authentication login LAUTHEN local aaa authentication login TAUTHEN local group tacacs+ enable aaa authorization console aaa authorization exec LAUTHOR local if-authenticated aaa authorization exec TAUTHOR local group tacacs+ if-authenticated ! ! ! ! ! aaa session-id common clock timezone PST -8 0 clock summer-time PDT recurring ! no ipv6 cef no ip source-route ip cef ! ! ! ! ! no ip domain lookup multilink bundle-name authenticated ! !
! ! username ubiadmin privilege 15 secret 4 .JbeuWXuZvchrG0OL.5BftFtqrrEyxcnVHn5rIuCnTk username umitsnoc01 privilege 15 secret 4 cUmoRUjey9O1x.wk9S.kleX.iAAhCwihupr6Z98p6OA ! redundancy ! ! ip ssh version 2 ! track 1 interface GigabitEthernet0/0 line-protocol ! class-map match-any AutoQoS-VoIP-RTP-Trust match access-group name SIP-Media-INBOUND class-map match-any AutoQoS-VoIP-Control-Trust match ip dscp cs3 match ip dscp af31 class-map match-any Customer-Voice match access-group name Customer-VPNs class-map match-any media match access-group name SIP-Media class-map match-any signaling match access-group name SIP-Signaling ! ! policy-map AutoQoS-Policy-Trust class AutoQoS-VoIP-RTP-Trust priority percent 70 class AutoQoS-VoIP-Control-Trust bandwidth percent 5 class class-default fair-queue policy-map queue class signaling bandwidth percent 5 class media priority percent 50 class Customer-Voice priority percent 40 class class-default fair-queue policy-map shape class class-default shape average 10000000 service-policy queue ! ! ! ! ! interface Embedded-Service-Engine0/0 no ip address shutdown ! interface GigabitEthernet0/0 description BGP Peer 1 ip address 126.96.36.199 255.255.255.252 no ip redirects ip flow ingress ip flow egress duplex auto speed auto service-policy output shape ! interface GigabitEthernet0/1 description LAN ip address 188.8.131.52 255.255.255.0 no ip redirects ip flow ingress ip flow egress standby 255 ip 184.108.40.206 standby 255 priority 105 standby 255 preempt standby 255 mac-address 1a2b.3c4d.5e6f standby 255 track 1 decrement 10 duplex auto speed auto service-policy output AutoQoS-Policy-Trust ! interface FastEthernet0/0/0 description BGP Peer 2 ip address 220.127.116.11 255.255.255.252 ip flow ingress ip flow egress duplex full speed 100 service-policy output shape ! interface FastEthernet0/0/1 no ip address shutdown duplex auto speed auto ! ! router bgp 7777 bgp router-id 18.104.22.168 bgp log-neighbor-changes network 22.214.171.124 mask 255.255.255.0 neighbor 126.96.36.199 remote-as 5555 neighbor 188.8.131.52 update-source FastEthernet0/0/0 neighbor 184.108.40.206 prefix-list L3-DEFGW in neighbor 220.127.116.11 route-map L3-LPREF-IN in neighbor 18.104.22.168 remote-as 6666 neighbor 22.214.171.124 ebgp-multihop 2 neighbor 126.96.36.199 update-source GigabitEthernet0/0 neighbor 188.8.131.52 send-community neighbor 184.108.40.206 prefix-list COLO-DEFGW in neighbor 220.127.116.11 route-map COLO-LPREF-IN in neighbor 18.104.22.168 route-map COLO-OUT out ! ip forward-protocol nd ! ip bgp-community new-format ip as-path access-list 5 permit _5555_ ip as-path access-list 5 deny .* ip as-path access-list 10 permit ^6666$ no ip http server no ip http secure-server ip flow-top-talkers top 50 sort-by bytes ! ip route 0.0.0.0 0.0.0.0 22.214.171.124 254 name L3 ip route 0.0.0.0 0.0.0.0 126.96.36.199 255 name COLO1 ip route 10.0.0.0 255.0.0.0 10.10.10.10 name FW_OUTSIDE ip tacacs source-interface GigabitEthernet0/1 ! ip access-list standard SNMP_SOURCES permit 188.8.131.52 0.0.0.255 deny any log
! ! ip prefix-list L3-DEFGW seq 5 permit 0.0.0.0/0 ! ip prefix-list COLO-DEFGW seq 5 permit 0.0.0.0/0 ! ip prefix-list COLO-LPREF-OUT seq 5 permit 184.108.40.206/24
Question We run asr9001 with XR 6.1.3, and we have a very long delay to
login w/ SSH 1 or 2 to the device compare to IOS device. After
investigation, the there is 1s delay between the client KEXDH_INIT and
the server (XR) KEXDH_REPLY. After debug ssh serv...
Introduction The purpose of this document is to demonstrate the Open
Shortest Path First (OSPF) behavior when the V-bit (Virtual-link bit) is
present in a non-backbone area. The V-bit is signaled in Type-1 LSA only
if the router is the endpoint of one or ...
Hi, I am seeing quite a few issues with patch install and wanted to
share my experience and workaround to this. Login to admin via CLI, then
access root with the “shell” command Issue “df –h” and you’ll probably
see the following directory full or nearly ...