Router has been online and working fine with one BGP neighbor for almost 2 years and no downtime. 2 weeks ago, added a 2nd BGP peer. Everything worked fine for 2 weeks, then all of a sudden yesterday the 2nd BGP peer is disconnected and does not come back. ISP checks and sees everything looks fine on their end. We cannot even ping each other now.
Upon investigation, the ARP table is not even populating the MAC address for the BGP peer IP anymore (same local subnet). Stays "incomplete" in the table no matter what we do, including clearing arp table, changing IP address, etc.
Plug a laptop directly into the 2nd BGP peer FE port and replicate the IP addressing. Laptop cannot ping Router, but Router CAN ping laptop. Check ARP table, but STILL no mac address assigned and now not even the ARP table showing "incomplete".
Thinking it could be the FE interface, switch to the 2nd FE interface and perform same laptop test, this time with arbitrary IP addressing. Now cannot ping each other, no MAC in ARP table.
End up rebooting the router and lo-and-behold, everything is working normally again. 2nd BGP peer peers up instantly.
I should also mention that the 1st BGP peer worked flawlessly throughout, taking all the Internet load and having no issues throughout.
Also, the FE ports for the 2nd BGP peer are on an HWIC FE card plugged into the router. The 1st BGP peer is plugged into the built-in GE interface. 2901 running: c2900-universalk9-mz.SPA.151-4.M4.bin
Lastly, no router resource issues, no error messages, no logs. Just the BGP peer disconnecting.
I have never, in 20 years working with Cisco routers seen something like this before. This is the most fundamental aspect of IP and Ethernet that was not working.
Has anyone ever seen this behavior before??
Here is the router config (IP's changed):
version 15.1
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service internal
service sequence-numbers
!
boot-start-marker
boot-end-marker
!
!
logging buffered 150000
!
aaa new-model
!
!
aaa authentication login LAUTHEN local
aaa authentication login TAUTHEN local group tacacs+ enable
aaa authorization console
aaa authorization exec LAUTHOR local if-authenticated
aaa authorization exec TAUTHOR local group tacacs+ if-authenticated
!
!
!
!
!
aaa session-id common
clock timezone PST -8 0
clock summer-time PDT recurring
!
no ipv6 cef
no ip source-route
ip cef
!
!
!
!
!
no ip domain lookup
multilink bundle-name authenticated
!
!
!
!
!
username ubiadmin privilege 15 secret 4 .JbeuWXuZvchrG0OL.5BftFtqrrEyxcnVHn5rIuCnTk
username umitsnoc01 privilege 15 secret 4 cUmoRUjey9O1x.wk9S.kleX.iAAhCwihupr6Z98p6OA
!
redundancy
!
!
ip ssh version 2
!
track 1 interface GigabitEthernet0/0 line-protocol
!
class-map match-any AutoQoS-VoIP-RTP-Trust
match access-group name SIP-Media-INBOUND
class-map match-any AutoQoS-VoIP-Control-Trust
match ip dscp cs3
match ip dscp af31
class-map match-any Customer-Voice
match access-group name Customer-VPNs
class-map match-any media
match access-group name SIP-Media
class-map match-any signaling
match access-group name SIP-Signaling
!
!
policy-map AutoQoS-Policy-Trust
class AutoQoS-VoIP-RTP-Trust
priority percent 70
class AutoQoS-VoIP-Control-Trust
bandwidth percent 5
class class-default
fair-queue
policy-map queue
class signaling
bandwidth percent 5
class media
priority percent 50
class Customer-Voice
priority percent 40
class class-default
fair-queue
policy-map shape
class class-default
shape average 10000000
service-policy queue
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description BGP Peer 1
ip address 2.2.2.2 255.255.255.252
no ip redirects
ip flow ingress
ip flow egress
duplex auto
speed auto
service-policy output shape
!
interface GigabitEthernet0/1
description LAN
ip address 1.2.3.4 255.255.255.0
no ip redirects
ip flow ingress
ip flow egress
standby 255 ip 1.2.3.1
standby 255 priority 105
standby 255 preempt
standby 255 mac-address 1a2b.3c4d.5e6f
standby 255 track 1 decrement 10
duplex auto
speed auto
service-policy output AutoQoS-Policy-Trust
!
interface FastEthernet0/0/0
description BGP Peer 2
ip address 1.1.1.1 255.255.255.252
ip flow ingress
ip flow egress
duplex full
speed 100
service-policy output shape
!
interface FastEthernet0/0/1
no ip address
shutdown
duplex auto
speed auto
!
!
router bgp 7777
bgp router-id 2.2.2.2
bgp log-neighbor-changes
network 1.2.3.0 mask 255.255.255.0
neighbor 1.1.1.2 remote-as 5555
neighbor 1.1.1.2 update-source FastEthernet0/0/0
neighbor 1.1.1.2 prefix-list L3-DEFGW in
neighbor 1.1.1.2 route-map L3-LPREF-IN in
neighbor 2.2.2.1 remote-as 6666
neighbor 2.2.2.1 ebgp-multihop 2
neighbor 2.2.2.1 update-source GigabitEthernet0/0
neighbor 2.2.2.1 send-community
neighbor 2.2.2.1 prefix-list COLO-DEFGW in
neighbor 2.2.2.1 route-map COLO-LPREF-IN in
neighbor 2.2.2.1 route-map COLO-OUT out
!
ip forward-protocol nd
!
ip bgp-community new-format
ip as-path access-list 5 permit _5555_
ip as-path access-list 5 deny .*
ip as-path access-list 10 permit ^6666$
no ip http server
no ip http secure-server
ip flow-top-talkers
top 50
sort-by bytes
!
ip route 0.0.0.0 0.0.0.0 1.1.1.2 254 name L3
ip route 0.0.0.0 0.0.0.0 2.2.2.1 255 name COLO1
ip route 10.0.0.0 255.0.0.0 10.10.10.10 name FW_OUTSIDE
ip tacacs source-interface GigabitEthernet0/1
!
ip access-list standard SNMP_SOURCES
permit 12.12.12.0 0.0.0.255
deny any log
!
!
!
ip prefix-list L3-DEFGW seq 5 permit 0.0.0.0/0
!
ip prefix-list COLO-DEFGW seq 5 permit 0.0.0.0/0
!
ip prefix-list COLO-LPREF-OUT seq 5 permit 1.2.3.0/24
!
!
!
!
route-map COLO-LPREF-IN permit 5
match as-path 5
set local-preference 250
!
route-map COLO-LPREF-IN permit 10
set local-preference 150
!
route-map COLO-LPREF-IN permit 20
!
route-map COLO-OUT permit 10
match ip address prefix-list COLO-LPREF-OUT
set as-path prepend 7777 7777 7777
set community 29795:1004
!
route-map COLO-OUT permit 20
!
route-map L3-LPREF-IN permit 10
match as-path 10
set local-preference 200
!
route-map L3-LPREF-IN permit 20
set local-preference 150
!
!
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps vrrp
snmp-server enable traps flowmon
snmp-server enable traps transceiver all
snmp-server enable traps ds1
snmp-server enable traps call-home message-send-fail server-fail
snmp-server enable traps tty
snmp-server enable traps license
snmp-server enable traps envmon
snmp-server enable traps ethernet cfm cc mep-up mep-down cross-connect loop config
snmp-server enable traps ethernet cfm crosscheck mep-missing mep-unknown service-up
snmp-server enable traps flash insertion removal
snmp-server enable traps mac-notification
snmp-server enable traps aaa_server
snmp-server enable traps cef resource-failure peer-state-change peer-fib-state-change inconsistency
snmp-server enable traps memory bufferpeak
snmp-server enable traps config-copy
snmp-server enable traps config
snmp-server enable traps config-ctid
snmp-server enable traps event-manager
snmp-server enable traps hsrp
snmp-server enable traps cpu threshold
snmp-server enable traps rsvp
snmp-server enable traps syslog
snmp-server enable traps vtp
snmp-server enable traps ipsla
!
