12-19-2013 06:53 AM - edited 03-07-2019 05:10 PM
We have a 7204 router that has two BGP sessions to our upstream provider. The first one provides us a single default route, and the second one provides us a small list of Internet2 related routes. This setup has been working just fine for many years, but recently the second, Internet2 one has been having trouble. Both sides are trying to connect, but it never establishes. It mainly shows OpenConfirm when I check status. Nothing has been changed on this router in years, and no one has logged into it in many months. Our provider insists that the problem in on our end and they say they're not receiving keepalives from our side, but it looks like they're being sent. What can we do?
12-19-2013 06:57 AM
Hi,
Do you see any clue in the log?
Regards
12-19-2013 06:59 AM
Not sure... I've looked at a lot of output, I'm not sure what the problem could be. If I post 'debug ip bgp' output, would that help?
12-19-2013 07:01 AM
Have you run debugging "debug ip bgp" to see exactly what is happening when they try to establish a connection ? Obviously debugging can have an adverse effect on the router so be careful when/if you run it ?
OpenConfirm is almost all the way there. Either a notification message can be sent which transitions the state to idle or a keepalive is not received which again transitions it to idle.
Jon
12-19-2013 07:16 AM
Here's 'debug ip bgp'... W.X.Y.A is the local end, W.X.Y.Z is the remote end
Dec 18 15:58:30.278: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 32829ms (35000ms max, 28% jitter)
Dec 18 15:59:03.110: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 15:59:33.110: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active
delayed 28160ms (35000ms max, 28% jitter)
Dec 18 15:59:56.323: BGP: W.X.Y.A passive open to W.X.Y.Z
Dec 18 15:59:56.323: BGP: W.X.Y.A went from Active to Idle
Dec 18 15:59:56.327: BGP: W.X.Y.A went from Idle to Connect
Dec 18 15:59:56.327: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44
Dec 18 15:59:56.327: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds
Dec 18 15:59:56.327: BGP: W.X.Y.A went from Connect to OpenSent
Dec 18 15:59:56.327: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds
Dec 18 15:59:56.327: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6
Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4
Dec 18 15:59:56.327: BGP: W.X.Y.A unrecognized capability code: 65 - ingored
BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112
Dec 18 15:59:56.327: BGP: W.X.Y.A went from OpenSent to OpenConfirm
Dec 18 15:59:56.327: BGP: W.X.Y.A send message type 1, length (incl. header) 45
Dec 18 16:01:26.327: BGP: W.X.Y.A connection timed out - has not accepted a message from us for 90000ms (hold
time), 0 messages pending transmition
Dec 18 16:01:26.327: BGP: W.X.Y.A went from OpenConfirm to Closing
Dec 18 16:01:26: %BGP-3-NOTIFICATION: sent to neighbor W.X.Y.A 4/0 (hold time expired) 0 bytes
Dec 18 16:01:26.327: BGP: W.X.Y.A send message type 3, length (incl. header) 21
Dec 18 16:01:26.327: BGP: W.X.Y.A local error close after sending NOTIFICATION
Dec 18 16:01:26.327: BGPNSF state: W.X.Y.A went from nsf_not_active to nsf_not_active
Dec 18 16:01:26.327: BGP: W.X.Y.A went from Closing to Idle
Dec 18 16:01:26.327: BGP: W.X.Y.A closing
Dec 18 16:01:27.331: BGP: W.X.Y.A went from Idle to Active
Dec 18 16:01:27.331: BGP: W.X.Y.A open active delayed 30201ms (35000ms max, 28% jitter)
Dec 18 16:01:57.535: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:02:27.536: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active
delayed 32450ms (35000ms max, 28% jitter)
Dec 18 16:02:59.988: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:03:29.988: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active
delayed 26383ms (35000ms max, 28% jitter)
Dec 18 16:03:54.328: BGP: W.X.Y.A passive open to W.X.Y.Z
Dec 18 16:03:54.328: BGP: W.X.Y.A went from Active to Idle
Dec 18 16:03:54.328: BGP: W.X.Y.A went from Idle to Connect
Dec 18 16:03:54.332: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44
Dec 18 16:03:54.332: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds
Dec 18 16:03:54.332: BGP: W.X.Y.A went from Connect to OpenSent
Dec 18 16:03:54.332: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds
Dec 18 16:03:54.332: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6
Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4
Dec 18 16:03:54.332: BGP: W.X.Y.A unrecognized capability code: 65 - ingored
BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112
Dec 18 16:03:54.332: BGP: W.X.Y.A went from OpenSent to OpenConfirm
Dec 18 16:03:54.332: BGP: W.X.Y.A send message type 1, length (incl. header) 45
Dec 18 16:05:24.329: BGP: W.X.Y.A rcv message type 3, length (excl. header) 2
Dec 18 16:05:24: %BGP-3-NOTIFICATION: received from neighbor W.X.Y.A 4/0 (hold time expired) 0 bytes
Dec 18 16:05:24.333: BGP: W.X.Y.A went from OpenConfirm to Closing
Dec 18 16:05:24.333: BGPNSF state: W.X.Y.A went from nsf_not_active to nsf_not_active
Dec 18 16:05:24.333: BGP: W.X.Y.A went from Closing to Idle
Dec 18 16:05:24.333: BGP: W.X.Y.A closing
Dec 18 16:05:24.433: BGP: W.X.Y.A went from Idle to Active
Dec 18 16:05:24.433: BGP: W.X.Y.A open active delayed 34021ms (35000ms max, 28% jitter)
Dec 18 16:05:33: %SEC-6-IPACCESSLOGP: list 110 denied tcp 10.2.48.85(443) (GigabitEthernet2/0.738 b0c6.9ae6.552d)
-> 208.108.175.254(50390), 1 packet
Dec 18 16:05:58.457: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:06:28.457: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active
delayed 28111ms (35000ms max, 28% jitter)
Dec 18 16:06:56.570: BGP: W.X.Y.A open active, local address W.X.Y.Z
Dec 18 16:07:26.570: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active
delayed 33307ms (35000ms max, 28% jitter)
Dec 18 16:07:52.334: BGP: W.X.Y.A passive open to W.X.Y.Z
Dec 18 16:07:52.334: BGP: W.X.Y.A went from Active to Idle
Dec 18 16:07:52.334: BGP: W.X.Y.A went from Idle to Connect
Dec 18 16:07:52.334: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44
Dec 18 16:07:52.338: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds
Dec 18 16:07:52.338: BGP: W.X.Y.A went from Connect to OpenSent
Dec 18 16:07:52.338: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds
Dec 18 16:07:52.338: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6
Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6
Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4
Dec 18 16:07:52.338: BGP: W.X.Y.A unrecognized capability code: 65 - ingored
BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112
Dec 18 16:07:52.338: BGP: W.X.Y.A went from OpenSent to OpenConfirm
Dec 18 16:07:52.338: BGP: W.X.Y.A send message type 1, length (incl. header) 45
12-19-2013 08:02 AM
Harold may be able to provide a better analysis but it looks like your router is having problems connecting to the ISP router ie. from the debug -
1) your router tries an active open to the ISP router but it keeps failing
2) only when the ISP router initiates the connection does anything happen
3) You receive an OPEN message and you send one so your router transitions to OpenConfirm but i suspect the ISP router is showing OpenSent only because i don't think it is seeing your OPEN message.
Can you ping the ISP router without any drops ?
How are they connected ie. what media and are they peering on directly connected interfaces ?
Jon
12-19-2013 08:24 AM
Yes, I can ping the neighbor address without drops. We connect to them over fiber, and they are peering on directly connected interfaces.
12-20-2013 07:28 AM
Sorry... got the addresses backwards when posting the debug output. W.X.Y.A is the remote end and W.X.Y.Z is the local end
12-20-2013 07:35 AM
Can you post BGP config at your end ?
I did a bit of googling and found a few with a similiar problem and the only way they fixed it was to reboot the router. Suspect that is not what you wanted to hear and i don't like that as a solution as it doesn't really help you if you get the same issue again.
I know they will probably say no but are the ISP sure they made no changes at their end ?
Jon
12-20-2013 07:43 AM
Sure...
router bgp 64875
no synchronization
bgp log-neighbor-changes
network 208.108.168.0 mask 255.255.248.0
neighbor 192.153.37.89 remote-as 3112
neighbor 192.153.37.89 description OARnet-Internet2
neighbor 192.153.37.89 password
neighbor 192.153.37.89 soft-reconfiguration inbound
neighbor 192.153.37.89 route-map AS3112-IN in
neighbor 192.153.37.89 route-map AS3112-OUT out
neighbor 199.18.194.89 remote-as 600
neighbor 199.18.194.89 description OARnet
neighbor 199.18.194.89 password
neighbor 199.18.194.89 soft-reconfiguration inbound
neighbor 199.18.194.89 route-map AS600-IN in
neighbor 199.18.194.89 route-map AS600-OUT out
no auto-summary
12-20-2013 07:47 AM
And the ISP does seem to be implying that no changes have been made on their side...
12-23-2013 12:49 PM
Today I did a 'debug ip bgp keepalives' and it looks like the troubled session is showing keepalives sent, but none received, which is pretty much what the ISP is seeing on their end.
12-23-2013 03:02 PM
Are interface counters incrementing on your end? Also any errors on the interface? Have you tried to clear BGP?
Sent from Cisco Technical Support iPhone App
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: