cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2080
Views
0
Helpful
12
Replies

BGP session having trouble all of a sudden on 7204 router

spfister336
Level 2
Level 2

We have a 7204 router that has two BGP sessions to our upstream provider. The first one provides us a single default route, and the second one provides us a small list of Internet2 related routes. This setup has been working just fine for many years, but recently the second, Internet2 one has been having trouble. Both sides are trying to connect, but it never establishes. It mainly shows OpenConfirm when I check status. Nothing has been changed on this router in years, and no one has logged into it in many months. Our provider insists that the problem in on our end and they say they're not receiving keepalives from our side, but it looks like they're being sent. What can we do?

12 Replies 12

Harold Ritter
Cisco Employee
Cisco Employee

Hi,

Do you see any clue in the log?

Regards

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

Not sure... I've looked at a lot of output, I'm not sure what the problem could be. If I post 'debug ip bgp' output, would that help?

Jon Marshall
Hall of Fame
Hall of Fame

Have you run debugging "debug ip bgp" to see exactly what is happening when they try to establish a connection ? Obviously debugging can have an adverse effect on the router so be careful when/if you run it ?

OpenConfirm is almost all the way there. Either a notification message can be sent which transitions the state to idle or a keepalive is not received which again transitions it to idle.

Jon

Here's 'debug ip bgp'... W.X.Y.A is the local end, W.X.Y.Z is the remote end

Dec 18 15:58:30.278: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active delayed 32829ms (35000ms max, 28% jitter)   

Dec 18 15:59:03.110: BGP: W.X.Y.A open active, local address W.X.Y.Z

Dec 18 15:59:33.110: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active

delayed 28160ms (35000ms max, 28% jitter)   

Dec 18 15:59:56.323: BGP: W.X.Y.A passive open to W.X.Y.Z          

Dec 18 15:59:56.323: BGP: W.X.Y.A went from Active to Idle               

Dec 18 15:59:56.327: BGP: W.X.Y.A went from Idle to Connect              

Dec 18 15:59:56.327: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44

Dec 18 15:59:56.327: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds

Dec 18 15:59:56.327: BGP: W.X.Y.A went from Connect to OpenSent          

Dec 18 15:59:56.327: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds                                     

Dec 18 15:59:56.327: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34   

Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6                                            

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4  

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1  

Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2                                            

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families                           

Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2                                            

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0  

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families                          

Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8                                            

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6 

Dec 18 15:59:56.327: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6                                            

Dec 18 15:59:56.327: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4 

Dec 18 15:59:56.327: BGP: W.X.Y.A unrecognized capability code: 65 - ingored                                                                

BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112                                 

Dec 18 15:59:56.327: BGP: W.X.Y.A went from OpenSent to OpenConfirm      

Dec 18 15:59:56.327: BGP: W.X.Y.A send message type 1, length (incl. header) 45                                                                   

Dec 18 16:01:26.327: BGP: W.X.Y.A connection timed out - has not accepted a message from us for 90000ms (hold

time), 0 messages pending transmition      

Dec 18 16:01:26.327: BGP: W.X.Y.A went from OpenConfirm to Closing       

Dec 18 16:01:26: %BGP-3-NOTIFICATION: sent to neighbor W.X.Y.A 4/0 (hold time expired) 0 bytes                                

Dec 18 16:01:26.327: BGP: W.X.Y.A send message type 3, length (incl. header) 21                                                                   

Dec 18 16:01:26.327: BGP: W.X.Y.A local error close after sending NOTIFICATION                                                               

Dec 18 16:01:26.327: BGPNSF state: W.X.Y.A went from nsf_not_active to nsf_not_active                                                   

Dec 18 16:01:26.327: BGP: W.X.Y.A went from Closing to Idle              

Dec 18 16:01:26.327: BGP: W.X.Y.A closing                                

Dec 18 16:01:27.331: BGP: W.X.Y.A went from Idle to Active               

Dec 18 16:01:27.331: BGP: W.X.Y.A open active delayed 30201ms (35000ms max, 28% jitter)                                                  

Dec 18 16:01:57.535: BGP: W.X.Y.A open active, local address W.X.Y.Z

Dec 18 16:02:27.536: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active

delayed 32450ms (35000ms max, 28% jitter)   

Dec 18 16:02:59.988: BGP: W.X.Y.A open active, local address W.X.Y.Z

Dec 18 16:03:29.988: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active

delayed 26383ms (35000ms max, 28% jitter)   

Dec 18 16:03:54.328: BGP: W.X.Y.A passive open to W.X.Y.Z          

Dec 18 16:03:54.328: BGP: W.X.Y.A went from Active to Idle               

Dec 18 16:03:54.328: BGP: W.X.Y.A went from Idle to Connect              

Dec 18 16:03:54.332: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44                                                                      

Dec 18 16:03:54.332: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds

Dec 18 16:03:54.332: BGP: W.X.Y.A went from Connect to OpenSent          

Dec 18 16:03:54.332: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds                                     

Dec 18 16:03:54.332: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34   

Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6                                            

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4  

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1  

Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2                                            

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families                           

Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2                                            

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0  

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families                          

Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8                                            

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6 

Dec 18 16:03:54.332: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6                                            

Dec 18 16:03:54.332: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4 

Dec 18 16:03:54.332: BGP: W.X.Y.A unrecognized capability code: 65 - ingored                                                                       

BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112                                 

Dec 18 16:03:54.332: BGP: W.X.Y.A went from OpenSent to OpenConfirm      

Dec 18 16:03:54.332: BGP: W.X.Y.A send message type 1, length (incl. header) 45                                                                   

Dec 18 16:05:24.329: BGP: W.X.Y.A rcv message type 3, length (excl. header) 2                                                                        

Dec 18 16:05:24: %BGP-3-NOTIFICATION: received from neighbor W.X.Y.A 4/0 (hold time expired) 0 bytes                    

Dec 18 16:05:24.333: BGP: W.X.Y.A went from OpenConfirm to Closing       

Dec 18 16:05:24.333: BGPNSF state: W.X.Y.A went from nsf_not_active to nsf_not_active                                                   

Dec 18 16:05:24.333: BGP: W.X.Y.A went from Closing to Idle              

Dec 18 16:05:24.333: BGP: W.X.Y.A closing                                

Dec 18 16:05:24.433: BGP: W.X.Y.A went from Idle to Active               

Dec 18 16:05:24.433: BGP: W.X.Y.A open active delayed 34021ms (35000ms max, 28% jitter)                                                  

Dec 18 16:05:33: %SEC-6-IPACCESSLOGP: list 110 denied tcp 10.2.48.85(443) (GigabitEthernet2/0.738 b0c6.9ae6.552d)

-> 208.108.175.254(50390), 1 packet          

Dec 18 16:05:58.457: BGP: W.X.Y.A open active, local address W.X.Y.Z

Dec 18 16:06:28.457: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active

delayed 28111ms (35000ms max, 28% jitter)   

Dec 18 16:06:56.570: BGP: W.X.Y.A open active, local address W.X.Y.Z

Dec 18 16:07:26.570: BGP: W.X.Y.A open failed: Connection timed out; remote host not responding, open active

delayed 33307ms (35000ms max, 28% jitter)   

Dec 18 16:07:52.334: BGP: W.X.Y.A passive open to W.X.Y.Z          

Dec 18 16:07:52.334: BGP: W.X.Y.A went from Active to Idle               

Dec 18 16:07:52.334: BGP: W.X.Y.A went from Idle to Connect              

Dec 18 16:07:52.334: BGP: W.X.Y.A rcv message type 1, length (excl. header) 44                                                                      

Dec 18 16:07:52.338: BGP: W.X.Y.A rcv OPEN, version 4, holdtime 90 seconds

Dec 18 16:07:52.338: BGP: W.X.Y.A went from Connect to OpenSent          

Dec 18 16:07:52.338: BGP: W.X.Y.A sending OPEN, version 4, my as: 64875, holdtime 180 seconds                                     

Dec 18 16:07:52.338: BGP: W.X.Y.A rcv OPEN w/ OPTION parameter len: 34   

Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6                                            

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 1, length 4  

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has MP_EXT CAP for afi/safi: 1/1  

Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2                                            

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 128, length 0

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(old) for all address-families                           

Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 2                                            

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 2, length 0  

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has ROUTE-REFRESH capability(new) for all address-families                          

Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 8                                            

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 64, length 6 

Dec 18 16:07:52.338: BGP: W.X.Y.A rcvd OPEN w/ optional parameter type 2 (Capability) len 6                                            

Dec 18 16:07:52.338: BGP: W.X.Y.A OPEN has CAPABILITY code: 65, length 4 

Dec 18 16:07:52.338: BGP: W.X.Y.A unrecognized capability code: 65 - ingored                                                                       

      BGP: W.X.Y.A rcvd OPEN w/ remote AS 3112                                 

Dec 18 16:07:52.338: BGP: W.X.Y.A went from OpenSent to OpenConfirm      

Dec 18 16:07:52.338: BGP: W.X.Y.A send message type 1, length (incl. header) 45

Harold may be able to provide a better analysis but it looks like your router is having problems connecting to the ISP router ie. from the debug -

1) your router tries an active open to the ISP router but it keeps failing

2) only when the ISP router initiates the connection does anything happen

3) You receive an OPEN message and you send one so your router transitions to OpenConfirm but i suspect the ISP router is showing OpenSent only because i don't think it is seeing your OPEN message.

Can you ping the ISP router without any drops ?

How are they connected ie. what media and are they peering on directly connected interfaces ?

Jon

Yes, I can ping the neighbor address without drops. We connect to them over fiber, and they are peering on directly connected interfaces.

Sorry... got the addresses backwards when posting the debug output. W.X.Y.A is the remote end and W.X.Y.Z is the local end

Can you post BGP config at your end ?

I did a bit of googling and found a few with a similiar problem and the only way they fixed it was to reboot the router. Suspect that is not what you wanted to hear and i don't like that as a solution as it doesn't really help you if you get the same issue again.

I know they will probably say no but are the ISP sure they made no changes at their end ?

Jon

Sure...

router bgp 64875

no synchronization

bgp log-neighbor-changes

network 208.108.168.0 mask 255.255.248.0

neighbor 192.153.37.89 remote-as 3112

neighbor 192.153.37.89 description OARnet-Internet2

neighbor 192.153.37.89 password

neighbor 192.153.37.89 soft-reconfiguration inbound

neighbor 192.153.37.89 route-map AS3112-IN in

neighbor 192.153.37.89 route-map AS3112-OUT out

neighbor 199.18.194.89 remote-as 600

neighbor 199.18.194.89 description OARnet

neighbor 199.18.194.89 password

neighbor 199.18.194.89 soft-reconfiguration inbound

neighbor 199.18.194.89 route-map AS600-IN in

neighbor 199.18.194.89 route-map AS600-OUT out

no auto-summary

And the ISP does seem to be implying that no changes have been made on their side...

Today I did a 'debug ip bgp keepalives' and it looks like the troubled session is showing keepalives sent, but none received, which is pretty much what the ISP is seeing on their end.

joshuacmoore
Level 1
Level 1

Are interface counters incrementing on your end? Also any errors on the interface? Have you tried to clear BGP?

Sent from Cisco Technical Support iPhone App

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco