Routing Problem if I change the route priority

[richard.raehal]

Hi Everyone,

I inherited a network that was set up by our corporate office when our local branch was sold.  I just recently started studying for the CCNA and have complete the first Cisco Network Academy class.  I am farily new to routers and seem to be overlooking something.

We have a dual link (or bonded) T1.  Today the ISP's side of the network started having problems.  I would see high ping times and lost packets.  The overall result is a very slow connection to any internet based service.  We are using a few SaaS applications.

If I administrativly down the interface ( MFR1.500 ) I switch to my backup connection and the problem goes away.  I lose all of my Web services when I do this because my public IP's are all using the T1.  I can't keep the T1 down.

My route table looks like this from the configuration and everything works perfectly.

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 MFR1.500

ip route 0.0.0.0 0.0.0.0 x.x.x.x 250

ip route 10.0.0.0 255.0.0.0 Null0

ip route 10.46.0.0 255.255.128.0 MFR1.500

ip route 10.46.0.0 255.255.128.0 173.8.235.50 250

ip route x.x.x.x 255.255.255.255 MFR1.500 permanent

ip route x.x.x.x 255.255.255.255 MFR1.500 permanent

I deleted the route:

ip route 10.46.0.0 255.255.128.0 MFR1.500

I chaged change the route "ip route 10.46.0.0 255.255.128.0 173.8.235.50 250" so the 250 was not on the end to make it the primary route.

I can not route to the 10.46.0.0 network when I do that.  I am not sure why.  What should I check to determien why the route change fails.

I use a tunnel to connect to the 10.46.0.0 network.  It is up if I set the routes to the orighinal config but refuses to come up when I use the changed config.  If I use either config, I can ping the partner on the internet so I am not sure why the tunnel stays down.

[Richard Burts]

Richard

Without some information about the runnel it is difficult for us to tell you why routing does not work when you attempt to delete the primary route and to make the backup route become primary. (and since the primary route for 10.46 just points to the serial subinterface I am a little puzzled how it is going through a tunnel). Without knowing any details my first guess would be that you may be changing your end (and changing how you send data out) but the other end is still trying to use the original/primary route for return traffic and perhaps it is that mismatch that is causing problems.

HTH

Rick

[richard.raehal]

I see what you are saying.  If the T1 interface is disabled, I get a good tunnel and traffic.  The problem only occurs when I change the route with both interfaces active.  Shouldn't the traffic leave via the fast ethernet interface from my office to the remote office but return over the T1 if the original route is still in use from the remote office to my office?

Does this info help in any way?

I am still not 100% clear on how my tunnels are set up.  I was under the impression they are ipsec-isakmp tunnels that get built anytime they can connect over an interface.  We did have to add isakmp keepalives to the connection to keep it from flapping.  The loopback addres and interface MFR1.500 have the same IP address. 

rtrDenver#sh crypto isakmp sa

IPv4 Crypto ISAKMP SA

dst             src             state          conn-id slot status

63.x.x.x   65.x.x.x   QM_IDLE           1370    0 ACTIVE

My serial interfaces are configured like this.  This is the same of serial 0/1/0 and 0/2/0.

!

interface Serial0/1/0

mtu 4470

bandwidth 1536

no ip address

no ip redirects

no ip proxy-arp

ip flow ingress

ip flow egress

encapsulation frame-relay MFR1

no arp frame-relay

Here are the interfaces:

interface MFR1.500 point-to-point

ip unnumbered Loopback0

ip access-group Firewall in

no ip redirects

no ip proxy-arp

ip inspect Internet out

ip nat outside

ip virtual-reassembly

snmp trap link-status

no cdp enable

no arp frame-relay

frame-relay interface-dlci 500 IETF  

crypto map VPN

interface MFR1

mtu 4470

no ip address

ip flow ingress

ip flow egress

frame-relay multilink bid UXXXXXX

frame-relay lmi-type ansi

max-reserved-bandwidth 100

service-policy output QOS-OUT

interface Loopback0

ip address 65.x.x.x 255.255.255.240

[lgijssel]

Just heard about a new type of interface: a runnel Great typo Rick, I like it! +5 from your friend.

When looking at the minimal information provided, I also noticed the frequent use of an interface as the next-hop.

This is a dangerous practice which should be avoided. See the following (ancient) link:

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800ef7b2.shtml

My advice: Try to replace interface MFR1.500 by a valid next-hop-ip.

regards,

Leo

[Richard Burts]

Leo

I did type that didn't I;) thanks for the points and I should try to do better about proof reading my posts.

In general I share your concern about static routes that point to interfaces. But in this case the interface is a point to point serial interface and there really are no concerns about static routes that point to serial point to point.

The additional information that Richard provides shows that there are IPSec crypto maps. So this brings up the possibility that the IPSec negotiation was done and Security Associations formed when the routing was sending the traffic to 10.46 over the primary interface. changing the route so that it now goes out a different interface could be a problem with the security association on that interface did not include the traffic. And since there is NAT configured on the interfaces there is also some possibility that there is some NAT issue when the route is changed.

We need some more details from Richard if we are to understand the issue and suggest any solution.

HTH

Rick

[richard.raehal]

I will look over your article.  I think I figured out what is going on.  I just do not know how to change it.

The MFR1.500 interface is my primary interface.  The backup interface is where my second route tries to go out.

If the primary interface is up, the VPN tunnel builds over the primary interface.  If the primary interface goes down, the VPN tunnel builds over the secondary interface.  When I try to change routing, the route will nto work because the tunnel is on the wrong interface.

If I run the sh crypto isakmp sa command I see the tunnel over the primary interface as idle.  Is there a way Ic an force the tunnel to build over the secondary interface without turning off the primary interface?

[richard.raehal]

I tried adding the command below to my config.  I could see the tunnels try to build over the backup connection but the tunnel I wanted to connect never became idle.  FE0/1 is the connection to my backup internet connection.  My tunnels to my remote offices showed tunnels connected and idle.

crypto map VPN local-address FastEthernet0/1  and

crypto map VPN local-address 173.x.x.x (per the article about not using interface names)

The tunnel showed MM_No_STATE or MM_SA_SETUP.  I need to do a little more research.

[cadet alain]

Hi,

Could you run debug crypto isakmp when trying to initiate the tunnel on the backup interface.

Regards.

Alain

[richard.raehal]

When I enter the command "debug crypto isakmp" I simply get the message: Crypto ISAKMP debugging is on

All of my IPSEC tunnels show the varied states of MM_KEY_EXCH, MM_NO_STATE, and MM_SA_SETUP when I run that command.

I was reading this article:

http://www.cisco.com/en/US/tech/tk583/tk372/technologies_tech_note09186a00800949c5.shtml

I do not get the output this article indicates I should get.  I have also tried "debug ip packet" and used th eping command with no change in results as per this article: http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6057.shtml

I am not sure why my router seems to ignore these comamnds or provide the same output with the debug options on as off.

Most of my users have gone home for the day.  It appears out ISP has escalated the issue from my T1's to their DS3 provider.  Hopefully this will be resolved soon.

[Richard Burts]

Richard

Here are a couple of comments/suggestions.

- most of the debugs discussed have to do with when the router is negotiating ISAKMP or IPSec parameters and these only happen on a periodic basis (ranging from every hour to every x hours). So it is understandable that you do not see output shortly after you enter the debug command.

- if you want to restart the tunnel and negotiation of ISAKMP and IPSec parameters use these commands

clear crypto sa

clear crypto isakmp sa

HTH

Rick

[cadet alain]

Hi Rick,

yes indeed I forgot to mention this fact, it's late here in Belgium and my brain cells are a little bit  fuzzy 

Thanks for correcting me.

Regards.

Alain

[richard.raehal]

I am leaving for the day so I won't be troubleshooting anymore until Monday.

I did try some suggestion to try and get the debug info but couldn't seem to get any information that is different.

I deleted the routes to the 10.46.x.x network (primary and secondary)

I changed the local VPN source to my Fast Ethernet Interface

I added a route to the 10.46.x.x network (primary route for backup connection)

I check the status of the IP tunnels.  They were all in the states as mentioned above.

I turned on debugging using "debug crypto ipsec" and "debug crypto isakmp"

I ran the command "clear crypto sa" and "clear crypto isakmp sa"

The command "clear crypto isakmp sa" did not work.  I had to change it to"clear crypto isakmp standby" and "clear crypto isakmp active"

I still did not get any output on the terminal.

Running the command "sh crypto isakmp sa" or "sh crypto ipsec" did not yeild any changed results.  The output for both command matched the output wehen debug was off.

If there a way to disable debugging overall?  I was curiosu because when I turn debugging on, the CPU and Memory usage do not change.  I though IP Packet debugging, another command I tried, would cause a high utilization on the router.

Thanks for your help.  I have learned a lot today.

[Richard Burts]

Richard

Here are several observations and suggestions:

- to disable debugging overall use this command

undebug all

- am I correct in understanding that you are not seeing any output of syslog messages in your terminal session? If so my first suggestion would be to be sure that you issue the command terminal monitor after you log in to the router. This will generate copies of syslog messages for your terminal session. my second suggestion would be to check the setting of the severity level in your logging command. It is possible that the logging is set to a severity level higher than debug and this would result in not seeing any debug output.

There are a couple of ways to check the severity level of logging. You could use show running-config | include logging

This would show all commands in the config that have the word logging in them. So you look for the command that enables logging monitor.

I realize as I type this that I am assuming that logging monitor is enabled. But if it were not enabled that would be a cause of why you are not seeing syslog output in your session.

So the best way to check is to look at the lines displayed at the beginning of show log. In the lines beginning the display it tells each of the logging destinations and their severity level.

If this does not make good sense to you then just do show logging and post the first page of output. That should get us what we need to see how your syslogging is configured.

- I think that I see what is possibly the reason why things do not work when you just try to change the routing. I see this in the configuration of the serial MFR1.500 interface

ip inspect Internet out

If you are inspecting this traffic it is checking incoming traffic to see if it is in response to something that went out that interface. If the original traffic went out the Ethernet and the response comes back in the MFR1 interface then the inspection would not see it as a response and would probably deny it.

Bottom line is that I believe that the failover seems to work ok if the serial interface just fails and passes no traffic. But if you want to change the routing so that 10.46 is sent over the other interface then you probably will need to make changes on both ends to get this to work.

HTH

Rick

[richard.raehal]

Thank you again for your help.  The ISP service has been restored so I am running live again.  I will try working with the routing and tunnels again sometime when I have a maintenance window.

I see these logging lines so I will read about these commands. 

logging buffered 50000

logging console critical

no logging trap

logging synchronous (for the console and aux ports)

The only line that has the word debug is this one:

service timestamps debug datetime localtime

[Richard Burts]

Richard

Thanks for the update. When you have an opportunity to work with routing and the tunnels again, if there is still a problem it would be helpful to us if you would give us a fresh statement of what the problem is.

HTH

Rick