Dual WAN 1841 Router troubleshooting

[IM-Design]

Hi,

We have an 1841 router with both WAN ports being used. The router will failover and failback between the 2 ports when either one loses connectivity (using object tracking).

At the moment we are experiencing a problem that the ISP insists lies with the router. We are contending that the router is behaving properly.

The issue is that we cannot successfully ping both ports on the router when they are both up and connected. We have enabled debugs on icmp packets and can confirm that the router sends replies to all of the pings on both ports, but the replies from one of the WAN ports never reach their intended destination (and the pings time out). The replies from the other WAN port are received successfully.

From the debug monitoring we have done we are of the opinion that the packets are properly leaving the router on the way back to their origin and they are being dropped somewhere after they leave the router.

How can we prove thie definitively?

Is there anything else that we can do to isolate each WAN port so that the 1841 behaves more like a pair of single-wan-port routers?

Any other ideas?

TIA

[Edison Ortiz]

Charlie,

Can you post configs ?

Thanks

[Richard Burts]

Charles

From your description of failover and failback and object tracking I am assuming that you have the router configured with a primary default route and a backup (perhaps floating) default route. Your description does not tell us whether both interfaces connect to the same provider or different. Perhaps you can clarify what your routing logic and the topology is?

I am guessing that ping to the interface with the primary default route works and that ping to the interface with the backup route does not work. If that is the case then I would believe that the issue is that when you ping the backup interface the ping response is being sent out the primary interface (since it has the active default route) and that probably the provider is rejecting the packet since it has a source address that does not belong on that interface.

If that does not explain the behavior then perhaps you can give us some more detail about your environment.

HTH

Rick

[IM-Design]

Hi,

Yes, as you can see from the config posted above, the primary and backup routes are there. Both interfaces connect to the same provider, with 2 different /29 blocks (.146 next hops to .145 and .162 hops to .161).

The idea of the system is that when both interfaces are 'up', traffic will prefer the primary interface. When either interface is down, traffic will be routed out the other interface, which will most likely be up. Weve successfully tested this and we are pleased with the results. But the inability to ping one port makes it unlikely that we will be able to maintain a VPN connectivity with the router on at least one if not both ports.

I can understand that the icmp replies may well be getting routed out the primary default route. But since both interfaces are with the same provider, it seems unlikely that they would reject packets that they have assigned. Would you agree? What would be the best way to test this?

Also, to further confound, there was a time when I could successfully ping both addresses from some source IPs, and when I tried from a different source IP, the failure of one port to reply is seen. On the router (with debug ip icmp on), both sets of packets are replied to.

Charlie

[IM-Design]

Here is the config:

********************************************

...

!

security authentication failure rate 3 log

security passwords min-length 6

logging buffered 51200 warnings

!

no aaa new-model

!

resource policy

!

clock timezone UTC -5

clock summer-time UTC recurring

ip cef

!

track 123 rtr 1 reachability

track 456 rtr 2 reachability

!

interface Null0

no ip unreachables

!

interface FastEthernet0/0

description PS WAN$ETH-WAN$

ip address x.x.x.146 255.255.255.248

ip access-group PSAcl in

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat outside

ip inspect SDM_LOW out

ip virtual-reassembly

duplex auto

speed auto

!

interface FastEthernet0/1

description SB WAN$ETH-WAN$

ip address x.x.x.162 255.255.255.248

ip access-group SBAcl in

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat outside

ip inspect SDM_LOW out

ip virtual-reassembly

duplex auto

speed auto

!

interface FastEthernet0/0/0

switchport mode trunk

!

interface Vlan1

description Default Vlan

ip address 10.1.10.1 255.255.255.0

ip access-group sdm_vlan1_in in

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat inside

ip virtual-reassembly

!

!

ip route 0.0.0.0 0.0.0.0 x.x.x.145 track 123

ip route 0.0.0.0 0.0.0.0 x.x.x.161 track 456

!

ip http timeout-policy idle 60 life 86400 requests 10000

ip nat inside source route-map PSFA00 interface FastEthernet0/0 overload

ip nat inside source route-map SBFA01 interface FastEthernet0/1 overload

!

ip access-list extended DenyNAT

deny ip 10.1.10.0 0.0.0.255 192.168.0.0 0.0.0.255

permit ip 10.1.10.0 0.0.0.255 any

ip access-list extended PSAcl

permit tcp any host x.x.x.146 eq 3389

permit tcp any host x.x.x.146 eq 5900

deny ip 10.1.10.0 0.0.0.255 any

permit icmp any any

permit tcp any host x.x.x.146 eq 22

permit tcp any host x.x.x.146 eq 8443

deny ip 10.0.0.0 0.255.255.255 any

deny ip 172.16.0.0 0.15.255.255 any

deny ip 192.168.0.0 0.0.255.255 any

deny ip 127.0.0.0 0.255.255.255 any

deny ip host 255.255.255.255 any

deny ip any any log

ip access-list extended SBAcl

permit tcp any host x.x.x.162 eq 3389

permit tcp any host x.x.x.162 eq 5900

deny ip 10.1.10.0 0.0.0.255 any

permit icmp any any

permit tcp any host x.x.x.162 eq 22

permit tcp any host x.x.x.162 eq 8443

deny ip 10.0.0.0 0.255.255.255 any

deny ip 172.16.0.0 0.15.255.255 any

deny ip 192.168.0.0 0.0.255.255 any

deny ip 127.0.0.0 0.255.255.255 any

deny ip host 255.255.255.255 any

deny ip any any log

ip access-list extended sdm_vlan1_in

remark SDM_ACL Category=1

deny ip x.x.x.0 0.0.0.255 any

deny ip host 255.255.255.255 any

deny ip 127.0.0.0 0.255.255.255 any

permit ip any any

!

access-list 1 remark INSIDE_IF=Vlan1

access-list 1 remark SDM_ACL Category=16

access-list 1 permit 10.1.10.0 0.0.0.255

access-list 100 remark VlanInbound

access-list 100 deny ip host 255.255.255.255 any

access-list 100 deny ip 127.0.0.0 0.255.255.255 any

access-list 100 permit ip any any

access-list 101 deny ip 10.1.10.0 0.0.0.255 192.168.0.0 0.0.0.255

access-list 101 permit ip 10.1.10.0 0.0.0.255 any

!

route-map PSFA00 permit 10

match ip address 101

match interface FastEthernet0/0

!

route-map SBFA01 permit 10

match ip address 101

match interface FastEthernet0/1

!

...

end

********************************************

Ive removed/edited it a bit for the sake of anonymity.

[Richard Burts]

Charlie

Thanks for posting the config. I understand the need to protect private information and hope that nothing that you removed/edited changes the presentation of the problem.

In your description you have been talking about primary and backup. But what I see in the config is 2 equal cost static default routes. I do not see anything that leads one interface to be primary and the other to be backup. Is there something that I am not seeing that does create a primary and a backup?

Can you let us know which interface does respond to ping and which does not?

In addition to testing with ping I wonder if you could test with traceroute (or tracert if from a Windows platform) to both addresses and post the output. It would be interesting to see if both take the same path and at what point the traceroute stops.

As another way of testing I wonder if you could configure a static route for some destination network or subnet pointing to the interface that does not respond and then ping to some destination in that network or subnet. It would be helpful to know if packets sent out that interface make it to external destinations. I would suggest that you first do a ping to the destination to demonstrate that it does respond to ping, then configure the new static route, then ping the destination when we know that we are sending traffic out the particular interface. Remember to remove the new static route when the test is completed.

HTH

Rick

[IM-Design]

Rick,

Thanks for the response.

Yes I believe that Ive included all of the relevant details for the config.

I understand that the 2 default routes can have the same cost or different costs,and that the router will use that cost to determine how to route.

At this moment the entire setup is in a bit of disarray since Ive lost connectivity to one port entirely. Ill be travelling next week to be onsite to resove that problem. Which is why Ive opened this conversation on the forum. I want to be prepared to troubleshoot exhaustively the problems we have been seeing here.

Our ISP is providing service to both ports. They have supplied 2 identical modem devices, one connected to each port. These devices are defined as the next hop for each port. The troubleshooting goal is to determine where the icmp packets (and by extension, other traffic) is/are being dropped and to provide a configuration which resolves this problem.

We have done a lot of pinging and tracerouting to try to see where these packets are being dropped. One of the limitations we have is that the modem devices are not able to monitor at the packet level. So, Im planning to tap the cable between the router port and each modem and capture the traffic. To date, we have been able to enable the ip icmp debug commands on the router. In doing so, we can see that the router responds to all icmp requests with an icmp reply. So, its our sense that the modem devices are blocking or dropping the reply packets that are failing to make it to their request origins.

I like the idea of a static route. In fact, we have these configured to support the SLA/Object tracking, since we are using pings to detect connectivity. At the moment this portion of the config has been removed.

What I would like to do is to have the router in a 'base' configuration, with both modems connected and traffic flowing. Then I would like to re-add the various aspects such as failover, etc.. How could I isolate the WAN ports from one another yet allow traffic to pass form the default vlan? Id like to have these 2 WAN ports look as much like 2 separate routers as possible.

Charlie

[Richard Burts]

Charlie

As you are probably aware we can not fully separate the 2 WAN ports. I would suggest that the closest that you can come to that is to use Policy Based Routing. If you want to do this I would suggest this approach:

- make one of your static default routes (and its associated output interface) the preferred or primary default route by adding an administrative distance to the other static route. You can verify that this is working by doing show ip route. Right now if you do show ip route you should see 2 default routes in the routing table. After you add an administrative distance to one default route there should be only 1 default route in the routing table. This will send most traffic out the designated interface.

- then configure PBR to send some specified traffic out the other interface. You would configure a route map which would use an access list to match some traffic (perhaps by source address or perhaps by protocol type) and would set next-hop to point out the other outbound interface.

This would send certain traffic out each interface and is as close as I know how to do to logically separate the interfaces.

I would suggest that one of the things that you ought to do (perhaps before you travel) is to try to determine if the other interface is passing traffic. I would probably start with testing to determine if you can ping the next hop address. And if that works I would try my suggestion of a static route for some particular destination (perhaps a static route for the address of www.cisco.com) pointing to the other interface. And try ping or try the web browser and see if you can access that destination.

HTH

Rick

[IM-Design]

Thanks for that Rick,

Would it be practical/possible to use PBR to route traffic according to the WAN port through which it is initiated (inbound or outbound).

If this is load-balancing by session, then I think that we are already doing this.

Can you point me to a good starter reference for PBR?

Thanks

[Richard Burts]

Charlie

I am not sure that I understand the question in the first part of your post: "route traffic according to the WAN port through which it is initiated". Since you are doing NAT on outbound traffic, the response will come back to the same interface it went out on. Choosing which interface to use for outbound traffic is what we have been talking about doing with Policy Based Routing. And if someone outside initiates traffic to you it will arrive on which ever interface your provider chooses and I do not believe that you have control over that. If this does not answer what you were asking then perhaps you can clarify.

This white paper may be a good place to start in understanding Policy Based Routing:

http://www.cisco.com/en/US/products/ps6599/products_white_paper09186a00800a4409.shtml

And this link discusses configuration of Policy Based Routing. Look especially at the section about using set ip next-hop:

http://www.cisco.com/en/US/tech/tk364/technologies_configuration_example09186a00801f3b54.shtml#casetwo

HTH

Rick

[Richard Burts]

Charlie

You had indicated that you would be interested in configuring the interfaces so that they essentially acted like two separate routers. I did not believe that you could do this. Here is a link to an interesting posting on the forum in which Martin says that configuring vrf will do that. I have not used it and can not speak for it but Martin is good and usually knows what he is talking about.

http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=Getting%20Started%20with%20LANs&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.1ddcea0e

HTH

Rick

[Gerard Roy]

Please post your config again. Do you want a primary path and then a backup path or both to run at the same time? At the same time is almost impossible. I do not see your rtr 1 and rtr 2 statements in your config.

This is what Primary and Backup ISP should look like.

Primary path:

ip route x.x.98.0 255.255.254.0 FastEthernet0 x.x.99.1 track 456

Secondary Path:

ip route x.x.98.0 255.255.254.0 x.2.2.1 10

track 123 rtr 1 reachability

delay down 180

!

track 234 rtr 2 reachability

delay down 180

!

track 456 list threshold weight

object 123 weight 5

object 234 weight 5

threshold weight up 5

ip sla 1

icmp-echo 4.2.2.1 source-interface FastEthernet0

timeout 1000

threshold 100

frequency 5

ip sla schedule 1 life forever start-time now

ip sla 2

icmp-echo 4.2.2.2 source-interface FastEthernet0

timeout 1000

threshold 100

frequency 5

ip sla schedule 2 life forever start-time now

[IM-Design]

Hi Rick,

I looked at the link, but have not done any real reading on it yet.

Anyway, Here is where I am with this issue:

-Im onsite with the 1841 and set up packet sniffing between the 2 wan ports on the 1841 and their next hop ports. During the sniffing I continually pinged both interfaces from 3 different source addresses. These were not extended pings (could they have been?), I actually had 2 command windows open in 3 different locations and pinged the WAN ports.

Also during the sniffing I enabled icmp debugging on the router and was able to see the router replies being sent out. The problem has been that the ping-replies were never making it back to their originator (Time-out) for one or the other WAN port.

So, I modified the default routes slightly by adding a cost to one. Lo and behold I was able to get the pings to succeed by changing the costs of the 2 default routes.

What this suggests to me is that the 1841 is replying to ping-request packets coming in over port 0/0 by sending a ping reply-packet over port 0/1. Ping-reply packets to ping-requests arrriving over port 0/1 are returning over port 0/1. This was confirmed in the packet captures. It does not seem to make sense that the load-balancing (destination-based) would work this way, this appears to be more like packet-based load balancing. My load balancing is the default (ip cef) so I understand that this is the destination-based sort.

So, is it possible that my isp is blocking the packets that return over a different port than the request packet arrived? Is this standard practice.

TIA

[Richard Burts]

Charlie

Some clarification of what you report would be helpful. You report that you added a cost to one of the default routes then things worked. Did you test putting that default route back the way it was and putting a cost on the other default route? Does it work on either route if there is just one route in the table or is it that one route works and other does not?

Also I am not clear what you found when you were doing the packet capture between the 1841 WAN ports and their next hop. When things were not working did you see packets from the provider coming to the 1841 and did you see packet responses from the 1841 to the next hop (but not get to the destination)?

Understanding these aspects would be very helpful to us.

It is possible the the ISP does block packets that return over a different port - if they are doing stateful inspection of traffic that is quite possible. And it is a possibility that I raised in my first response in this thread.

One other clarification: your mention of per-destination or packet based forwarding is a factor for traffic passing through the router (traffic generated by some other device which the router is forwarding toward the destination). But for packets which the router is generating itself (like replying to ping) it will per-packet forward.

HTH

Rick

[IM-Design]

Rick,

I hope this clarifies:

I set up packet sniffing between each of the 1841 WAN ports and their respective 'next hops'. All traffic from the 2 WAN ports passes over these 2 lines. My test consisted of pings from 3 different IP addresses. What I saw in the packet captures is that the icmp reply packets (responding to icmp request packets directed at either WAN port) were sent out over only one port or the other. I was able to direct these packets to flow out over either port by modifying the costs associated with either of the 2 default routes.

At first the 1841 configuration had 2 equal-cost default routes. In this config, the icmp-replies were going out port B. If I modified the default route cost of port be to be higher than the default route cost of port A, I could force the packets out port A. If the cost of default route A was modified to be greater than the cost of default route B, icmp replies would go out of port B.

I have not tried removing one of the default routes altogether.

In all, the point of the sniffing exercise has been to conclusively determine that the packets are leaving the 1841 properly and to determine where they are being dropped on their return path.

I do not know if the ISP is blocking the icmp-replies that are sent over port B in response to icmp-requests that are received over port A, but it would appear so since these icmp-replies never reach the end point (causing a ping time-out).

Thanks for your last clarification. I was not aware that the load-balancing behavior of the router would be different for traffic which the router is generating itself.

We want to have VPN tunnels to a single remote endpoint on each of these WAN ports. Sometimes both tunnels would be 'Up', other times it would be only or the other. How will traffic flow over these tunnels?

Thanks

Charlie