I have an issue and I am hoping someone can help.
The example is we have a HQ with two Cisco 1841 routers
Router A 10.0.0.101
Router B 10.0.0.102
The branch routers are
Router C 192.168.104.2
Router D 192.168.104.3
Router A and C are linked together via a GRE/IPSEC tunnel over the internet running EIGRP.
Router B and D are linked together via and MPLS network with a GRE/IPSEC tunnel running EIGRP as well.
At the branch the default router is router D and at HQ router B.
Failover works absolutely fine and when we lose the main MPLS link our data goes over the internet based link between router A and C.
The problem we have is that we are passing SMTP and POP3 internal mail server traffic from and to the branch via the Internet based link using route maps.
The route maps on routers B and D point the appropiate traffic via a set ip next-hop to Router A at HQ and then Router A points the traffic via another route map on this router down the internet link.
This works great and keeps the MPLS network for apps that need the lower latency. However if we lose the internet link our e-mail stops working.
Is the traffic getting into some kind of loop where the first route map sends it to router A and then router A sends it back to B as the tunnel via the internet is down ?.
Is there a way round this. Can I advertise the route-map somehow so that when the link is down the route-map is not advertised so our e-mail will go back down the MPLS again.
I hope this all makes sense and I have attached part of the config.
kindly check when your internet link goes down whether you have the valid routing entry on router B and D for the next-hop that you have configured. The next-hop atribute of PBR works with the following rule:-
The set ip next-hop command verifies the existence of the next hop specified,andif the next hop exists in the routing table, then he command policy routes the packet to the next hop.
if the next hop does not exist in the routing table, the command uses the normal routing table to forward the packet.
I am not sure that your issue is a loop as much as it may be a black hole. Without seeing some more details in your config we can not know for sure. But I believe that the issue is that the route maps continue to operate and forward traffic even when the Internet based tunnel can not deliver traffic and producing a sort of black hole.
One aspect of GRE tunnels is that by default the IOS will report the tunnel up/up as long as it has a valid route in its routing table to the tunnel destination address. So I guess that the tunnel still reports as up and the route map continues to forward but the traffic can not be delivered.
There are several approaches that you might use to resolve this issue. One alternative is an enhancement in PBR to use object tracking. You could track availability of the destination address at the other site and if it became unavailable then PBR would stop forwarding packets over the Internet tunnel. While that should work, I would suggest a slightly different alternative.
The first element in my suggestion is that the route map on the router that is connected to MPLS should not use the next hop of the connected router but should use the next-hop of the other routers tunnel destination (assuming that the tunnel destination is in the routing table). Originally in PBR the next hop was required to be on a connected interface, but in recent IOS Cisco has introduced an enhancement that supports recursive lookup of the next hop. The second element of my suggestion is that you configure GRE tunnel keepalive (another enhancement in recent IOS). With GRE tunnel keepalive the IOS sends keepalive over the GRE tunnel and expects to receive the keepalive in response. If the response is not received (for x many attempts) then IOS marks the tunnel as down and any routes lerned over the tunnel are withdrawn.
So if you adopt my suggestions the GRE tunnel keepalive will remove the remote address is the Internet based tunnel stops forwarding. And the PBR using recursive lookup for the tunnel destination address would stop forwarding if the Internet based GRE tunnel stops working.
Give it a try and let us know how it works.
Firstly I tried to set
ROUTER A so that it no longer had a route-map to go to Tunnel3 and i set ROUTER B so that it went to
set ip next-hop recursive 192.168.144.1 (other end of the GRE TUNNEL VIA ROUTER A)
and on ROUTER C i got rid of the router map setting interface tunnel0 and on ROUTER D
set ip next-hop recursive 192.168.144.2 ( other end of the GRE TUNNEL VIA ROUTER C)
However I could no longer route e-mail so I tried again with the detsination of the other end of each GRE Tunnel
ROUTER A Public IP 126.96.36.199
ROUTER C Public IP 188.8.131.52
So i set the following on Router B
set ip next-hop recursive 184.108.40.206
and on ROUTER D
set ip next-hop recursive 220.127.116.11
But still no joy and I was unsure how that could work as they are external. The only way to get it to use the internet line was to go back to set interface tunnel?. But this grings me back to it no failing over.
If someone needs me to send full configs I can
I believe that our responses have been crossing each other in transit. And I believe that you have not fully understood my suggestion. I am not saying that router A no longer needs a route map - in fact I believe that router A does still need the route map - otherwise the routing table on router A probably points to the MPLS network as the way to get there. My suggestion is that router B should specify the tunnel address (which from this response I assume would be 192.168.144.1).
Without the route map on router A it will no longer forward e-mail over the Internet GRE tunnel but will forward back to router B according to the routing table - and therefore does create a loop. Which is what you describe.
The advantage of the route map on router B pointing to the remote tunnel interface (recursive lookup - along with the tunnel keepalive) is that router B will stop forwarding to router A if the Internet based GRE tunnel stop working. If router A get the e-mail packet then it still needs the route map to forward it over the Internet based GRE tunnel.
So it is the combination of 2 features (GRE recursive lookup along with GRE keepalive) that is the essence of my suggestion. So if I am understanding your topology correctly (which I am not sure about) router B should specify next-hop as the tunnel interface on router C (not the tunnel destination address but the tunnel interface address - which might be 192.168.144.1)
If you have already gone to bed, then perhaps after a night's sleep this will make better sense. If it does not, or if you try it and it still does not work , then I suggest that you post full configs.
Hi the other router would be there say router A or C so are you saying the next hop should be the tunnel interface on router A for instance although I think this is not possible in ios as it has to been router A's Lan address.
I am not clear whether you are responding to Mohmmad or to me. The text of the response seems more to deal with my response than Mohmmad's so I will go from there.
You are correct that the traditional requirement in PBR was that a next-hop must be a connected interface. But as I stated in my response PBR now supports a recursive lookup and the next-hop can be an address other than a connected interface as long as the specified next-hop is in the routing table.
You should see the response I have just posted before seeing this one.
My routing tables do show the 192.168.144 network in there as it is being shown via eigrp but would I need to put a manual route in their just for 192.168.144.1 on ROUTER D via ROUTER C and on ROUTER B for 192.168.144.2 VIA router A
I hope that makes sense as I am pretty tired now and off to bed.
Thanks for all your help
I believe that I have addressed most of the issues here in my other response. But in terms of whether you would need a specific route for 192.168.144.1 I believe that you would not need that specific route. The GRE keepalive should control whether the 192.168.144.0 appears in the routing table or not. And the GRE keepalive would satisfy the requirement on router B to know whether to policy route to the tunnel on router A or not.
After waking up this morning I tried again and I put in recursive next-hop in and setup the GRE keepalives. I had the PBR working fine which I did not last night but the failover did not work.
I then tried adding another set ip next-hop to the other end of the Local GRE tunnel at both ends.
set ip next-hop recursive 192.168.144.2
set ip next-hop 192.168.134.2
set ip next-hop recursive 192.168.144.1
set ip next-hop 192.168.134.1
When i did this hey presto the failover works. Under normail circumstances stmp and pop3 gover the 144 internet GRE link and then when this is down it goes dow the MPLS link 134 GRE link.
Thanks for all your help it is appreciated
I am glad that my suggestion was able to help you resolve your problem. The forum can be very useful and I encourage you to continue your participation in the forum.