SLA on 5505 with two static routes sending packets to wrong interface

[Massimo Giustiniani]

I have a Cisco 5505 8.4(1) version (50 users) so configured: 

dual static routes to two different ISP routers (192.168.30.1 and 192.168.30.2)

route outside 0.0.0.0 0.0.0.0 192.168.30.1 128 track 10
route outside 0.0.0.0 0.0.0.0 192.168.30.2 128 track 11

with two sla monitoring processes: 

sla monitor 10
 type echo protocol ipIcmpEcho 1.1.1.1 interface outside
 frequency 30
sla monitor schedule 10 life forever start-time now
sla monitor 11
 type echo protocol ipIcmpEcho 2.2.2.2 interface outside
 frequency 30
sla monitor schedule 11 life forever start-time now

1.1.1.1 and 2.2.2.2 are placeholders for the next hop address of the two gateways. 

What I expect: in my understanding, the ASA should periodically ping 1.1.1.1 through gateway 192.168.30.1  (tracking process 10) and 2.2.2.2 through gateway 192.168.30.2 (tracking process 11) to checkif the line is on. 

What happens: ASA send ICMP through gateway 192.68.30.2 even when is tracking process 1. Load balancing seems to work (several istances of "what is my ip" return both the ISP public IP addresses). Below the result of a show route

C 192.168.30.0 255.255.255.0 is directly connected, outside
C 10.0.0.0 255.255.255.0 is directly connected, inside
S* 0.0.0.0 0.0.0.0 [128/0] via 192.168.30.2, outside
 [128/0] via 192.168.30.1, outside

How do I know it: mirroring outside port and intercepting the traffic, I checked the mac addresses. ICMP packet sent to monitor process 1 has amc address of gateway 2.

Why am I complaining: sending ICMP packets to wrong interfaces, monitor process never detects one line is down (it pings through the other one). Following is the result of a show  sla monitor operational state. Any hint is very much appreciated Thanks.

ciscoasa# sh sla monit oper
Entry number: 10
Modification time: 11:56:51.793 CEST Sun Jan 22 2017
Number of Octets Used by this Entry: 1480
Number of operations attempted: 795
Number of operations skipped: 0
Current seconds left in Life: Forever
Operational state of entry: Active
Last time this entry was reset: Never
Connection loss occurred: FALSE
Timeout occurred: FALSE
Over thresholds occurred: FALSE
Latest RTT (milliseconds): 1
Latest operation start time: 18:33:51.796 CEST Sun Jan 22 2017
Latest operation return code: OK
RTT Values:
RTTAvg: 1 RTTMin: 1 RTTMax: 1
NumOfRTT: 1 RTTSum: 1 RTTSum2: 1

Entry number: 11
Modification time: 11:56:51.787 CEST Sun Jan 22 2017
Number of Octets Used by this Entry: 1480
Number of operations attempted: 795
Number of operations skipped: 0
Current seconds left in Life: Forever
Operational state of entry: Active
Last time this entry was reset: Never
Connection loss occurred: FALSE
Timeout occurred: FALSE
Over thresholds occurred: FALSE
Latest RTT (milliseconds): 1
Latest operation start time: 18:33:51.789 CEST Sun Jan 22 2017
Latest operation return code: OK
RTT Values:
RTTAvg: 1 RTTMin: 1 RTTMax: 1
NumOfRTT: 1 RTTSum: 1 RTTSum2: 1

[Ajay Saini]

What is the intention here. If the requirement is load balancing, it won't work due to ASA design.

SLA monitoring on ASA is used for tracking ISP state and failover if there is a trigger based on dropped icmp packets.

http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/118962-configure-asa-00.html 

If you can specify the exact requirement and the reason for creating 2 tracks, maybe someone can help.

HTH

-

AJ

[Massimo Giustiniani]

First of all, thanks for taking the time to answer my post.

The intention here is two use two connections to (two) different ISPs in order to increase the overall  (not per connection) throughput and use one line as back up of the other in case one fails.

This is done using to static routes with same distance (ECMP).  

I call this redundancy, but we can use a different term. ASA provides this service, I understand. Please see the relevant part of the ASA 8.4 configuration guide, where it says "The ASA distributes the traffic among the specified gateways". Licensing requirements says: All models. Base License. Here is the link. (otherwise, please have mercy and tell me what I am not understanding).

What do I mean by "it seems to work": once configured the two static routes with equal cost, I see  browsing sessions distributed on the two ISP connections, as expected. I mean that if I browse a site, the connection goes through ISP1, and if I open another browsing instance it goes through ISP2.

That is, opening several pages of "what is my ip" for example, returns pages alternatively reporting the public IP address of the two IPSs.  

Now, if one of the connections fails, it goes unnoticed by the ASA, because this usually happens outside the ISP router. This is the reason of SLA.

Monitoring the status of the connections, I see what I described above, that is the monitoring process seems to ping the control hosts incorrectly (pings host1 through the path of ISP2). I would be very glad to receive help in order to correctly configure the ASA to do this.

Thanks

Massimo

[Ajay Saini]

- ECMP on ASA load balances traffic for equal cost routes per interface. Here, you have 2 ISP and that per interface criteria fails. If you have 2 ISP on each of the 2 external interfaces, then only first route will take preference. ASA does not capability to load balance across 2 separate external interfaces. 

SLA monitoring will kick in if one ISP fails and then the other route will come into picture.

Track the route you would prefer so that it is active whenever its available. 

The intention here is two use two connections to (two) different ISPs in order to increase the overall  (not per connection) throughput and use one line as back up of the other in case one fails.

The first requirement can not work unless you have specific routes in place going across second ISP. So, one default route across first ISP and specific routes across second ISP. Or, you can use the source based routing introduced in 9.4 version to load balance the traffic. 

HTH-

AJ

[Massimo Giustiniani]

Ajay thanks for your fast reply. Both ISPs are connected to the same (outside) interface. ISP routers are on the same subnet, with internal IP addresses: 192.168.30.1 and 192.168.30.2

These two routers are connected to one ADSL line each. I do not see two external interfaces, here. Am I wrong? What am I missing?

[Marius Gunnerud]

as long as the next hops from your ASA are on the same subnet as the outside interface you are good to go.

Give my suggestions some though and let me know if you have any questions. Again, I suggest using dynamic routing to solve this and remove the SLAs.

--

Please remember to select a correct answer and rate helpful posts

[Massimo Giustiniani]

Dear fellows, I think you are both right. Ajay, I suspect ASA is doing exactly what you say, it sends the SLA packet to the proper interface, and then it is load balanced to the outside. The keypoint is what you say: SAL is binded to one interface.

Marius,  using different distances and backup works, and is a good suggestion.

So the probably only solution is or to give up the idea of SLA monitoring, or give up load balancing.

Anyway I will try what you suggest, Ajay, and I'll let you know. It is my home configuration, and I am testing it for a customer, so I'm quite motivated.

In a couple of days I'll be able to test it, and I'll let you know. I'd like to be able to mark two answers as helpful :-)

Thanks again

[Ajay Saini · Accepted solution]

ok, I undertsood now. Sorry for the confusion. This design is tricky though, especially since we plan to have sla monitoring. 

In my opinion, it should not work. The reason is that sla monitoring process is binded to an interface. You are trying to bind it with default gateway. For sla monitoring, lets say ASA needs to ping 1.1.1.1. Here ASA will just do send the packet out as per load balancing mechanism (since we have multiple routes) and hence you see icmp packet sometimes across the ISP2. I wont blame ASA for that.

I am wondering what will happen if you add a specific routes across the ISP1 for the tracking ip address like below:

route outside 1.1.1.1 255.255.255.255 192.168.30.1 1

I believe then the traffic will always exit through ISP1. Please try. FYI, its a wild suggestion because I have not seen exact setup earlier. This is not a solution, just a workaround on basis of which we can think forward if it works.

-AJ

[Massimo Giustiniani]

Been away for a while, had time to test it today:

adding to statics to the monitoring targets seems to work. So we have 

route outside 1.1.1.1 255.255.255.255 192.168.30.1 

route outside 1.1.1.2 255.255.255.255 192.168.30.2 

and SLA monitoring sends packets to the proper gateways and properly detects failure. 

Thanks for the support

[Marius Gunnerud]

I see.  You are correct on the distributing of traffic between the ISPs, so long as the default route points out the same interface (as it says in the document you posted).  

As for the SLA issue I do not believe this can be solved using SLA.  as long as there is a route to 1.1.1.1 through the outside interface (whether it is through ISP1 or ISP2) it will still be pingable and therefore there will never be a failure situation unless both ISPs fail.

I can see two possible solutions that might work for a solution.

 1. set up dynamic routing between your ASA and both ISPs, then the routing protocol should take care of this for you.

 2. upgrade to ASA version 9.4 or higher and then try to solve this by using route-maps to set the next hop for those spesific IPs (1.1.1.1 to IPS1 and 2.2.2.2 to ISP2). then this should force traffic to these IPs the correct way and the SLA tracker should work.  This is a messy way to get this to work though and I would suggest going with dynamic routing protocol instead.

--

Please remember to select a correct answer and rate helpful posts

[Marius Gunnerud]

What do you mean wihen you say "Load balancing seems to work" ? The ASA  running version 8.4 does not support load balancing traffic between two ISPs.

As for the route tracking you only need to track the main route you want in the routing table and set the admin distance on the second default route to a higher value.  for example:

sla monitor 10 now

  type echo protocol ipIcmpEcho 1.1.1.1 interface outside

  frequency 30

track 1 rtr 10 reachability

route outisde 0.0.0.0 0.0.0.0 192.168.30.1 track 1

route outside 0.0.0.0 0.0.0.0 192.168.30.2 254

--

Please remember to select a correct answer and rate helpful posts