DUAL ISPs ON CISCO PIX

Unanswered Question
Oct 19th, 2013
User Badges:

Hi All,


We have a strange problem with a PIX 515E running 8.0 (4).


BACKGROUND

The Pix is configured with two ISP connections, primary and secondary. We use a simple SLA monitor to check for reachability of the ISP 1 gateway, if this goes down ISP 2 comes up and internet is restored. In addition there are a few site to site VPNs terminating on the Pix and some ad hoc dynamic VPN connections.


PROBLEM

When the primary ISP 1 failed the other day all was well and the Pix transferred to ISP 2 almost in a heartbeat as it is supposed to do. The site to site VPN also changed over without an issue. However, when the primary ISP came back up the secondary route did not clear down in the routing table as evidenced by the "sh route" output. Furthermore the site to site VPN remained connected to the secondary connection. We cleared this fault by doing a simple "shut" and "no shut" on the secondary interface.


Has anyone any idea why this is happening ?


Thanks.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
amitaaga Sat, 10/19/2013 - 11:04
User Badges:
  • Cisco Employee,

Looks like SLA monitoring is not working as expected for you. Make sure that it is configured as per the following doc:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_configuration_example09186a00806e880b.shtml#show


If it is configured correctly but is still not working as expected then check to see if SLA monitoring is operational when the primary route comes back up after failing using "show sla monitor operational-state".

Ajay Saini Sun, 10/20/2013 - 12:10
User Badges:

If the secondary ISP came up, I believe that the sla monitoring is working fine. When the issue happens next, and the primary does not come back up. Try to ping the ip address which is being tracked sourced from the primary ISP.


if the ping works fine, then the primary should come back up as well. Also, get syslogs to see if you see something unusual.

tech01cisco Tue, 10/22/2013 - 13:06
User Badges:

Hi All,


Sorry for delayed response. We have run the sh ip sla command and here is the output.


firewall# show sla monitor operational-state 123

Entry number: 123

Modification time: 20:44:07.659 BST Sun Sep 8 2013

Number of Octets Used by this Entry: 1300

Number of operations attempted: 0

Number of operations skipped: 0

Current seconds left in Life: 0

Operational state of entry: Inactive

Last time this entry was reset: Never


This suggests that we have an error with the SLA monitor, but as we said in the original post it fails over perfectly with a failure of the primary connection, but is not flipping over when the primary connection comes live. We have been through the configuration guide and verified all steps which are fairly simple.


Yes, after the primary connection comes back up we can ping the IP address that is being monitored using the primary interface as source. Also, interestingly if we set a ping going to both IP gateways, ie primary and secondary we even see the primary response time of the echo change when the primary comes back up. We can tell because the primary is a dedicated fibre whereas the secondary is good old ADSL. It is however still showing the ISP2 within the routing table. This tends to point towards the routing table not flushing the secondary connection after a fail over and perhaps nothing to do with the SLA..


Anyone got anymore suggestions?


Regards,

jumora Tue, 10/22/2013 - 14:08
User Badges:
  • Cisco Employee,

Can you please share the configuration.


Known issues:


SLA Monitoring on ASA


Problem:


SLA monitoring does not work after the ASA is upgrade to version       8.0.


Solution:


The problem is possibly be due to the IP       Reverse-Path command configured in the OUTSIDE interface. Remove the command in ASA and try to check the SLA       Monitoring.


Bug ID CSCtc16148


http://tools.cisco.com/squish/6B83a

tech01cisco Tue, 10/22/2013 - 16:17
User Badges:

Hi All,


We need to make a correction to our earlier post.. We said


"Yes, after the primary connection comes back up we can ping the IP address that is being monitored using the primary interface as source"


In fact we can not ping the IP gateway from the primary interface after the connection has come back up. The routing table still shows the route as the secondary connection thereby requiring an admin shutdown of the secondary interface which then puts the primary interface back into the routing table.


Here is the configuration relating to the IP SLA monitor..


sla monitor 123

type echo protocol ipIcmpEcho XXX.XXX.XX.XXX interface Outside

num-packets 3

frequency 10

sla monitor schedule 123 life forever start-time now


track 1 rtr 123 reachability


route Outside 0.0.0.0 0.0.0.0 X.X.X.X 1 track 1

route ISP2 0.0.0.0 0.0.0.0 X.X.X.X 254


Any ideas anyone ???


Regards,

jumora Tue, 10/22/2013 - 16:24
User Badges:
  • Cisco Employee,

Ok, I am not sure if you checked my post can you please get me the configuration so we can check.

jumora Tue, 10/22/2013 - 16:48
User Badges:
  • Cisco Employee,

Plus, you can not recover the main link if the IP address that you set to monitor is not available so I am not sure how you are recovering

jumora Tue, 10/22/2013 - 16:52
User Badges:
  • Cisco Employee,

Please run the next command:


show run ip verify reverse-path


If the command is in place just remove it


enable

config t

no ip verify reverse-path interface Outside

jumora Wed, 10/23/2013 - 13:41
User Badges:
  • Cisco Employee,

Did the information given help out; please remember to reply "Answered" if the information given to you resolved your problem.

tech01cisco Thu, 10/24/2013 - 04:13
User Badges:

Hi Jumora,


Unfortunately the problem is not resolved.


ip verify reverse-path is not running on the outside interface. Here is a further screenshot of the results of the SLA output:


firewall# sh sla monitor operational-state 123

Entry number: 123

Modification time: 23:41:42.374 BST Tue Oct 22 2013

Number of Octets Used by this Entry: 1480

Number of operations attempted: 13064

Number of operations skipped: 0

Current seconds left in Life: Forever

Operational state of entry: Active

Last time this entry was reset: Never

Connection loss occurred: FALSE

Timeout occurred: FALSE

Over thresholds occurred: FALSE

Latest RTT (milliseconds): 1

Latest operation start time: 11:58:52.372 BST Thu Oct 24 2013

Latest operation return code: OK

RTT Values:

RTTAvg: 1       RTTMin: 1       RTTMax: 1

NumOfRTT: 3     RTTSum: 3       RTTSum2: 3


This all seems fine and as we have said the IP SLA fails to the secondary ISP without an issue, it just will not reinstate the primary ISP connection after it becomes available again. The only way we can get this to flip back to the primary connection is to do a "shut" and "no shut" on the secondary interface. The "sh route" output verifies that this is happening which seems to point towards the secondary route not getting cleared from the routing table correctly.


Any more ideas?


Regards,

jumora Thu, 10/24/2013 - 10:29
User Badges:
  • Cisco Employee,

Can you send me the show run ICMP?


Also, can you try changing the IP address that you are using to monitor for 4.2.2.2 just as a test.


The strange thing is that you state that the main link recovers but this is not possible if the address that you are using to monitor is never available.


If you cannot change the monitoring address on the SLA configuration, please check with a packet-tracer how the ASA would consider to route to that destine address from a LOCAL address (PC).


packet-tracer input inside icmp 8 0 detail


If you have any VPN configuration please confirm that you do not have a match for ip any any or icmp any any as this could affect SLA monitoring.

tech01cisco Fri, 10/25/2013 - 14:39
User Badges:

Hi Jumora,


Here are the answers to your questions.


firewall# sh run icmp

icmp unreachable rate-limit 1 burst-size 1

icmp deny any echo Outside

icmp permit any echo-reply Outside

icmp deny any echo ISP2

icmp permit any echo-reply ISP2


Yes, the primary link will recover when we shut down ISP2, which is the secondary connection. The same results were obtained using 4.2.2.2


We do have VPNs on this device and have checked the configuration which all seems fine, no ip any any or icmp any any..


It may be that this problem is not with the SLA monitor as we seem to get all the right output from the show monitor. I think now the problem maybe with the static routes which are not clearing the secondary route without shutting down the secondary interface. The question is why this is happening????


Any more thoughts??

jumora Fri, 10/25/2013 - 15:32
User Badges:
  • Cisco Employee,

Answer, remove the next line:


icmp deny any echo Outside

tech01cisco Fri, 10/25/2013 - 16:05
User Badges:

Hi Jumora,


Just logged in and removed "icmp deny any echo Outside" and it didn't work. This is a real head scratching problem as we configure a lot of these failovers and have not had this problem before. I am going to do some more debug tomorrow and see if I can see anything on the routing side that may be causing the routing table to retain the secondary ISP route, in particular I am going to look at the DHCP config on the Outside interface, although it picks up a static IP. Although rarely required I am thinking of a full system restart to see if that flushes any cached errors in the routing table.


Any more ideas ??


Regards,

jumora Fri, 10/25/2013 - 16:14
User Badges:
  • Cisco Employee,

I need a number to reach you, I think I need to take a look at it!!

jumora Fri, 10/25/2013 - 16:48
User Badges:
  • Cisco Employee,

I would suggest is to remove the SLA monitoring configuration and then try to reach the addresses that you are using for monitoring, if they are not reachable and you remove the line that I told you to remove then the ISP is blocking ICMP or you have some other configuration that is conflicting with the ICMP portion that we might resolve if you send me the packet-tracer that I requested.


Please check with a packet-tracer how the PIX would consider to route to that destine address from a LOCAL address (PC).


packet-tracer input inside icmp 8 0 detail

jumora Mon, 10/28/2013 - 18:08
User Badges:
  • Cisco Employee,

Do you still need assistance?

tech01cisco Tue, 10/29/2013 - 14:11
User Badges:

Hi Jumora,


Yes we are still troubleshooting this issue but I have been tied up on a big VPN build recently.


I think this issue maybe a routing one and I need to go through the config line by line. This is assumed because for some reason the routing table is not flushing when ISP 1 comes back up, which I know is triggered by the track command but could be something unrelated as you have said. Both routes are static so it should all be fine. The AD is also correct with 1 for ISP 1 and 254 for ISP 2.


Just to clarify the ping results. The SLA monitor is sending pings to the ISP 1 gateway, if this goes down and therefore is not reachable ISP 2 comes into to play, all good so far. Then when ISP 1 comes back up and is therefore reachable by a simple ping, it should then put the ISP 1 route back in the routing table, which is not happening.


I have not had time as yet to run packet tracer as the system is in a live network.


Any more thoughts ?

jumora Wed, 10/30/2013 - 11:31
User Badges:
  • Cisco Employee,

We can progress as soon as you can get me the packet-tracer, I think it is key to resolving this issue.

jumora Thu, 10/31/2013 - 10:48
User Badges:
  • Cisco Employee,

If this persist after confirming that the SLA monitored IP address is available and packet tracer does not demonstrate anything I would suggest to upgrade to interim release as I have not found a bug but this could be an issue with the code.

Actions

This Discussion