I am load balancing two isp's. One isp is fine, but the other I have very high packet loss when pinging the corresponding wan interface from any machine located outside our network. During the packet loss, I cannot https to our firewall from outside our network, but I can rdp using a different address on that same isp and ping another machine located inside our network, it seems only the wan interface is having the issues. Our isp uses icmp to our wan interface and they started noticing the loss the other day, again the other isp loses no packets. I have no issues with clients losing connection from that isp so it looks like it is an issue from outside in only on that wan interface.
I have a spare ISA and that is experiencing no packet loss when using another ip from the problem isp on it's wan interface. The isa's have the same configuration and when I tried moving completely to the backup isa it continued with the same issue. Upgraded to a later firmware, still the same issue. I even took a laptop and put it on our switch that is before the isa and gave it a static public address and I can ping it with no problem from the outside. I can also ping from the laptop to the wan port of the prod isa that is losing packets and it replies as it should, which I assumed would lose packets if it were the isa having issues, but it didn't.
I know this has nothing to do with Cisco security services as on the backup firewall it was still losing packets when we moved the connections over to it.
I can ping from inside my network to the public wan address with no loss when outside machines are having problems.
I can ping the problem wan address from a laptop or from isa(IP = 64.x.x.42) to isa(IP = 64.x.x.45) with no packet loss.
This is a head scratcher and I need some real help here.
Looks like we are able to replicate the issue. If we have both isp's plugged in the isa, we experience packet loss to the one wan of isp 1. The other wan, isp 2, is fine and never loses packets. When we disable this good wan, isp 2, isp1 wan then responds fine! Once we plug in isp2 again, we lose packets to isp 2 wan. No idea if this is a firewall issue, but it seems we ruled out an isp issue, since isp 1 wan responds fine when isp 2 wan is disabled. Anybody have any idea what is going on here?
There are two things you should pay attention to.
1. when in dual WAN there is always needed static default route for traffic generated on ISA
2. Link Failover Detection mechanism should be working as expected so perhaps ping to the DNS server would be safer option than ping to IP gateway
Never figured this one out. However, I was able to figure out how this starts. A couple of weeks ago we had an issue with an ISP and we disabled that interface during the day and re-enabled at the end of the day and thats when we noticed ICMP breaks to other ISP.. I don't know why but the ISP we renabled we can ping just fine, but the other one always has issues. The only fix is to enable link failover detection and both WAN interfaces become pingable again.
Now, we use load balancing and one issue we are experiencing is that randomly both WAN interfaces go down and I can confirm this with a ping utility we use off site that pings both WAN interfaces and also each ISP gateway they are connected to. When this happens the ISP's gateways are reachable and neither WAN interface on the ISA are. I am still on .15 version of the firmware, because there were too many issues with newer releases. Is this a known issue?
Hello, we do not use link failover detection, we have that turned off right now. I am not referring to loss of ping in regards to link failover detection. Our isp pings from the outside to our Wan interface (To monitor connectivity) and they are losing packets and so am I from any machine I tried from outside our network. But again, the other isp is absolutely fine no matter what port we use on the isa, so it does seem like some sort of issue with configuration on the good isp. So are you saying I should expect packet loss to one isp when working with two connections in a load balancing setup? To me that doesn't make sense, even 3 different Cisco engineers couldn't explain it and it is being escalated apparently to "Subject matter experts". If you could expand on how the default route works when pinging either interface that might explain things, not sure.
Thank you for more details in regards the way it is tested.
So Link Failover Detection mechanism is to ensure not only physical connectivity but also logical connectivity of the WAN interface. If the logical connectivity issue is not detected properly you might experience packet loss since the WAN interface might sends traffic over failed link. Physical connectivity is not the only measure.
For the traffic passing through the router you have default gateway IPs however traffic generated on ISA500 might need to be specified.
Do you have packet capture when ICMP echo replay fails on problematic WAN?
Yeah, I get that the link failover is for physical and logical. The thing though is that it is turned off and both isp links are good, so it shouldn't be failing over to the other isp unless there is a setting that is off or something bad in the firmware. Doubtful on the firmware being an issue, but setting, maybe possible. I have taken packet captures but it doesn't seem to show anything when the failure happens, it only shows the good requests/responses.
Hello Mr. Boliewski,
Sorry for just jumping in here, Aleksandra and I work on the same team and I was following this.
As a test does if you switch the WAN ports the ISP's are connected to, does this issue change in any way?
One of the tests I did at the end of the day on Friday. I flip flopped ports on the isa and it exhibited the same behavior, the same isp was the one still experiencing the issue and the other isp was fine.
Thank you for that quick information. I have done some further research and I am also looking at the case that you currently have open.
Am I correct that from reading the case that you have two ISA's and that when you replaced the ISA the same WAN/ISP was still having the issue?
Just for more information,
What is the connection type, Both Cable, DSL, one of both, etc? The topology that was attached to the case, did not attach to good. A lot of the lines and boxes were very distorted.
Also what is the make and model of you modems?
Just emailed you the network map again in pdf, should show correctly now, let me know. Both isp's are business class, no modems.
One isp is Cbeyond (10MB, t1's)--->Good isp
Second is handed off as ethernet from Business Only Broadband--->The one we are having issues with
Sorry for the slow response back, I have other cases that I was having to update.
Looking at the Topology and trying to quantify the different scenarios that you have tried, have to tried pinging from outside to the separate network that is the fail-over to B.O.B Circuit? Or from that network(220.127.116.11) to the ISA WAN?
Has there been thought to checking the layer 2 switch either by replacing or removing from network for a test?
When your ISP noticed the loss the other day, what did they say about it? Did they do any testing? If so can you share any results?
What are the exact ICMP messages when ping fails? Also I would recommend you to try tracert to both WANs to compare. If ISP has some routing issues you might see response such a destination network unknown.
Getting request timed out. By the way, last night I took our backup ISA and put stock firmware on it and it seemed to then work okay, so sounds like a setting issue. I sent an email to Eric, but he hasn't responded. I just don't know what it could be, will have to compare side by side and see what happens. The only thing is I did not have my lan switches trunked to the ISA, just pinged the configured wan interfaces from an outside machine and they both worked. Any ideas on the setting would be great. I can forward you my config if needed. Thanks!
Hey Bob, Sorry I have tied up last 24 hours. I have over 20 cases that I am working concurrently. Could you sent me the two configurations files. I have a way to compare them, if that does not work I have two devices I can put in the lab.