Network Trouble

[r.d.schnitzer]

Scenario:  Replaced 2 Pix 515e firewalls with 2 ASA 5510 firewalls.  Turned old firewalls off.  Turned one of the fully-configured new firewalls on.  All traffic to and from the IP address configured on the outside interface of the ASA flowed properly.  Also, one of the IP address in the middle of our routed block was also able to send/receive behind the firewall.  When I did a packet capture on the outside interface of the new ASA, I only saw traffic destined for these two IP addresses.  All other traffic was not flowing properly into the outside interface of the ASA properly.  The outside interface of the ASA is connected to a port on a Cisco switch provided by the ISP for our fiber run.  We restarted that switch, but nothing changed.  The capture still only showed traffic destined for 2 of our many public IP addresses.  We connected a laptop to the switch and verified that we could in fact browse the internet using the IP addreses that we weren't seeing any traffic coming to while capturing on the ASA.  I turned on full debugging, and there were no errors being logged, and no traffic at all was coming into the outside inteface of the ASA for the majority of our public IP addresses (which have static NAT translations configured on the ASA).  We also called the ISP who verified the duplex and speed settings on the switch ports.  Both sides were operating at 100 Mbps full-duplex.  They hadn't made a change to that switch in years.

This was actually the second time we experienced this phenomenon.  Both times after many hours of waiting, everything returns to normal.  The first time it occurred, we had simply failed over from one Pix firewall to the other.  Something we had done numerous times in the past without any trouble.  At that time, it was only the IP address configured on the outside interface of the Pix that could traffic.  Failing back to the primary firewall did not resolve the problem.  We tried everything we could think of, including powering down all of the network switches (WAN and LAN), and then turning them back on.  After a few hours of frustration, and at least 30 minutes of not touching anything, everything started working again like magic.

We can't have this problem looming over us, as we do occassionally need to failover to our standby firewall.  Any help would be greatly appreciated!

[sachinraja]

hi schnitzer

This is peculiar, but needs to be troubleshooted when the problem occurs again.. so, if i  understand it right, you have a public block say (200.200.1.0/27).. out of which you have 200.200.1.1 (outside IP of the ASA), and another IP 200.200.1.2 working fine.. where  is the second IP assigned ? is it statically natted ? now, u say the rest of the IPs 200.200.1.3 - 200.200.1.31 is not able to access internet through ASA (static , nats etc) ? and if you assign 200.200.1.3 through a directly connected PC, it works good ?

Is the ASA directly connecting to your ISP through a Fastethernet Layer 3 link ? Next time you encounter this issue try these:

1) ping the static global IP for eg 200.200.1.10 from a laptop which is connected on the same VLAN as the ASA outside.. This makes sure ASA works good. you can put a laptop on the switch which connects your asa and ISP link to tes this..

2) Look at the "show xlate" and "show conn" tables, to see what exactly happens with regards to translations, and connections on your ASA.. see how many connections are established at the time you have issue..

3) test both outgoing and incoming connections when having issue.. you can isolate if it is a problem coming into or exiting your network through ASA

4) Look for any logs on your ASA firewall when this issue happens.. It can give you some useful info..

5) check the cpu utilization (just to make sure) of the firewall

6) ask your service provider to ping the NATed IP which has issues, from their end, to check the reachability.

7) check for any known bugs related to NAT and the version of ASA you are runnig... sometimes open caveats can leave you troubleshooting issues for long !

Let us know.. hope this helps.. all the best

Raj

[ericn8484_2]

So you have a Cisco ASA firewall between your corporate network and the ISP's switch. When you run into problems with your connection, you see no inbound traffic for your public IP's to the Cisco ASA firewall?

The thing I would try if this occurs again as it sounds like it lasts quite some time is to put a switch between your ASA and the ISP's switch and perform a port monitor and with a packet sniffer. If you do not see traffic coming into your firewall from your ISP when this is occurring, it is an issue with the ISP wither its a routing issue or arp table issue, its on their end. Make sure to have someone from the outside world try pinging your IP's if possible while performing the test so you know for sure you have inbound traffic.

The reason I would suggest this test is don't depend on the logs and debugs on your firewall by nature, firewalls loves to block undesired traffic and you might not see that traffic when running a packet capture. If it does turn out that the traffic is coming in put your firewall but it is somehow dropping it, I change IOS version, might be a bug. A firewall issue is typically all or nothing, it doesn't usually come and go if its a config issue.

However with your symptoms of it showing up for several hours and then disappearing like nothing happens, it honestly points to your ISP as the problem. But like with nearly all ISP's (I even worked at one for five years), they will never believe it is an issue on their end and will do very little to dig deep into the problem. They aren't like that because they don't want you as a customer, they are like that because for every legit issue that occurs, they have another 20 customers for blaming them things are not working when the fault is really on the customers end. They just don't have the resources to dig deep into every issue a customer thinks is on their end.

[Giuseppe Larosa · Accepted solution]

Hello,

>>   After a few hours of frustration, and at least 30 minutes of not touching anything, everything started working again like magic.

if the ISP switch performs only L2 the ARP table is placed on another device that is not under your control, this would explain why power cycle of ISP switch didn't solve the issue.

The ARP table of the ISP L3 device contains ARP entries for all the public IP addresses of your block but the MAC address associated with all entries was likely that of the old device (effect of NAT) that you have replaced with the ASA. The ASA has a different MAC address.

So until the remote device does not ARP again for all the IP addresses you are stucked and probably only the ASA IP address can be reached because it can have sent a gratuituos ARP for its IP address.

the test with the PC directly connected to the ISP switch confirms this.

Hope to help

Giuseppe

[r.d.schnitzer]

Giuseppe,

I will check with the ISP in the morning to determine if the switch is only operating at layer 2.  I really hope that's the case, as that would make this all make sense, and be the easiest to remedy.  If you ever find yourself traveling through Indiana, I'll buy you dinner if this solves the problem.  Regardless, thanks for taking the time to answer.

Ryan

[r.d.schnitzer]

Giuseppe, thank you very much for your assistance in this matter, and thank you to everyone else for your responses as well.  The ISP cooperated when we tested the failover from one ASA to the other.  Before clearing any ARP caches or doing anything, everything was already working though.  It's a little unnerving that we don't know for sure whether or not the problem is gone, but we've failed over a few times without any problems now.  If it does occur again in the future, I'll be back for more help.  Thanks again!

Ryan

[qubenetworks]

We've seen this before when moving firewalls, if you can co-ordinate with your ISP to do a 'clear ip arp xxx.xxx.xxx.xxx' for the problem address' on the upstream layer 3 device it should clear the issue, if they dont maybe you could ask them to reduce the arp timeout so the problem gets resolvedd quicker.