ASA5540 intermittent failure to ping DMZ servers in native vlan

simbachirara · ‎08-13-2010

All

I am looking for assistance on a problem that I believe could be on the firewall. I have an ASA5540 connected to a DMZ via switch A. Switch A connects to Switch B. There are servers in the DMZ but 4 of these become unreacheable all at the same time from internal network, but if you connect to the DMZ switches, you can ping these 4 servers ok. These servers are in vlan 1 and timeout intermittently primarily from 1630h to approx 0930h. During the day, they sometimes do the same but it could be once or twice. There are other servers in DMZ also in vlan 1 that dont timeout from internal network. When you ping the 4 servers and any vlan 1 ipaddresses (including the directly attached DMZ switch, they timeout, BUT from the DMZ switches, you can ping the DMZ interface on the ASA firewall. When that timeout period to these 4 servers stops (approx 3-15minutes long) all hosts in the DMZ can be pinged from the firewall. The firewall and switch processors will have processes running below 1% CPU utilisation. At the time the 4 servers are timing out, one can still ping from firewall to other servers in different vlan. 3 of the 4 failing servers are VMWare machines. The other server is a dedicated server with 2 teamed cards, one into each of the DMZ switches. To eliminate the servers, I have shutdown all ports to these 4 servers, but I still received a timeout of the 2 DMZ switches.

Another thing is that when I span vlan 1 in switch A attached to ASA5540 and capture with wireshark, I see lots of malformed packets (errors) to these 4 servers and also to another MAilMarshal server in the DMZ. The malformed packets are inbound and outbout. I scanned the servers for viruses and they are clean. I am running IOS 8.2(1). After shutting down the ports for the 4 servers that time out and still had problems, I am thinking of trying to upgrade IOS to 8.2(3).

Questions

1. What could be causing the malformed packets to some of these servers that are in vlan 1? Some vlan 1 servers dont have the malformed packets.

2. Why would firewall fail to ping the vlan 1 servers when the 4 servers time out?

Your assistance will be appreciated.

Kureli Sankar · ‎08-13-2010

According to what you say below:

When you ping the 4 servers and any vlan 1 ipaddresses (including the directly attached DMZ switch, they timeout, BUT from the DMZ switches, you can ping the DMZ interface on the ASA firewall.

In the future it would help us if you could pls. add a simple text based topology:

inside hosts---ASA--vlan1--(dmz servers and other dmz hosts)

1. The probelm is that the inside hosts are unable to ping the dmz server - 4 of them, the dmz switch or anything on the dmz segment during certain times.

2. At the same time dmz switch is able to ping the dmz interface ip of the ASA.

Pls. do not upgrade the code to 8.3.x for this problem. If you are looking for new features and are looking to upgrade that is ok but, the problem is here is not due to the ASA code.

Pls. check the logs and see what they say when the pings fail.

conf t

logging en

logging buffered 7

sh logg | i x.x.x.x

where x.x.x.x is the ip address of the dmz server. You can collect captures as well. Pls. read here: packet capture ASA/PIX/FWSM: https://supportforums.cisco.com/docs/DOC-1222

-KS

simbachirara · ‎08-14-2010

Thanks. Will try this on Monday.

simbachirara · ‎08-16-2010

All

I managed to get my DMZ stable by updating the DMZ switches IOS from 12.2(25) to 12.2(55). I did this on Sat. I had also updraed the ASA from 8.2(1) to 8.2(3), but this did not change things. I may have forgotten to mention that I was seeing underruns on the DMZ Sw A where ASA was connected. After the ASA upgrade I could connect to the DMZ Sw A and B. I decided to reboot the DMZ switches and I started failing to connect to DMZ till I connected ASA to Sw B instead of Sw A. The underruns still clocked up on new ASA connection on Sw A till I upgraded the Sw IOS.

What I still see on Wireshark are malformed packets to a Proxy Load Balancer, Mail Marshal server and a webserver. BUT the servers are no longer timing out intermittently as before. These servers are upto date on Antivirus and patches.

Any ideas where the malformed packets maybe coming from?

Thanks for your feedback.