Internal hosts not talking behind PIX - dropping packets

Answered Question
Feb 15th, 2009
User Badges:

Hi

We have a PIX515e setup where we are using NAT for our internal hosts.


inside-network is 192.168.7.zzz/24

outside network is 194.xxx.yyy.zzz/24

where zzz in the inside and outside are exactly the same


The issue we are facing is that, our machines on the internal interface can at times talk to each other and at times cannot. Pinging in almost all cases is dropping packets, Remote desktop into another host works and then disconnects, but also that we have DNS servers which cant communicate to each other.


If you have any ideas, please let me know. Bear in mind that there is communication but there is major packet drops and loss of communication after connection.


We have 4 rules on the inside interface allowing ip, tcp, udp and icmp from ANY to ANY on the inside interface. Although these rules are not exaclty needed as the firewall is setup to block all inbound traffic except for specific inbound rules, we created them to ensure connectivity, but no luck!!


Thanks in advance

Ali



Correct Answer by robertson.michael about 8 years 5 months ago

Hi Tahir,


I have to agree with Eddie--traffic between 2 hosts on the same subnet will not be affected by the firewall's rule set.


Given the intermittent success of the connection, I would start by checking the cables and NICs and swapping them out with known good ones. If the problem persists, try collecting packet captures on the hosts at each end of the connection using Wireshark or tcpdump. You should see something along these lines:


1. Source host sends an ARP request for the MAC address that corresponds to the destination IP address.

2. Destination host sends an ARP reply with its MAC address.

3. Source host starts sending traffic to the correct IP and MAC address

4. Destination host acknowledges/replies with more traffic to the source host using the correct IP and MAC address.


You should see that the traffic is identical on both ends of the connection (including MAC addresses since it never leaves the subnet). If any of the steps are missing, or the traffic does not match on both ends, that should give you an idea of where to start troubleshooting this issue.


Also, as Eddie mentioned, you should check the switch to make sure the error counters are not increasing. Even with a new switch, errors can be caused by anything from bad cables to speed/duplex mismatches.


Hope that helps.


-Mike

Correct Answer by eddie.mitchell@... about 8 years 5 months ago

That is strange. Is the default gateway for all hosts the inside interface of the firewall?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.7 (6 ratings)
Loading.
eddie.mitchell@... Sun, 02/15/2009 - 14:48
User Badges:
  • Silver, 250 points or more

So, you're having issues with 192.168.7.zzz hosts talking to other 192.168.7.zzz hosts? If this is just communication occuring between hosts on a directly connected network, the traffic should not be arriving at the firewall.

smbtest12 Sun, 02/15/2009 - 15:35
User Badges:

Thanks Eddie.


That's right, however hosts are talking at times but they seem to get disconnected, so we know there is a link, but a pretty weak one. All the hosts go into a switch and from there into the inside interface of the PIX.


We are not sure why we arent getting connectivity. Any ideas welcome.


Thanks

eddie.mitchell@... Sun, 02/15/2009 - 15:42
User Badges:
  • Silver, 250 points or more

Any errors on the switch? Approximately how many hosts are being affected?

smbtest12 Sun, 02/15/2009 - 15:47
User Badges:

Its a brand new switch, we even used a different new switch - same issue.


All the hosts are being affected, which leads us to think it that its something on the firewall


we can ping the internal interface fine without dropping packets from any inside host, but cant seem to ping properly to another indside host from one.


thanks

Correct Answer
eddie.mitchell@... Sun, 02/15/2009 - 16:23
User Badges:
  • Silver, 250 points or more

That is strange. Is the default gateway for all hosts the inside interface of the firewall?

smbtest12 Sun, 02/15/2009 - 16:50
User Badges:

Yes the default gateway is the internal interface - 192.168.7.1


Could there be an issue with the NATTING or ACLs - we tried to go through this thoroughly by trying to use process of elimination with certain rules to see if any were blocking the access. Still no luck ??


Thanks

eddie.mitchell@... Sun, 02/15/2009 - 16:57
User Badges:
  • Silver, 250 points or more

I would try to create a specific ACE at the top of the inside ACL to match the traffic that you're having a problem with.


For example, specific host to specific host for RDP.


access-list inside_in line 1 permit tcp host 192.168.7.x host 192.168.7.x eq 3389


Then I would initiate RDP traffic between those hosts and verify that the hitcount on this ACE is growing. This will confirm whether or not the traffic between the inside hosts is attempting to traverse the firewall (it shouldn't be).


Hope this helps.

smbtest12 Sun, 02/15/2009 - 17:55
User Badges:

Thanks Eddie


I will try this out tomorrow and post back results


Thanks

Correct Answer
robertson.michael Sun, 02/15/2009 - 18:49
User Badges:
  • Silver, 250 points or more

Hi Tahir,


I have to agree with Eddie--traffic between 2 hosts on the same subnet will not be affected by the firewall's rule set.


Given the intermittent success of the connection, I would start by checking the cables and NICs and swapping them out with known good ones. If the problem persists, try collecting packet captures on the hosts at each end of the connection using Wireshark or tcpdump. You should see something along these lines:


1. Source host sends an ARP request for the MAC address that corresponds to the destination IP address.

2. Destination host sends an ARP reply with its MAC address.

3. Source host starts sending traffic to the correct IP and MAC address

4. Destination host acknowledges/replies with more traffic to the source host using the correct IP and MAC address.


You should see that the traffic is identical on both ends of the connection (including MAC addresses since it never leaves the subnet). If any of the steps are missing, or the traffic does not match on both ends, that should give you an idea of where to start troubleshooting this issue.


Also, as Eddie mentioned, you should check the switch to make sure the error counters are not increasing. Even with a new switch, errors can be caused by anything from bad cables to speed/duplex mismatches.


Hope that helps.


-Mike

smbtest12 Sun, 02/15/2009 - 23:55
User Badges:

Hi Mike


Thanks for your input, i will try this out once in the office today and report back.


Do you think using a mixture of Cat5 and Cat6 cabling and that we have Gigabit switches with a PIX (100Mb max) will affect performance ?


thanks, i really appreciate both your inputs.


Regards


You should also look at proxy arp configuration on the inside interface. By default is it turned on and sometimes it can cause issues with communication on the local network. It is rather aggressive and will populate the local pc/server arp tables with its mac address for all pc/servers IP addresses on the local network, along with the default gateway. If you look at the arp table on a pc/server having issues, and proxy arp is on, then as you ping multiple pc/servers on the local subnet, you will see the ip addresses all have the same mac, which will be the PIX. YOu can try and turn if off on the PIX. As long as the pc/servers have their default gateway as the PIX, it should not affect anything on the local subnet.

smbtest12 Tue, 02/17/2009 - 02:31
User Badges:

Hi All


Many thanks for your suggestions/ideas


1. We tested specific connections using netmon from one host to another and noticed that some packets were being translated correctly, but others were going from the outside 194 range, bypassing the static NAT rule and going to some random internal 192 IP address eg 192.168.7.255 which wasnt even created. So after drilling down, we came across a Dynamic NAT rule that allowed ANY on the outside to ANY on the inside, and funnily enough this was created o its own, without us doing anyhting. However, we have managed to delete this and internal comms is now up and running as it should.


2. We have left Proxy ARP as enabled for now, but we did test it in "disabled" mode, communication was not affected, but if in the future we are having issues, we will refer to this point.


3. Both internal and external interfaces are on Full 100 duplex becuase the Router from where the internet is flowing has been configured like so.


All that is left for now is what we are made to understand is called DNS Doctoring, so that internal servers can resolve DNS queries on the outside interface. We are working on this, and will post back with any issues.


We really appreciate ALL your help on this issue, and will be rating your comments.


Thanks and Regards

Ali


Thank you very much to all of you

Actions

This Discussion