Re: PIX 6.3(3) RST problem

nnw11903 · ‎11-05-2003

Hi,

since I upgraded our redundant PIX 515E from 6.3(1) to 6.3(3) there is a noticeable higher amount of dropped packets (x5).

The dropped connections are equal in one point: While closing the tcp client/server session, there is a RST packet involved (instead of only FIN/FIN-ACK) (for example "half-duplex" close, like described in RFC 1122).

[tcp back channel established]

Nov 05 2003 15:45:39: %PIX-6-302013: Built outbound TCP connection 6177026 for DMZ-public:10.30.13.150/8080 (10.30.13.150/8080) to inside:10.30.7.152/22758 (10.30.7.152/22758)

[Data flows normally...]

[tcp back channel closed via RST from inside]

Nov 05 2003 15:45:39: %PIX-6-302014: Teardown TCP connection 6177026 for DMZ-public:10.30.13.150/8080 to inside:10.30.7.152/22758 duration 0:00:01 bytes 882 TCP Reset-I

But then the ACK Packet, which is the result to the RST (Last log line: Reset inside from 10.30.7.152) is dropped by the PIX:

Nov 05 2003 15:45:39: %PIX-4-106023: Deny tcp src DMZ-public:10.30.13.150/8080 dst inside:10.30.7.152/22758 by access-group "acl-dmz-public"

I used a sniffer to exclude false positives in the syslog of the pix. This behavior occurs not in 6.2(3) and 6.3(1) but in 6.3(2) and 6.3(3). Bug toolkit does not show an entry similar to this topic.

I tried "sysopt timewait", but this does not help.

Thanks for helping,

Christian

scoclayton · ‎11-05-2003

Christian,

The changes you note are a result of the new feature known as policy NAT. One of the things we had to do with respect to this feature was change the way the PIX processes incoming packets. We now log inbound packets that have no connections associated with them as being denied by the ACL. Prior to this, they would be denied because of no xlate which the PIX silently dropped (no logs). The 106023 messages can be ignored in most cases if you see a corresponding 302014 message preceeding it. Hope this helps.

Scott

nnw11903 · ‎11-06-2003

Scott,

thanks for your reply.

Its good to know that this comes from a planned feature and not from a bug.

But the situation is more complicated, I think.

First, how should I distinguish between "real" 106023 messages and "false positive" 106023 messages? Looking for corresponding 302014 messages is no option, because this requires "logging informational" (very large logs!) and the administrative overhead in finding the difference of 106023 messages will also be quite high.

OTOH, we have other firewalling devices in the communication path. All of them let the traffic in question pass (for example IOS FW CBAC / Checkpoint FW-1).

There are further logs which are not normal, IMHO:

Nov 06 2003 08:33:49: %PIX-6-302014: Teardown TCP connection 6237076 for DMZ-public:10.30.

13.150/8080 to inside:10.30.7.152/18677 duration 0:00:01 bytes 776 TCP Reset-I

This is ok (RST from Inside).

Nov 06 2003 08:33:49: %PIX-6-106015: Deny TCP (no connection) from 10.30.7.152/18677 to 10

.30.13.150/8080 flags RST on interface inside

The PIX seems to use this packet first to tear down the connection and after that, dropping the packet instead of forwarding it to the host ("flags RST" in the second log). Also, the Message ID is different than 106023.

There was already a sanity check before the RST packet arrives at the PIX inside, because it passed a FW-1 system.

Second example:

Nov 06 2003 08:31:04: %PIX-6-302016: Teardown UDP connection 6235939 for outside:130.75.1.

40/53 to DMZ-public:10.30.13.152/53 duration 0:00:01 bytes 49

Zhis is ok (Reply to DNS query, no more data to come).

Nov 06 2003 08:31:04: %PIX-4-106023: Deny udp src outside:130.75.1.40/53 dst DMZ-public:62

.153.103.142/53 by access-group "acl-outside"

Again, there was also a sanity check of that packet, because before it arrives at the outside of the PIX it has to pass a IOS CBAC enabled device.

I hope this helps explaining things a little bit better.

Before, during and after the upgrade PIX OS from 6.3(1) to 6.3(3) there was no configuration change in the PIX or any other related device.

Best regards,

Christian

scoclayton · ‎11-06-2003

Christian,

Thank you for the information and the investigative work you have done on this. I do not know of any known issues relating to this behavior and I do not think we have enough hard evidence to make a determination on this. My suggestion at this point would be to collect sniffer traces showing this along with the corresponding syslog messages and then open a TAC case for review of the issue. Again, I have seen some folks reporting this issue since 6.3(3) was released but in all cases thus far, we have been able to attribute these extra log messages to the new design behavior. Good luck.

Scott