ASA seems to be dropping valid TCP SYN packets

Unanswered Question
Feb 29th, 2012

Hello

We have a setup with a MS-TMG - ASA (8.2.4(4) in routing mode) - (internal) Router - FWSM - Router - Exchange with NLB.  We have now the problem that IMAPS is not really working through this setup. It works from internal (without ASA and TMG inbetween), but not reliably through the internet.

There is a rule on the ASA which permits the ports from the TMG to the Exchange NLB address.

We opened a case with Microsoft and they told us that not all tcp-syn packets are received by the Exchange server which were sent by the TMG.

Thus I sniffed on the ASA with a packet capture and indeed, a lot of syn packets were on the interface to the TMG, but not anymore on the interface to the internal router.

I don't find anything really helpful in the ASA log and am at a loss.

This ASA also filters all other internet<->company traffic, so there's a lot of stuff running.

Maybe it's dropped in the ASP, or is the capture maybe not valid?

Here the show asp drop:

ASA01-Internet# sh asp drop

Frame drop:
  Invalid TCP Length (invalid-tcp-hdr-length)                                  1
  Reverse-path verify failed (rpf-violated)                                  319
  Flow is denied by configured rule (acl-drop)                            477077
  First TCP packet not SYN (tcp-not-syn)                                   10212
  TCP data send after FIN (tcp-data-past-fin)                                 41
  TCP failed 3 way handshake (tcp-3whs-failed)                               824
  TCP RST/FIN out of order (tcp-rstfin-ooo)                                 1419
  TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff)                             6
  TCP SYNACK on established conn (tcp-synack-ooo)                              1
  TCP packet SEQ past window (tcp-seq-past-win)                              821
  TCP invalid ACK (tcp-invalid-ack)                                          331
  TCP Out-of-Order packet buffer full (tcp-buffer-full)                      393
  TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout)               1538
  TCP RST/SYN in window (tcp-rst-syn-in-win)                                4228
  TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue)                 500
  TCP packet failed PAWS test (tcp-paws-fail)                              23039
  Early security checks failed (security-failed)                              13
  DNS Inspect id not matched (inspect-dns-id-not-matched)                    148
  FP L2 rule drop (l2_acl)                                                  4674
  Dropped pending packets in a closed socket (np-socket-closed)               26

Last clearing: 16:43:13 CEST Feb 29 2012 by xxxxxxxxxxxx

Flow drop:   
  Flow is denied by access rule (acl-drop)                                    56
  NAT failed (nat-failed)                                                    342
  Inspection failure (inspect-fail)                                         6552

Last clearing: 16:43:13 CEST Feb 29 2012 by xxxxxxxxxxx
ASA01-Internet#

I hope somebody could help me to debug this better

Thanks,

Patrick

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
johuggin Wed, 02/29/2012 - 08:30

Patrick,

Could you give us a peak at the capture? Also, could you provide the config relevant to this connection (any translation statements, ACLs, policies, etc).

As you can tell from your 'show ASP drop', there are many reasons why the ASA drops packets. The above may help us understand why in this case they are being dropped.

Thanks!

Joey

patoberli Wed, 02/29/2012 - 08:39

Hi Joey

Here the captures showing ingress and egress traffic. It was captured through the ASDM simultaneous. As you can see there are many more TCP Syn packets in the ingress as on the egress (I hope the pictures are visible).

The configuration is a tad more complicated to post, as it's our live config.

vschivuk Wed, 02/29/2012 - 08:38

Hi Patrick,

Please check which of the counters in the 'show asp drop' increment after the intrested traffic passes through ASA. It would be better, if you configure captures for the specific tcp traffic and asp drops, then check which packets are getting dropped.

Following link helps you with packt capture configuration :

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/c1.html#wp2129312

Thanks

Narayana

patoberli Thu, 03/01/2012 - 04:47

That is what I tried, but it did not really work out.

The previous capture was done with these commands (through ASDM):

! DMZ_Transit

! Apply ingress  capture on the DMZ_Transit interface.
capture asdm_cap_ingress match tcp  src_ip 255.255.255.255 dst_ip 255.255.255.255 eq 993
capture asdm_cap_ingress packet-length 1522 circular-buffer buffer 1524288
capture asdm_cap_ingress interface DMZ_Transit


! inside

! Apply egress  capture on the inside interface.
capture asdm_cap_egress match tcp  src_ip 255.255.255.255 dst_ip 255.255.255.255 eq 993
capture asdm_cap_egress packet-length 1522 circular-buffer buffer 1524288
capture asdm_cap_egress interface inside

And as you can see on the capture, around 9/10 of all TCP-SYN were not anymore on the inside, which were on the DMZ_Transit.

vschivuk Thu, 03/01/2012 - 07:24

Hi Patrick,

As mentioned by Joey please provide all relevant configurations for this traffic flow and then the capture outputs from the CLI of ASA on both interfaces. That would help in understanding this better.

Narayana

patoberli Mon, 03/05/2012 - 09:19

Talked with my external support in the mean time. They checked the whole config and think that it might be a software bug. I'll update now (soon) to 8.2.5.22(interims) and hope that this will help.
I'll try to keep you updated.

patoberli Thu, 04/05/2012 - 08:17

I've opened now a TAC case with Cisco and so far it indeed looks to be a software bug. Will try to keep you updated on the progression.

patoberli Thu, 05/03/2012 - 02:24

Small update. TAC case still open.

I did make a discovery now though, it seems the affected packets get dropped because of 'tcp-rst-syn-in-win' in the ASP path. When I explicit capture that asp drop reason, then I more or less have only the affected packets!

Now I just need a solution.

patoberli Mon, 05/07/2012 - 00:48

The issue is solved!

The reason for the drops was a timeout issue on our ASA. We once set the tcp connection timeout to 6 hours (from the default of 1) which is fairly high. It seems now that the TMG had a lower timeout for tcp connections and thus killed some connections from it's table after they timeouted. Then the TMG started to re-use the tcp ports, which our ASA still had in an existing connection, so the asa dropped the valid, but for the ASA duplicate, TCP Syn packets.

After chaning the timeout on the ASA to a much lower time (currently on 7 minutes for IMAP connections, but we are still experimenting with the timeout) it is now working as it should!

Actions

Login or Register to take actions

This Discussion

Posted February 29, 2012 at 8:15 AM
Stats:
Replies:9 Avg. Rating:
Views:4915 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard

Rank Username Points
1 7,861
2 6,140
3 3,170
4 1,473
5 1,446