Solved: Disable "Stateful Packet Inspection" on ASA 5510

nwhitesel · ‎09-21-2010

We have a new ASA 5510 appliance that we are using in a fairly simple environment. We have an internal server that is hosting a variety of interface applications that work with our resort's lodging software. There are several interfaces that are operating correctly and connecting to external (Internet) services such as Expedia. Our credit card processor interface, however, is having problems.

The vendor originally told us that all we need to do is create an access rule that opens up port 443 for incoming traffic from their web server: XXX.XXX.190.218. We did this, yet their test application keeps failing. For the sake of argument, they had me temporarily bypass the firewall and the service worked. The issue was elevated to their senior engineer, and he said that the culprit is most likely SPI (Stateful Packet Inspection), which their service is incompatible with. He instructed me to disable it for that access rule.

I see that there is some level of packet inspection under the Service Policy Rules screen, but it appears that port 443 is not being inspected by default, and frankly, I don't even think it is possible to inspect port https. Can anyone tell me how I can make sure that SPI si turned off for that application? Is SPI the culprit, or are there some troubleshooting steps I can take to identify the root cause? I'd be happy to answer any clarification questions that you may have.

Thanks in advance for any replies!

Magnus Mortensen · ‎09-21-2010

By any chance do you have a websense server or some kind of filter/IDS on the inside? I have seen that kind of traffic pattern when there is a websense server that is monitorring traffic in a promiscuos mode. Those kind of filters will simply spoof a reset if they see https traffic they do not like. - magnus

Posted from my mobile device.

View solution in original post

Jitendriya Athavale · ‎09-21-2010

i am not too sure how the stateful behaviour has got somehting to do with this, if you want to bypass staeful check to this you can try this out

http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a0080b2d922.shtml

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpstatebypass.pdf

but i am not too sure how this would help as i do not ho wthis server bahaves, could you please apply captures and elabaorate more about the services and if it opens any dynamic ports

nwhitesel · ‎09-21-2010

For clarification, here is a capture of the access rule.

Kevin Redmon · ‎09-21-2010

The ASA is a stateful firewall and does support Deep Packet Inspection. However, as you stated, HTTPS is not a protocol that can be inspected and modified - at least not by the ASA. The devices that do impact HTTPS are playing "Man in the Middle" - unencrypting and re-encrypting the HTTPS data. Again, the ASA is NOT capable of doing this.

The issue that you are seeing can be any number of things. The two key troubleshooting steps that we have are:

1.) Packet captures.

Capture the data between the two hosts, following the link below:

2.) Syslogs (preferably at the 'debugging' level).

At the time of the issue, gather all 'debugging' level logs that you can for the relevant flow.

3.) Packet-tracer

If it is a "simple" flow (ie non-VPN, dropping on the first packet, etc.), packet-tracer is a great tool. For this flow, the packet-tracer command would likely look like:

packet-tracer input tcp 5555 443 detailed

This will give us a play-by-play and tell us each step of the ASA Finite State Machine that the flow goes through. The key takeaway here is the "result" of each step - if it says "Drop", this is likely our guilty party.

4.) ASP Drops:

Monitoring the output of 'show asp drop' at the time of the issue may give you some insight as to why the ASA may be dropping the flow - or at least narrow it down to a few options. Just prior to running the test connection, you may want to do a 'clear asp drop', run the test, and then immediately look at the 'show asp drop' output. This can also be supplemented with 'capture capasp type asp-drop ' to assist with narrowing it down.

If you can provide us the output from these steps, that will hopefully give us some added insight as to what the problem may be. If you are able to isolate the issue with these troubleshooting steps, please be sure to mark this thread as answered.

Best Regards,

Kevin

nwhitesel · ‎09-21-2010

Hi Kevin,

Thanks for the prompt reply!

Here are the results of some of the tests you wanted me to run:

1. Packet Captures

I couldn't see the link you referenced, but I ran the Packet Capture Wizard through the ASDM on the ASA 5510. Here are the results after running the test twice:

14 packets captured

1: 10:18:51.391627 12.180.11.111.64431 > 216.235.190.218.443: S 1322654048:1322654048(0) win 8192	nop	wscale 8	nop	nop	sackOK>
2: 10:18:51.460150 216.235.190.218.443 > 12.180.11.111.64431: S 3428970047:3428970047(0) ack 1322654049 win 5840
3: 10:18:51.461203 12.180.11.111.64431 > 216.235.190.218.443: . ack 3428970048 win 64860
4: 10:18:51.462286 12.180.11.111.64431 > 216.235.190.218.443: P 1322654049:1322654123(74) ack 3428970048 win 64860
5: 10:18:51.530413 216.235.190.218.443 > 12.180.11.111.64431: . ack 1322654123 win 5840
6: 10:18:51.530734 12.180.11.111.64431 > 216.235.190.218.443: R 1322654123:1322654123(0) win 0
7: 10:18:51.531390 216.235.190.218.443 > 12.180.11.111.64431: P 3428970048:3428970122(74) ack 1322654123 win 5840
8: 10:21:47.531970 12.180.11.111.64446 > 216.235.190.218.443: S 527919168:527919168(0) win 8192	nop	wscale 8	nop	nop	sackOK>
9: 10:21:47.630063 216.235.190.218.443 > 12.180.11.111.64446: S 3712120816:3712120816(0) ack 527919169 win 5840
10: 10:21:47.630933 12.180.11.111.64446 > 216.235.190.218.443: . ack 3712120817 win 64860
11: 10:21:47.632383 12.180.11.111.64446 > 216.235.190.218.443: P 527919169:527919243(74) ack 3712120817 win 64860
12: 10:21:47.729576 216.235.190.218.443 > 12.180.11.111.64446: . ack 527919243 win 5840
13: 10:21:47.729835 12.180.11.111.64446 > 216.235.190.218.443: R 527919243:527919243(0) win 0
14: 10:21:47.729942 216.235.190.218.443 > 12.180.11.111.64446: P 3712120817:3712120891(74) ack 527919243 win 5840
14 packets shown

2. Syslogs

We could not get the syslogs to show any activity from the given source and destination. Not sure if we were filtering it correctly, or if there were truly no entries.

3. Packet Tracer

Result of the command: "packet-tracer input outside tcp 216.235.190.218 5555 12.180.11.111 443 detailed"

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

static (Inside,Outside) 12.180.11.111 10.1.5.115 netmask 255.255.255.255 dns

match ip Inside host 10.1.5.115 Outside any

static translation to 12.180.11.111

translate_hits = 10155, untranslate_hits = 2247

Additional Information:

NAT divert to egress interface Inside

Untranslate 12.180.11.111/0 to 10.1.5.115/0 using netmask 255.255.255.255

Phase: 3

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group Outside_access_in in interface Outside

access-list Outside_access_in extended permit tcp host 216.235.190.218 host 12.180.11.111 eq https log

Additional Information:

Forward Flow based lookup yields rule:

in id=0xac1f4498, priority=12, domain=permit, deny=false

hits=1, user_data=0xa8a92cc0, cs_id=0x0, flags=0x0, protocol=6

src ip=216.235.190.218, mask=255.255.255.255, port=0

dst ip=12.180.11.111, mask=255.255.255.255, port=443, dscp=0x0

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xab8b2260, priority=0, domain=permit-ip-option, deny=true

hits=822577, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 5

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group Inside_access_out out interface Inside

access-list Inside_access_out extended permit ip any any log

Additional Information:

Forward Flow based lookup yields rule:

out id=0xab973c78, priority=12, domain=permit, deny=false

hits=83127, user_data=0xa8a92d40, cs_id=0x0, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 6

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

static (Inside,Outside) 12.180.11.111 10.1.5.115 netmask 255.255.255.255 dns

match ip Inside host 10.1.5.115 Outside any

static translation to 12.180.11.111

translate_hits = 10155, untranslate_hits = 2247

Additional Information:

Forward Flow based lookup yields rule:

out id=0xab9719a0, priority=5, domain=nat-reverse, deny=false

hits=892, user_data=0xab970f48, cs_id=0x0, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=10.1.5.115, mask=255.255.255.255, port=0, dscp=0x0

Phase: 7

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

static (Inside,Outside) 12.180.11.111 10.1.5.115 netmask 255.255.255.255 dns

match ip Inside host 10.1.5.115 Outside any

static translation to 12.180.11.111

translate_hits = 10155, untranslate_hits = 2247

Additional Information:

Reverse Flow based lookup yields rule:

in id=0xab971b48, priority=5, domain=host, deny=false

hits=11046, user_data=0xab970f48, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip=10.1.5.115, mask=255.255.255.255, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 8

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Reverse Flow based lookup yields rule:

in id=0xab8f8e18, priority=0, domain=permit-ip-option, deny=true

hits=850514, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 9

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 872833, packet dispatched to next module

Module information for forward flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_tcp_normalizer

snp_fp_translate

snp_fp_adjacency

snp_fp_fragment

snp_ifc_stat

Module information for reverse flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_translate

snp_fp_tcp_normalizer

snp_fp_adjacency

snp_fp_fragment

snp_ifc_stat

Result:

input-interface: Outside

input-status: up

input-line-status: up

output-interface: Inside

output-status: up

output-line-status: up

Action: allow

4. ASP Drop

Here are the results of the command you suggested:

Result of the command: "show asp drop"

Frame drop:

Flow is denied by configured rule (acl-drop) 598

First TCP packet not SYN (tcp-not-syn) 22

TCP failed 3 way handshake (tcp-3whs-failed) 2

TCP invalid ACK (tcp-invalid-ack) 31

FP L2 rule drop (l2_acl) 12

Dropped pending packets in a closed socket (np-socket-closed)

Kevin Redmon · ‎09-21-2010

Sorry that I forgot to include the link to the packet captures - here it is for future reference:

https://supportforums.cisco.com/docs/DOC-1222

Based on the packet captures that you had provided, it seems as though the server at 12.180.11.111 is sending a reset.

6: 10:18:51.530734 12.180.11.111.64431 > 216.235.190.218.443: R 1322654123:1322654123(0) win 0

Depending on where this packet capture was taken, this Reset could be coming from the server and/or this reset proxied by the ASA for some reason. If this was reset by another device, you will likely see a TCP Reset-I or TCP Reset-O. Do a packet capture on the interface that is closest to the 12.180.11.111 device - see if you see the Reset there as well. If you do, you can determine who sent the Reset by looking at the source MAC address via the command 'show capture detail | inc R'. A 'show arp | inc ' will likely point to a culprit.

This is the device that you'll want to focus on next.

Let me know if this helps.

Best Regards,

Kevin

nwhitesel · ‎09-21-2010

We ran the command you posted and the reset command is originating from the interface server where we are running the test from. I have also created a screen capture of our packet capture showing both Ingress and Egress (see below). It seems like the final ack is not making it back to the originating server and I'm wondering if that is casuing the issue.

The test we are running from that server is a terminal based java test to sample send a credit card. Here is a brief connection debug log of that test:

NGS XT Client ConnHandler-51:DEBUG:Thread starting.
NGS XT Client ConnHandler-51:DEBUG:Initializing encryption engines.
NGS XT Client ConnHandler-51:DEBUG:Connecting to Proxy Server.
NGS XT Client ConnHandler-51:DEBUG:Connect to server host
NGS XT Client ConnHandler-51:DEBUG:Gen a client key pair
NGS XT Client ConnHandler-51:DEBUG:Public key :XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

NGS XT Client ConnHandler-51:DEBUG:Public key sent to server, 72 bytes.
NGS XT Client ConnHandler-51:DEBUG:Waiting for receipt of server Public key...
NGS XT Client ConnHandler-51:ERROR: Socket Error during secure connection setup:
Software caused connection abort: recv failed disconnecting.
NGS XT Client ConnHandler-51:DEBUG:Host connect failed.

Also, here is the error code that is returned from the console:

RECEIVE TIME OUT

If this feels like I am getting too much into the specifics of the application, I apologize. Thanks for all of your help.

nwhitesel · ‎09-21-2010

We got some debugging logs to work. Here they are:

6|Sep 21 2010|14:12:04|106015|216.235.190.218|443|12.180.11.111|65324|Deny TCP (no connection) from 216.235.190.218/443 to 12.180.11.111/65324 flags PSH ACK on interface Outside
6|Sep 21 2010|14:12:04|302014|216.235.190.218|443|10.1.5.115|65324|Teardown TCP connection 968830 for Outside:216.235.190.218/443 to Inside:10.1.5.115/65324 duration 0:00:00 bytes 74 TCP Reset-I
6|Sep 21 2010|14:11:16|302014|216.235.190.218|443|10.1.5.115|65320|Teardown TCP connection 968732 for Outside:216.235.190.218/443 to Inside:10.1.5.115/65320 duration 0:00:00 bytes 74 TCP Reset-I
6|Sep 21 2010|14:11:16|106015|216.235.190.218|443|12.180.11.111|65320|Deny TCP (no connection) from 216.235.190.218/443 to 12.180.11.111/65320 flags PSH ACK on interface Outside

Thanks!

Magnus Mortensen · ‎09-21-2010

By any chance do you have a websense server or some kind of filter/IDS on the inside? I have seen that kind of traffic pattern when there is a websense server that is monitorring traffic in a promiscuos mode. Those kind of filters will simply spoof a reset if they see https traffic they do not like. - magnus

Posted from my mobile device.

nwhitesel · ‎09-21-2010

We have a similar appliance from St. Bernard - iPrism. It is set to allow all traffic to our local server, but I can see that it might just be inspecting those packets and sending a weird response. I will take it out of line tomorrow morning to test and will let you know if that was the problem. Thanks!

Namit Agarwal · ‎09-21-2010

Hi ,

According to the packet-tracer command run earlier "packet-tracer input outside tcp 216.235.190.218 5555 12.180.11.111 443 detailed", it appears that the server 216.235.190.218 is a server on the outside of the ASA and it tries to connect to 12.180.11.111(which is on the internal network of the ASA) over port 443.

But if we look at the captures on the ASA "10:18:51.391627 12.180.11.111.64431 > 216.235.190.218.443: S 1322654048:1322654048(0) win 8192"

, it looks like that the connection is being initiated by 12.180.11.111 to the destination 216.235.190.218 over port 443. The key point I am trying to highlight is that the connection is initiated from teh 12.180.11.111 to 216.235.190.218 to port 443 whereas the packet-tracer was run the other way round with the ports exchanged.

So my understanding of the application is that the system 216.235.190.218 is communicating on port 443 and the 12.180.11.111 is not.

Please run the packet-tracer as "packet-tracer input inside tcp 10.1.5.115 64431 216.235.190.218 443 detailed" and provide the output.

Please change the ACL from "access-list Outside_access_in extended permit tcp host 216.235.190.218 host 12.180.11.111 eq https log" to

"access-list Outside_access_in extended permit tcp host 216.235.190.218 eq https host 12.180.11.111 log" and let us know if it helps.

Regards

Namit

nwhitesel · ‎09-22-2010

The iPrism ended up being the root cause of the problem. We set a "do not scan" flag on the iPrism for the inside IP of the server and it cleared the errors. Sorry for not mentioning this network appliance in the first place.

Thanks to everyone for the help!