Trouble gettng TCP State bypass to function properly

[Joe Scharbrough]

I have a situation were we require asymmetrical routing. I have turned on TCP-State-Bypass and if I do a show conn I can see the connection with a B flag

TCP OUTSIDE 10.10.20.15:1157 INSIDE 192.168.135.15:80, idle 0:00:12, bytes 564, flags UIOB

However the connection fails and the syslog gives me this:

 ASA-session-6-106015: Deny TCP (no connection) from 192.168.135.15/80 to 10.10.20.15/1159 flags SYN ACK on interface INSIDE

So I get a connection but then I get denied with a SYN ACK which I was under the impression that what TCP-State Bypass did was tell the firewall to ignore these packets and pass them along.

Needless to say the traffic to the web app is failing.

What am I missing I have read everything I can find but I have not been able to figure this out. I am running ASA ver 8.4.2

Thanks to all that read and help I am stuck in the mud here and probably overlooking the obvious

[Vibhor Amrodia]

Hi ,

I think there is a misconfiguration in regards to the TCP state Bypass policy as this does not seem to match the interesting traffic.

The Flag for TCP state BYPASS is "b" and not "B" which is for initial SYN from outside. Quite similar :)

You need to verify the configuration and if it is correct you should see the correct flag for the connection. You would also see something quite clear in the syslog that Bypass policy is working.

You can use this command or the packet tracer for checking if the traffic is hitting this policy or not ?

show service-policy flow <>

http://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/S/cmdref3/s12.html#pgfId-1327492

Let me know if you have any other queries.

Thanks and Regards,

Vibhor Amrodia

[Joe Scharbrough]

Thanks for your reply. I got called away to put out more urgent fires and am back on this issue now. I will let you know what I learn. Again thanks for your help

Joe

[Joe Scharbrough]

OK so I cleared out my config an tried a very simple TCP State bypass with this:

class-map global-class-TCP-ByPass
 match any

policy-map global-policy
 class global-class-TCP-ByPass
  set connection advanced-options tcp-state-bypass

service-policy global-policy global

When I attempt to access the web page I get this

ciscoasa# sh conn
5 in use, 8 most used
TCP OUTSIDE 10.10.20.15:1086 INSIDE 192.168.135.15:80, idle 0:00:41, bytes 564, flags b
TCP OUTSIDE 10.10.20.15:1084 INSIDE 192.168.135.15:80, idle 0:02:46, bytes 564, flags b

and my syslog shows this:

%ASA-session-6-106015: Deny TCP (no connection) from 192.168.135.15/80 to 10.10.20.15/1090 flags SYN ACK on interface INSIDE

What have I missed?

[Vibhor Amrodia]

Hi,

I see the connection now being bypassed for the TCP state checks.

As per the syslog , if the show conn output was around the same time , it would explain the issue as the source port 1090 for 10.10.20.15 in the show conn output.

We can actually change the embryonic timeout from default 30 Secs to a higher value.

Thanks and Regards,

Vibhor Amrodia

[Joe Scharbrough]

Still an issue. Something is wrong but I can't see it

this is what I have, trying to keep it simple, this is a proof of concept  lab environment prior to moving it to production

class-map global-class-tcp=state-bypass
 match any

policy-map global-policy
 class global-class-tcp=state-bypass
  set connection timeout embryonic 0:00:00 idle 1:00:00
  set connection advanced-options tcp-state-bypass

service-policy global-policy global

This yields this output:

On the ASA:

ciscoasa# sh conn
4 in use, 14 most used
TCP OUTSIDE 10.10.20.15:1053 INSIDE 192.168.135.15:80, idle 0:03:08, bytes 570, flags b

In syslog:

%ASA-session-6-302303: Built TCP state-bypass connection 611 from OUTSIDE:10.10.20.15/1053 (10.10.20.15/1053) to INSIDE:192.168.135.15/80 (w.x.y.z /80)

the next line in the syslog is

%ASA-session-6-106015: Deny TCP (no connection) from 192.168.135.15/80 to 10.10.20.15/1054 flags SYN ACK on interface INSIDE

I am not sure where to go. I can see that a TCP state-bypass has been built but the syn ack is there causing a drop.

Very confused. I know I am missing something here. Any clues.

[Vibhor Amrodia]

Hi,

Would you be able to apply capture on the INSIDE and Outside interface on the ASA device for this traffic. Also apply the ASP captures and take the outputs on these captures simultaneously.

ASP captures:-

capture asp type asp-drop all buffer 3333333

Also , is there any NAT for this traffic on the ASA device ?

You can use this Email address also to send these captures:- vamrodia@cisco.com or give me the details of the TAC SR.

Thanks and Regards,

Vibhor Amrodia

[Joe Scharbrough]

Your posts have been extremely helpful. I am using a simulator to test this prior to implementing it in production so I can easily make changes and test different things. I think I have the issue surrounded, now if I can just get it to work.

Here is where I am now.

When I do a try to go to the inside website from isp B the health asa shows conn

Health(config)# sh conn
4 in use, 15 most used
TCP OUTSIDE 10.10.20.15:1059 INSIDE 192.168.135.15:80, idle 0:36:14, bytes 0, flags b

So I figure the tcp state bypass worked on this ASA

When I look at the ADMIN ASA to see the packets leaving the network Sh Conn shows nothing and the syslog shows

%ASA-session-6-106015: Deny TCP (no connection) from 192.168.135.15/80 to 10.10.20.15/1068 flags SYN ACK on interface INSIDE

Now what I find interesting is if I add ICMP to the access list TCP State Bypass setup on Admin ASA Ping works with out it I get the syslog error of no matching session and with HTTP I get the no connection SYN ACK.

I have learned so much by doing this but it is not working yet. I think I am very close but I am missing something.

I appreciate your help.

[Vibhor Amrodia]

Hi,

I think it is pretty clear from the topology that the issue is with this design.

Why are you sending the traffic from the inside Network router to these two interface on different ASA devices.

What is the router configuration pointing to the ASA devices ?

Are you doing load balancing on the router ? Please share the configuration from the router.

Thanks and Regards,

Vibhor Amrodia

[Joe Scharbrough]

Ok Let me see if this clears things up. I have created a better diagram of the design

Most of the users enter through the outside address b to get to the Web-APP. This works well. Straight in and out. The issue is with users who must enter through outside address A. They get to the Web-APP but are directed out the Health ASA.

This is exactly as demonstrated in this Cisco Document:http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/conns_tcpstatebypass.html

With this diagram

So I am following the TCP State Bypass instructions but when I look at the exiting traffic it is showing this error on the Health ASA:

%ASA-session-6-106015: Deny TCP (no connection) from 192.168.135.15/80 to 10.10.20.15/1067 flags SYN ACK on interface INSIDE

The same packets received this in the log on ADMIN ASA when they entered the network:

%ASA-session-6-302303: Built TCP state-bypass connection 466 from OUTSIDE:10.10.20.15/1066 (10.10.20.15/1066) to INSIDE:192.168.135.15/80 (outside nat /80)

According to the article referenced above this should work or am I just reading all this wrong.

I can't tell you how much I appreciate your helping me understand this.

[Vibhor Amrodia]

Hi,

This clears it up a bit more on the issue.

I think the Bypass is working fine on the Admin ASA and now as the packet [SYN-ACK] on the Health ASA , we would need a similar policy on this ASA to force it to not inspect the sequences of packets received on this Firewall as well as the other one.

Thanks and Regards,

Vibhor Amrodia

[Joe Scharbrough]

Ok I have isolated it to this. It builds a tcp state bypass connection on each firewall as long as the packet is coming from the outside. When a packet hits the firewall from the inside there is no building of the tcp bypass connection.

I build this in ASDM and used a global policy which should apply it to all intrerfaces, correct?

I will experiment with the building it on the inside but I think I have already done that and it still fails to show anything at all in the Show Conn and the logs show nothing happening.

[Vibhor Amrodia]

Hi,

I think you just need to configure the TCP state Bypass on the Other ASA device and it will allow the SYN-ACK to go through without dropping it.

This should resolve the issue.

Thanks and Regards,

Vibhor Amrodia

[Joe Scharbrough]

reat now I have packets flowing but I had do to some extra stuff and I am not sure why I had to do this extra stuff. I am using ASDM 6.4 and IOS ASA 8.4(2) just to clarify
I could not get it to work by just using ASDM and adding a Service Policy Rule. I also had to add an ACL to the interface and this is somehting that I did not find in any Cisco Dacumentation so I am trying to figure out what I did wrong and why the "text book" iconfiguration did not work.

This is my actual config and it builds the TCP State Bypass and Packets leave the firewall but I have a Syn SynAck RST on the host but more on that later.
In order to get this to work I had to add this line:
access-group INSIDE_mpc in interface INSIDE
I could find no reffernce to this step in any documantation.

ADMIN(config)# sh run
ASA Version 8.4(2)
!
interface GigabitEthernet0
 nameif INSIDE
 security-level 100
 ip address 192.168.1.2 255.255.255.0
!
interface GigabitEthernet1
 nameif OUTSIDE
 security-level 0
 ip address 192.168.121.2 255.255.255.0
!
object network webserver
 host 192.168.135.15
object network outsideNAT
 host 192.168.121.4
object-group service DM_INLINE_SERVICE_1
 service-object icmp
 service-object tcp destination eq www
 service-object tcp destination eq https
object-group service DM_INLINE_SERVICE_2
 service-object icmp
 service-object tcp destination eq www
object-group protocol TCPUDP
 protocol-object udp
 protocol-object tcp
object-group service DM_INLINE_SERVICE_3
 service-object icmp
 service-object tcp-udp destination eq www
 service-object tcp destination eq ftp
 service-object tcp destination eq ftp-data
access-list OUTSIDE_access_in extended permit object-group DM_INLINE_SERVICE_1 any object webserver log disable
access-list INSIDE_access_in extended permit object-group DM_INLINE_SERVICE_2 object webserver any log debugging
access-list INSIDE_access_in extended permit icmp object 135LAN any
access-list OUTSIDE_access_in_1 extended permit object-group DM_INLINE_SERVICE_3 any object webserver
access-list INSIDE_access_in_1 extended permit tcp object webserver any eq www inactive
access-list INSIDE_mpc extended permit tcp object webserver any
!object network webserver
 nat (any,any) static 192.168.121.3
!
nat (INSIDE,OUTSIDE) after-auto source dynamic any interface
access-group INSIDE_mpc in interface INSIDE
route OUTSIDE 0.0.0.0 0.0.0.0 192.168.121.1 1
route INSIDE 192.168.135.0 255.255.255.0 192.168.1.1 1
!
class-map global-class
 match default-inspection-traffic
class-map TCP-BYPASS
 match access-list INSIDE_mpc
class-map tcp_bypass
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global-policy
 class global-class
  inspect http
  inspect icmp
policy-map INSIDE-policy
 class TCP-BYPASS
  set connection timeout embryonic 0:00:00 idle 1:00:00
  set connection advanced-options tcp-state-bypass
!
service-policy global-policy global
service-policy INSIDE-policy interface INSIDE
: end

So now the packets do flow and I have TCP State bypass working, but on the host that originates the request the webpage fails to show up. I get this in Wireshark
334    13:45:57.977250    10.10.20.15    172.16.15.244    TCP    62    web2host > http [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
335    13:45:58.127720    192.168.121.3    10.10.20.15    TCP    62    http > web2host [SYN, ACK] Seq=0 Ack=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
336    13:45:58.127743    10.10.20.15    192.168.121.3    TCP    54    web2host > http [RST] Seq=0 Win=0 Len=0

What do I need to do to get past this last step. I am just starting to read about policy Nat. Is that something I should explore.

Thanks again for all your help
Joe

[Vibhor Amrodia]

Hi,

I don't think you would need that ACL on the Inside Interface.

Also , can you please provide the Wireshark Captures and IP address information for the Client and Server.

Thanks and Regards,

Vibhor Amrodia