Common WAAS/WCCP issues on interactions with Security Devices

Document

Feb 12, 2011 12:31 AM
Feb 12th, 2011

Introduction

As you know, WAAS is changing the layer 4 info of the packets it optimize. Security devices usually don't like so much when a device in the path is adding/changing some fields of the packet so some issues may arise when those changes are detected.

WCCP can also create some issues as it can encapsulate parts of the communication in GRE or when the device the traffic is sent to starts spoofing the reply from the real server.

Here is a summary of all the common problems you can run into and how they can be solved.

WAAS optimized traffic going through an ASA/FWSM

Problem definition:

It is common to have the WAE optimized traffic going through a Firewall in a setup like this one:

Firewall.jpg

As stated before, the WAE is modifying the TCP header of the packet and two of those changes will be detected by the Firewall:

1.) WAE is adding TCP option 0x21 to the packet

By default, the firewall will clear all of the non well-known TCP options in a packet so it will remove the WAAS option 0x21. This will prevent auto discovery to happen and will thus also prevent optimization.

2.) WAE is bouncing the TCP sequence number of the 3WHS ACK by 2Gb

Once the Edge WAE is sending the ACK from the 3WHS of an optimized connection, it increases the sequence number of the TCP connection by 2Gb to be able to differentiate the packets which have been optimized and the one which were not. When this packet is hitting the Firewall, it detects that it is out of the allowed TCP window and drops it. If you issue a show asp drop on the firewall, you'll see the value of "TCP packet SEQ past window" increasing.

Solution on ASA 7.2(2) or FWSM 3.1(11) and before:

First of all, you'll need to create a policy map that will allow the option 0x21 to go through the Firewall unstripped:

Allowing option 0x21 through
access-list All-TCP extended permit tcp any any
!
class-map Match-All-TCP
    match access-list All-TCP
!
tcp-map Allow21
    tcp-options range 33 33 allow
!
policy-map global_policy
    class
Match-All-TCP
        set connection advanced-options Allow21
!
service-policy global_policy global

Now to take care of the sequence number jump, you'll have to disable the sequence number randomization of the Firewall so that it doesn't modify it but you'll also need to disable TCP state checking for the traffic which is optimized:

Allowing TCP sequence jump
static (inside,outside) X.X.X.X X.X.X.X netmask Z.Z.Z.Z norandseq nailed
static (outside,inside) Y.Y.Y.Y Y.Y.Y.Y netmask Z.Z.Z.Z norandseq nailed
!
failover timeout -1

As weird as it seems, the failover timeout command is needed to have the nailed keyword effective on a static nat command

As you can see, disabling totally TCP state checking for all the optimized traffic defies the purpose of a Firewall and is a bit drastic so an improvement has been implemented in the newer software releases.

Solution on ASA 7.2(3) or FWSM 3.2(1) and after:

To make it a bit more user friendly and less impacting, a new feature has been introduced: WAAS inspection.

If you enable it on your firewall, it will automatically let the WAAS optimized traffic go through the Firewall.

Enabling WAAS inspection
policy-map global_policy
    class inspection_default
        inspect waas

For you reference, here are the release notes of the versions the feature was introduced in : ASA 7.2(3) FWSM 3.2(1)

WAAS optimized traffic going through a GRE over IPSEC or DMVPN tunnel

Problem definition:

When the optimized traffic is sent over a WAN, you often want it to be encrypted and you might end up with the following setup:

GRE.jpg

In such cases and with the default MSS values, the overhead of the WAAS optimization and of the Encryption can cause the IP packet to be fragmented and it can lead to 2 different types of issue:

1.) Packets with DF bit set

When a packet is set to be fragmented and has the DF bit set, the device supposed to fragment it will drop it instead and will send an ICMP “fragmentation needed and DF bit set” message to the source (Type 3 code 4).

The problem is that this ICMP message won't be redirected by WCCP and thus the Edge WAE will never be aware that the packet was dropped and will simply retransmit it after timeout.

2.) Fragmented packets

Even when the fragmented packet doesn't have the DF bit set, it will cause issues but on the Core WAE this time. Since WCCP will only redirect packet with TCP headers, it will only redirect the first fragment and the other ones will bypass redirection since they don't have it. This will cause the WAE to miss parts of the communication and will prevent the traffic from being optimized correctly.

Solution:

There are two commands on the WAE that will have him decrease the MSS of the connections going through it. Based on your network and on the overhead which will be added to your packet, you can adjust those values to prevent fragmentation from occurring. You can fine tune the MSS either on the WAN side:

tfo tcp optimized-mss <value>

or on the LAN one:

tfo tcp original-mss <value>

Once those are lowered, it should prevent the fragmentation from occurring.

For you reference, here are the entries for those in the commands guide: "tfo tcp optimized-mss" "tfo tcp original-mss"

WAAS optimized traffic going through an IPS

Problem definition:

When you setup your WAAS network, you might need to pass optimized traffic through an IPS running in Inline mode like this:

IPS.jpg

As for the Firewall, an IPS in Inline mode will run some checks on the layer 4 information. As the firewall does, it will clear the option 0x21 preventing TFO auto discovery and will report the sequence number changes done by the WAE. It can also report false positive attacks triggered by the files services handling of CIFSAO.

Solution:

1.) Change the Virtual Sensor mode to Asymmetric

By default the Virtual Sensor TCP inspection mode is set to Strict to prevent attackers from bypassing the IPS inspection by sending packets out of order. Because of the sequence shift, it can cause issues when inspecting the WAAS optimized traffic. If you set the mode to Asymmetric, the IPS won't normalize the TCP sequence number.

To change the mode on IDM, here is what you need to do:

-Go to "Configuration > Policies > IPS Policies"

-Select the virtual sensor the WAE traffic will be inspected by and click on "Edit > Advanced Options"

-Change the normalizer mode to Asymmetric:

Asymmetric.jpg

-Click on "Ok > Apply"

-Reboot the Sensor

If you want to change the mode via the CLI, here is how you need to do:

Changing normalizer mode via CLI
ips# conf t
ips(config)# service analysis-engine
ips(config-ana)# virtual-sensor vs0
ips(config-ana-vir)# inline-TCP-evasion-protection-mode asymmetric
ips(config-ana-vir)# exit
ips(config-ana)# exit
Apply Changes?[yes]: yes
Warning: Change of TCP evasion protection mode will not take effect until restart.
ips# reset

2.) Disable signatures that will fire on WAAS accelerated traffic

There are a bunch of signatures that you need to disable if you want to pass WAAS traffic through an IPS. If you don't, you'll get alarms like this one that will keep on firing on your monitoring station:

evIdsAlert: eventId=1243171268831920721 severity=low vendor=Cisco
   originator:
     hostId: IPS
     appName: sensorApp
     appInstanceId: 6631
   time: 2009/09/17 09:08:13 2009/09/17 12:08:13 GMT+03:00
   signature: description=TCP Option Other id=1306 created=20050304
type=anomaly version=S272
     subsigId: 0
     sigDetails: TCP Option Other Detected
     marsCategory: Info/Misc
   interfaceGroup: vs0
   vlan: 0
   participants:
     attacker:
       addr: locality=OUT X.X.X.X
       port: 0
     target:
       addr: locality=OUT 0.0.0.0
       port: 0
       os: idSource=unknown relevance=unknown type=unknown
   summary: final=true initialAlert=1243171268831920576
summaryType=Regular 3
   alertDetails: Regular Summary: 3 events this interval ;
   riskRatingValue: targetValueRating=medium 50
   threatRatingValue: 50
   interface: ge1_1
   protocol: tcp

This is the signature that will clear the TCP options when going through the IPS but there are others which will fire because of the sequence number change. Here is a list of which one you should disable:

Sig ID
Subsig ID
NameDescription
13060TCP Option Other Here
133012TCP Drop - Segment Out Of Order Here
133017TCP Drop - Segment out state order Here
133018TCP Drop - Segment out of window Here
133019TCP Drop - TCP timestamp option detected when not expected Here
30300TCP SYN Host Sweep Here

If you are using CIFSAO, you might also want to disable signature 5581/0 SMB Remote Srvsvc Service Access Attempt as it can also get triggered.

WCCP through an ASA

Problem definition:

If you are setting up web caching/filtering on that your cache has to be located in a DMZ, you might end up with the following setup:

CE.jpg


If this is how your network is connected you might run into two main issues:

1.) Reverse path forwarding denying spoofed traffic from your cache

If you have reverse path forwarding enabled on your interfaces, the Firewall will compare the Source of the packets he is receiving with his routing table. If he wouldn't route packets with this IP through the interface he got the packet on, he would drop it. If you have your cache engine spoofing the Web sites reply from the DMZ interface, the ASA will drop the reply as he is expecting to receive traffic from such IP to come to the outside interface and not to the DMZ one.

2.) GRE encapsulated WCCP redirected traffic without GRE return

If the traffic from the router to the cache engine is GRE encapsulated and the return traffic is not, the ASA will drop the SYN/ACK for the session is it didn't see the original matching SYN for this connection.

Solution:

1.) Disable reverse path forwarding

This one is pretty obvious. To prevent uRPF from dropping the traffic, just disable it on the interface your cache is connected to.

Here is how it can be done through the CLI:


asa(config)# no ip verify reverse-path interface dmz

If you want to do it on ASDM, just go to "Configuration > Firewall > Advanced > Anti-Spoofing" and set the value to False for your interface.

2.) Disable TCP state check

To prevent the ASA from discarding the SYN/ACK it gets, you'll need to have your Firewall bypass TCP state checking. If your ASA is running a version earlier then 8.2(1) or if your FWSM is older then 3.2(1), you'll need to implement the same solution based on static NAT which is describe in the first point.

If you are running at least those versions, you can use the "tcp-state-bypass" option which has been introduced to the Modular Policy Framework.

Here is how it can be setup:

Disabling TCP state check via the MPF
access-list tcp-bypass permit tcp any any
!
class−map tcp-bypass
match access−list tcp-bypass
!
policy−map tcp-bypass_policy
    class bypass-traffic
        set connection advanced−options tcp−state−bypass
!
service−policy tcp_bypass_policy DMZ

If you need to do this via ASDM, just go to "Configuration > Firewall > Service Policy Rules" and add a new policy bound to the interface your cache is connected to.

On this policy, go to "Rule Actions > Advanced options" and enable state bypass from there:

Bypass.jpg

In those situations, you might want to enable WCCP directly on the ASA. Just keep in mind that if you do so, the users and the cache needs to be connected to the ASA through the same interface, not through different ones.

Related Information:

Troubleshooting Prepositioning on WAAS 4.1.1 and above

GRE Redirection in WCCP Creates new tunnel interfaces

Average Rating: 4.8 (4 ratings)

Comments

nfournie Sun, 02/13/2011 - 23:34

Comments on the document itself or on what could be added to it are more than welcome.

Actions

Login or Register to take actions

This Document

Posted February 12, 2011 at 12:31 AM
Stats:
Comments:1 Avg. Rating:4.8
Views:7683 Contributors:1
Shares:0

Related Content