ACE default handling of the TCP Window Scale option

Unanswered Question
Dec 9th, 2009

Can someone tell me a reasonable explanation why the default action of the ACE is to clear the window-scale option in SYN-segments? We were long baffled why some customers (specifically the ones with larger latencies) were suffering from poor TCP transfer speeds and resets. Packet captures confirmed the fact that the ACE was clearing window-scale options, causing the advertised receive windows to be remarkably small as the scaling factors were removed.

Isn't intermediate tampering with the window-scale option a widely known source for TCP performance problems? Window scaling is after all a widely used TCP extension. That's why I'm curious to know why such default behaviour from the ACE.

Our problem was of course finally resolved by explicitly allowing the window-scale option using a parameter map.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
john.fredrik.ol... Thu, 04/15/2010 - 01:40

Hi.

We are experiencing the same issue. Could you post your parameter map which changes this behaviur? As I understand I can actively set a new scaling-factor, but that is not what I want. I want it to just forward it untouched. Did you manage to accomplish that?

tmiiluvaa Thu, 04/15/2010 - 01:55

Hi

Here you go:

parameter-map type connection TCPIP-PARAM-MAP
  tcp-options window-scale allow

If you want to double check whether window-scale is actually allowed for a certain connection, here's how you can do it:


- Check the connection id with the sh conn command

- Then for more detailed connection statistics use the command show np 1 me-stats "-c "

The latter command shows the negotiated TCP options.

sachinga.hcl Thu, 04/15/2010 - 02:45

HI Timo,

As you know if you allows the ACE to use a window scale factor that essentially increases the size of the TCP send and receive buffers. The sender specifies a window scale factor in a SYN segment that determines the send and receive window size for the duration of the connection.

You have defined tcp-options scale allow in the parameter map but have not specified to which scale factor this should scale to.


The TCP window scaling feature adds support for the Window Scaling option in RFC 1323. It is recommend that increasing the window size to improve TCP performance in network paths with large bandwidth, long-delay characteristics. This type of network is called a long fat network (LFN).

But you have not configure a TCP window-scale factor which is essential for network paths with high-bandwidth, long-delay characteristics, using the "set tcp window-scale " command.

The window scaling extension expands the definition of the TCP window to 32 bits and then uses a scale factor to carry this 32-bit value in the 16-bit window field of the TCP header.

You can increase the window size to a maximum scale factor of 14.

Typical applications use a scale factor of 3 when deployed in LFNs but in you case if this is a LFN you are using default which is 0.

So here in your case you can select how much scale factor is needed to be defined from the range 0-14 , otherwise even if you have defined in the tcp-options windlow-scale allow, but as you have not define to what factor it should scale to , so it will always use the default scale factor which is 0, means no scaling at all.

To set the TCP window-scale factor to 3, add the second command as well in your parameter map:


host1/Admin(config-parammap-conn)# tcp-options window-scale allow        ---> Defined already

host1/Admin(config-parammap-conn)# set tcp window-scale 3                      ----> Needed to be added else "default scale factor 0" will be taken


You can check in the following URL at cisco docs wiki for how ACE handles Connection at Layer 4 (L4) and Layer 7 (L7):

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_(ACE)_Module_Troubleshooting_Guide,_Release_A2(x)_--_Troubleshooting_Connectivity

Here are the details of how a tcp-options scale windows works:

The window scale extension expands the definition of the TCP
window to 32 bits and then uses a scale factor to carry this 32- bit
value in the 16-bit Window field of the TCP header (SEG.WND in RFC-793).
The scale factor is carried in a new TCP option, Window Scale.
This option is sent only in a SYN segment (a segment with the SYN bit on),
hence the window scale is fixed in each direction when a connection is opened.
(Another design choice would be to specify the window scale in every TCP segment.

It would be incorrect to send a window scale option only when the scale factor changed,

since a TCP option in an acknowledgement segment will not be delivered reliably
(unless the ACK happens to be piggy-backed on data in the other direction).
Fixing the scale when the connection is opened has the advantage of lower
overhead but the disadvantage that the scale factor cannot be changed
during the connection.) The maximum receive window, and therefore the
scale factor, is determined by the maximum receive buffer space.
In a typical modern implementation, this maximum buffer space is set
by default but can be overridden by a user program before a
TCP connection is opened. This determines the scale factor,
and therefore no new user interface is needed for window scaling.

Window Scale Option The three-byte Window Scale option
may be sent in a SYN segment by a TCP.
It has two purposes:
(1) indicate that the TCP is prepared to do both send and receive window scaling, and
(2) communicate a scale factor to be applied to its receive window.
Thus, a TCP that is prepared to scale windows should send the option,
even if its own scale factor is 1. The scale factor is limited to
a power of two and encoded logarithmically,
so it may be implemented by binary shift operations.

TCP Window Scale Option (WSopt):
Kind: 3 Length: 3 bytes
+---------+---------+---------+
| Kind=3 |Length=3 |shift.cnt|
+---------+---------+---------+

This option is an offer, not a promise; both sides must send
Window Scale options in their SYN segments to enable window
scaling in either direction. If window scaling is enabled,
then the TCP that sent this option will right-shift its true
receive-window values by 'shift.cnt' bits for transmission in
SEG.WND. The value 'shift.cnt' may be zero (offering to scale,
while applying a scale factor of 1 to the receive window).

This option may be sent in an initial segment (i.e., a
segment with the SYN bit on and the ACK bit off).
It may also be sent in a segment, but only if a Window
Scale op- tion was received in the initial segment.
A Window Scale option in a segment without a SYN bit should be ignored.


The Window field in a SYN (i.e., a or ) segment itself is never scaled.

Using the Window Scale Option

A model implementation of window scaling is as follows


All windows are treated as 32-bit quantities for storage in
the connection control block and for local calculations.
This includes the send-window (SND.WND) and the receive- window
(RCV.WND) values, as well as the congestion window.

* The connection state is augmented by two window shift counts,
Snd.Wind.Scale and Rcv.Wind.Scale, to be applied to the
incoming and outgoing window fields, respectively.

* If a TCP receives a segment containing a Window Scale
option, it sends its own Window Scale option in the segment.


* The Window Scale option is sent with shift.cnt = R, where R
is the value that the TCP would like to use for its receive window.


* Upon receiving a SYN segment with a Window Scale option containing
shift.cnt = S, a TCP sets Snd.Wind.Scale to S and sets Rcv.Wind.Scale
to R; otherwise, it sets both Snd.Wind.Scale and Rcv.Wind.Scale to zero.

* The window field (SEG.WND) in the header of every incoming segment,
with the exception of SYN segments, is left-shifted by Snd.Wind.Scale
bits before updating SND.WND: SND.WND = SEG.WND << Snd.Wind.Scale
(assuming the other conditions of RFC793 are met,
and using the "C" notation "<<" for left-shift).


* The window field (SEG.WND) of every outgoing segment, with the
exception of SYN segments, is right-shifted by
Rcv.Wind.Scale bits: SEG.WND = RCV.WND >> Rcv.Wind.Scale.

TCP determines if a data segment is "old" or "new" by testing
whether its sequence number is within 2**31 bytes of the left
edge of the window, and if it is not, discarding the data as "old".

To insure that new data is never mistakenly considered old and vice- versa,

the left edge of the sender's window has to be at most 2**31 away
from the right edge of the receiver's window. Similarly with the
sender's right edge and receiver's left edge. Since the right and
left edges of either the sender's or receiver's window differ by
the window size, and since the sender and receiver windows can be
out of phase by at most the window size, the above constraints imply
that
2 * the max window size must be less than 2**31,
or max window < 2**30 Since the max window is 2**S
(where S is the scaling shift count) times at most 2**16 - 1
(the maximum unscaled window),
the maximum window is guaranteed to be < 2*30 if S <= 14.

Thus, the shift count must be limited to 14 (which allows windows of 2**30 = 1 Gbyte).

If a Window Scale option is received with a shift.cnt value exceeding 14,
the TCP should log the error but use 14 instead of the specified value.
The scale factor applies only to the Window field as transmitted
in the TCP header; each TCP using extended windows will maintain
the window values locally as 32-bit numbers. For example,
the "congestion window" computed by Slow Start and Congestion Avoidance
is not affected by the scale factor, so window scaling will not introduce
quantization into the congestion window.

Hope it will resolve your issue, else get back to us for further discussion.

Best regards,

Sachin Garg

Message was edited by: sachinga.hcl

john.fredrik.ol... Thu, 04/15/2010 - 03:22

Thanks for your replies.

But does one need to set the "set tcp window-scale x" in addition to "tcp-option window-scale allow" also for traffic that is just passing the ACE (routed) or does this only applies to traffic originating from the ACE itself (proxied traffic to rservers ie). I do not really want to modify the servers/klients scaling factor, just keep it untouched.

sachinga.hcl Thu, 04/15/2010 - 03:48

Hi John,

As you know you can enable these option in a parameter map and that particular parameter map, you must associate  it to a policy-map to activate it.

So it will not activate on all the traffic either it is passing or originating to the ACE , it will affect only that portion of your traffic which comes under that particular policy-map and that policy-map is/are associated to class-maps (for exact kind of traffic filter of your requirement)

Rest of the traffic is untouched.

You have to define the parameter-map for each policy-map to get it work otherwise it will work selectively only on  those policy-maps to whom  it is associated to , and rest of the traffic passing or originating from you ACE will not see these parameter-map options.

Even if you define this tcp-optin window scale parameter  in some of you parameter-map and not associate it to any policy-map, then even these options enabled and exist in your ACE config they never work or seems non-existing.

Best regards,

Sachin Garg

Message was edited by: sachinga.hcl

john.fredrik.ol... Thu, 04/15/2010 - 04:17

Thanks.

I understand that parameter-maps must be applied to policies assossiated with 'something'.

But what you are saying is that the tcp scaling-option in the tcp header is by default not touched as long as there is no parameter-map/policy-map associated to that traffic?

We are experiencing that not to be true. Consider this simple config:

access-list inbound line 10 extended permit ip any any
access-list outbound line 10 extended permit ip any any

class-map type management match-all ICMP-ALLOW-CLASS
  2 match protocol icmp any
policy-map type management first-match mgmt-policy

  class ICMP-ALLOW-CLASS
    permit

service-policy input mgmt-policy

interface vlan 1
  description Outside

  ip address 10.0.0.5 255.255.255.248
  alias 10.0.0.4 255.255.255.248
  peer ip address 10.0.0.6 255.255.255.248
  access-group input inbound
  no shutdown
interface vlan 2

  description ServerLAN
  ip address 192.168.0.2 255.255.255.0

  alias 192.168.0.1 255.255.255.0
  peer ip address 192.168.0.3 255.255.255.0
  access-group input outbound
  no shutdown

ip route 0.0.0.0 0.0.0.0 10.0.0.1

Then, when a client from i.e. 172.16.0.1 (coming in om vlan1) connects to a server residing somewhere on vlan 2 (i.e. 192.168.0.10), and both server and client have scaling enabled, it seems that if the ACE clears that option and set it to "0".

If so, I need to put in a parameter map, like this:

parameter-map type connection allow-scale
  tcp-options window-scale allow

class-map match-any tcp-options
  2 match any          <- to keep it simple for now

policy-map multi-match tcp-options
  class tcp-options
    connection advanced-options allow-scale

interface vlan 1

service-policy input tcp-options

Now, if the server and client otherwise would agree on a scaling factor of 4, this wil also be true for connections passing the ACE?

And if I for example do an additional

parameter-map type connection allow-scale
  set tcp window-scale 3

The server/client connection would decrease its scaling factor to 3?

What I consider odd is that the ACE with the first simple configuration actually clears the scaling-option.

We are running ACE Service Modules in 6509-chassis, version A2(1.6a).

sachinga.hcl Thu, 04/15/2010 - 05:41

Hi John,

Is it happening in your environment the things you have shared in your last reply because in most cases if it is server -initiated connection , then they do not initiates for window-scale option when they sent SYN packet like in the case of windows 2003 TCP window scaling is negotiated on demand in Windows Server 2003, based on the value set for the SO_RCVBUF Windows Sockets option when a connection is initiated. Additionally, the Window Scale option is used by default on a connection if the received SYN segment for that connection as initiated by a TCP peer contains the Window Scale option. Windows Server 2003 TCP does not initiate connections with window scaling by default. To instruct the Windows Server 2003 TCP stack to attempt to negotiate a larger receive window size by making use of the Window Scale option, set the Tcp1323Opts registry value to 1.

It is possible if the client sends in the SYN packet a request for scale option server might also agrees for scaling the window.

Even some application do not support the scale option at all.

Can you be more elaborative with your description as you say at one place that ACE seems to clears that option and set it to "0".

Are you adding it to default policy map or to particular policy map. As in your case it is seeming you are doing it on your default traffic which is for all the traffic.

Its not clear . Can you share more of your configuration from the ACE.

Best regards,

Sachin Garg

Message was edited by: sachinga.hcl

john.fredrik.ol... Thu, 04/15/2010 - 05:55

My example config is actually the entire config for that context. For this example the ACE works as a regular router. No more policies.

The 'Server' and 'Client' is actually xChariot agents on Ubuntu Clients with tweaked kernel. When sending traffic with 64Kb window we get the results as expected. When we increase the window to 256K and above we get the same speed as with 64K (3 ms rtt, and around 250Mbit/s throughput).

Somewhere in the path the scaling-option is removed from the TCP-header (that's what I ment by 'setting it to '0', wich might be inaccurate, sorry about that). Wiresharking on the packet in front of the xChariot-nodes, we know that both the nodes have scaling enabled. The first post in this thread indicated that this is a common issue with traffic passing the ACE-module. When this is said we do not know 100% for sure that we can blame ACE. There are FWSMs, MPLS, ipsec and so on also in the path, but they are close to being checked out ok.

tmiiluvaa Thu, 04/15/2010 - 10:53

sachinga.hcl wrote:

So here in your case you can select how much scale factor is needed to
be defined from the range 0-14 , otherwise even if you have defined in
the tcp-options windlow-scale allow, but as you have not define to what
factor it should scale to , so it will always use the default scale
factor which is 0, means no scaling at all.

Is it really so? Then how does one allow the original window-scale option unaltered through the ACE? Would the only way be by disabling TCP Normalization altogether on the ACE interfaces? That would be quite harsh in my opinion.

Here is a connection entry example from a connection that has only tcp-options window-scale allow applied to it. Even though the parameter-map does not explicitly use the set tcp window-scale command, the connection statistics show a Window scale factor of 2:

Connection ID:seq: 687272[0xa7ca8].10
  Other ConnID    : 648423[0x9e4e7].2
  Proxy ConnID    : 0[0x0].0
  Next Q    : 0[0x0]

X.X.X.X:49316 -> Y.Y.Y.Y:143 [RX-NextHop: TX] [TX-NextHop: TX]
  Flags:  PAT: No  DynNAT: No  Implicit PAT: No On_Reuse: No
  L3 Protocol     : IPv4                L4 Protocol    : 6
  Inbound Flag    : 1
  Interface Match : Yes
    Interface MatchID: 45
  EncapsID:ver    : 281:0               TCP ACK delta  : 0x28f45128
  MSS             : 1460                TOS Stamp       : 0
  Repeat mode     : No          ARP Lookup      : No
  TCP Window Check: No
  ACE ID          : 9296                NAT Policy ID       : 0
  Post NAT hop    : 0
  Packet Count    : 54          Byte Count          : 3809
  TCP Information: (State = 3)
    Window size   : 32586               Window scale    : 2
    FIN seen      : No          FIN/ACK seen    : No
    FIN/ACK exp   : No          Close initiator : No
    FIN/ACK expval: 5b40000             Last seq        : 7858c6e8
   timestamp_delta: 0           Last ack        : 51ff9a01
    No Trigger    : 0           Trigger Status   : 0
    Timestamp : 43f0f364
  TCP options negotiated:
    Sack:Allow          TS:Allow        Windowscale:  Allow
    Reserved: Allow     Exceed MSS:  Deny       Window var: Allow

hostmaster@auto... Wed, 09/14/2011 - 16:39

This was such a promising post as it described my problem exactly with the ACE Module

A2(3.3).

I am seeing the client's WS and SACK options being cleared by the ACE when traffic is only routed through the ACE (i.e., not loadbalanced), and there are no connection paramater maps to match anything.

The issue results from whatever is the *default* tcp normalization behavior on the vlan interface.  From reading the security guide on what normalization takes place on an interface (not a policy map) by default, WS and SACK are not any of them.  Only IP flags are touched, not TCP options.

So why by default does the ACE clear WS and SACK for routed traffic with no connection maps?

I can *prove* this has nothing to do with connection maps since simply setting "no normalization" on the client-side vlan interface allows the WS and SACK to pass unaltered.  But I rather not disable normalization if I can help it, but this may be my only recourse.

Actions

Login or Register to take actions

This Discussion

Posted December 9, 2009 at 11:38 AM
Stats:
Replies:10 Avg. Rating:
Views:7181 Votes:0
Shares:0

Related Content

Discussions Leaderboard

Rank Username Points
1 1,551
2 369
3 333
4 228
5 212
Rank Username Points
5