ZBF, "match protocol http" makes youtube and some other sites unusable

[Utair Corporation]

Hi, Cisco 7201, IOS:

Cisco IOS Software, 7200 Software (C7200P-ADVENTERPRISEK9-M), Version 15.0(1)M8, RELEASE SOFTWARE (fc1)

I have implemented zone based firewall with LAN 2 INET policy like:

class-map type inspect match-any CLASS_LAN2INET

match protocol http

match protocol https

match protocol tcp

match protocol udp

match protocol tcp

policy-map type inspect LAN2INET

class type inspect CLASS_LAN2INET

  inspect

class class-default

  pass

!

many downloads gets stalled, slowed down, sky.fm radio plays with pauses, youtube page get loaded but videos not played.

some (some!) dowloads proceed without problems, like http://ftp.freebsd.org

web pages gets opened mostly without problem

First i blamed ISP, they checked their side and didn't find any problems.

Then is disabled firewall (should have to try it first) and everything started to work like a charm.

Then i narrowed down to removing "match protocol http".

If not use "match protocol http", everything works ok.

Is it a bug, is there a fix? any options to tweak "http"?

[Julio Carvajal]

Hello Utair,

Are you performing any kind of DPI for the HTTP protocol?

If not you can use the following:

ip access-list extended Julio

permit tcp any any eq 80

permit tcp any any eq 443

class-map type inspect Julio

match access-group name Julio

class-map type inspect CLASS_LAN2INET

no match protocol HTTP

no match protocol HTTPS

exit

policy-map type inspect LAN2INET

no class CLASS_LAN2INET

class Julio

     inspect

class CLASS_LAN2INET

     inspect

That way we are still going to inspect traffic but for the HTTP traffic will be inspected as a layer 4 protocol not layer 7.

Try it and let me know,

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

[Utair Corporation]

jcarvaja wrote:
permit tcp any any eq 443
...
no match protocol HTTPS

I wonder, how Cisco inspects HTTPS on L7? It's encrypted, after all..

ip access-list extended Julio
permit tcp any any eq 80
class-map type inspect Julio
match access-group name Julio
class-map type inspect CLASS_LAN2INET
no match protocol HTTP
Try it and let me know,

I came to this workaround myself, but it's "workaround", not a fix.

Just tried on another 2951 router with 15.1(4)M6 IOS. It has the same issue - youtube does not play videos.

How no one noticed such bug?

PS:

when starting youtube video, it spamming with:

Jul 26 11:13:24: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998688742 1500 bytes is out-of-order; expected seq:1998647862. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:25: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1993449876 1500 bytes is out-of-order; expected seq:1993385636. Reason: TCP reassembly queue overflow - session 10.10.10.128:55548 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:26: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1993451336 1500 bytes is out-of-order; expected seq:1993387096. Reason: TCP reassembly queue overflow - session 10.10.10.128:55548 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:29: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998717942 1500 bytes is out-of-order; expected seq:1998653702. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:33: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998719402 1500 bytes is out-of-order; expected seq:1998655162. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:42: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998720862 1500 bytes is out-of-order; expected seq:1998656622. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:52: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998722322 1500 bytes is out-of-order; expected seq:1998658082. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

parameter-map has default settings, there is only one user on this router (fresh from the box)

[Julio Carvajal]

Hello,

I am going to provide you a solution again not a workaround:

1-

came to this workaround myself, but it's "workaround", not a fix.

Why do you think this is a work-around. I mean this is the fix!

Why would you inspect a protocol at layer 7 if you are not doing any kind of DPI, it just does not make sense.

That's why you will need to do it at layer 4 , to make the checks faster or live with the fact the DPI engine will look and inspect the packet altough you do not have any policies for that......

2-Check the logs you are getting man, read them and tell me if they are expected on a network:

out-of-order; expected seq:1998655162. Reason: TCP reassembly queue overflow - !!!

I can see a lot of out of order packets (OOO) this is the down side of you network, no need to blame the ZBFW.

Now for this you have 2 solutions:

1- The real solution: Remove the OOO packets of the network

2- Implement the OOO support that means configure the parameter-map type out-of-order so we can hold this OOO packets in a queue (the default is of 5 mileseconds to receive all of the segments in ooo and we can hold up to 16 OOO packets)

Note: Remember to rate all of my posts, I definetly take my time to write this replies

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

[Utair Corporation]

I am going to provide you a solution again not a workaround:
1-
 came to this workaround myself, but it's "workaround", not a fix.
Why do you think this is a work-around. I mean this is the fix!
Why would you inspect a protocol at layer 7 if you are not doing any kind of DPI, it just does not make sense.
That's why you will need to do it at layer 4 , to make the checks faster or live with the fact the DPI engine will look and inspect the packet altough you do not have any policies for that......

This is a basic feature, and it should work out of the box. Youtube isn't a rare service. I might want to filter http access by URIs later and how i will guarantee normal operation to users?

1- The real solution: Remove the OOO packets of the network

They are not out of order, there is queue shortage.

2- Implement the OOO support that means configure the parameter-map type out-of-order so we can hold this OOO packets in a queue (the default is of 5 mileseconds to receive all of the segments in ooo and we can hold up to 16 OOO packets)

What reasonable parameters are?

[Julio Carvajal]

Hello Utair,

So you basically say that I am wrong with the information I have provided before,

This is a basic feature, and it should work out of the box. Youtube isn't a rare service. I might want to filter http access by URIs later and how i will guarantee normal operation to users?

They are not out of order, there is queue shortage.

Well.. Then I wish you the best with this deployment, maybe someone else you really trust will answer here,

For Networking Posts check my blog at

http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal

[Utair Corporation]

So you basically say that I am wrong with the information I have provided before

I say, that "match protocol http", should match HTTP protocol and not cause downloads to stall and youtube not working.

Out of the box, without fine-tuning.

Back on topic:

I tuned parameter-map type ooo global, maxed up queue length am set memory limit up to 128M. Everything started to work.

[Julio Carvajal]

Hello Utair,

Back on topic:

I tuned parameter-map type ooo global, maxed up queue length am set memory limit up to 128M. Everything started to work.

So you basically increase the software queue for OOO packets  as I said with the parameter-map tuning.(that should not exist) and increase the memory for that traffic. I am glad that it works but keep an eye on the network as basically your solution promotes latency (It´s just a work-around while you fix the real issue the OOO)

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

[Utair Corporation]

Well, after a while i ran into:

Jul 21 09:44:21 10.96.32.37 13665: tmn-pleh-7201: Jul 21 09:44:20: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:608968433 1500 bytes is out-of-order; expected seq:608842873. Reason: Unable to get a buffer to hold the packet - session x.x.x.x:59934 to 46.61.155.211:80 on zone-pair zbf_LAN2INET class zbfc_LAN2INET

Increased memory limit to 512M. Messages stopped. I wonder for how long.

How do i get memory usage statistics for inspection engine?

[Julio Carvajal]

Hello Utair,

Do you understand that those are OOO packets right? Those are not normal nor expected packets?

You are basically wasting resources of the router to support invalid packets.

How do i get memory usage statistics for inspection engine?

There is a limit (you should see it on your platafform if you hit the question mark) I dont think we can increase that as the router need memory for it's other tasks

Any other question? Otherwise mark the question as answered

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

[Utair Corporation]

Do you understand that those are OOO packets right? Those are not normal nor expected packets?

Yes, but who do i complain to? Google? UN?

Increasing memory limit helped for couple hours, then situation repeated.

Removed "match protocol http" completely.

[Utair Corporation]

Doesn't "match protocol tcp" track for OOO?

[Julio Carvajal]

Hello Utair,

Yes, but who do i complain to? Google? UN?

In this case the OOO packets come from the inside of the network, so the problem it's on the LAN side

Doesn't "match protocol tcp" track for OOO?

It does as well but here is the difference:

It's well known that the HTTP layer 7 inspection of the ZBFW has limitations with the OOO handeling this is why I recommended to use a match protocol TCP as long as possible.

If you then need to use DPI for the HTTP protocol then you should fix the OOO packets.

I wrote a post regarding a ZBFW enviroment on my blog www.laguiadelnetworking.com, You are more than welcome to visit it.

Cheers,

Julio Carvajal Segura

[Utair Corporation]

Yes, but who do i complain to? Google? UN?
In this case the OOO packets come from the inside of the network, so the problem it's on the LAN side 

It is session direction LAN->INET, but how do you know it's packets from LAN, which caused ooo? Packet size was 1500, which is surely packet with data, not just empty ACK, which is usually sent from client to server on download.

The problem appears on long, downloads and multimedia streams, like youtube or internet radio.

Testing environment was very simple, INET-router-Client PC with fresh Win7 setup with all latest service packs and fixes.

[Julio Carvajal]

Because the firewall is lettting you know that......

For Networking Posts check my blog at http://www.laguiadelnetworking.com

Cheers,

Julio Carvajal Segura