Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

ZBF, "match protocol http" makes youtube and some other sites unusable

Hi, Cisco 7201, IOS:

Cisco IOS Software, 7200 Software (C7200P-ADVENTERPRISEK9-M), Version 15.0(1)M8, RELEASE SOFTWARE (fc1)

I have implemented zone based firewall with LAN 2 INET policy like:

class-map type inspect match-any CLASS_LAN2INET

match protocol http

match protocol https

match protocol tcp

match protocol udp

match protocol tcp

policy-map type inspect LAN2INET

class type inspect CLASS_LAN2INET

  inspect

class class-default

  pass

!

many downloads gets stalled, slowed down, sky.fm radio plays with pauses, youtube page get loaded but videos not played.

some (some!) dowloads proceed without problems, like http://ftp.freebsd.org

web pages gets opened mostly without problem

First i blamed ISP, they checked their side and didn't find any problems.

Then is disabled firewall (should have to try it first) and everything started to work like a charm.

Then i narrowed down to removing "match protocol http".

If not use "match protocol http", everything works ok.

Is it a bug, is there a fix? any options to tweak "http"?

Everyone's tags (4)
20 REPLIES

Re: ZBF, "match protocol http" makes youtube and some other site

Hello Utair,

Are you performing any kind of DPI for the HTTP protocol?

If not you can use the following:

ip access-list extended Julio

permit tcp any any eq 80

permit tcp any any eq 443

class-map type inspect Julio

match access-group name Julio

class-map type inspect CLASS_LAN2INET

no match protocol HTTP

no match protocol HTTPS

exit

policy-map type inspect LAN2INET

no class CLASS_LAN2INET

class Julio

     inspect

class CLASS_LAN2INET

     inspect

That way we are still going to inspect traffic but for the HTTP traffic will be inspected as a layer 4 protocol not layer 7.

Try it and let me know,

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

Re: ZBF, "match protocol http" makes youtube and some other site

jcarvaja wrote:

permit tcp any any eq 443

...

no match protocol HTTPS

I wonder, how Cisco inspects HTTPS on L7? It's encrypted, after all..

ip access-list extended Julio

permit tcp any any eq 80

class-map type inspect Julio

match access-group name Julio

class-map type inspect CLASS_LAN2INET

no match protocol HTTP

Try it and let me know,

I came to this workaround myself, but it's "workaround", not a fix.

Just tried on another 2951 router with 15.1(4)M6 IOS. It has the same issue - youtube does not play videos.

How no one noticed such bug?

PS:

when starting youtube video, it spamming with:

Jul 26 11:13:24: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998688742 1500 bytes is out-of-order; expected seq:1998647862. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:25: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1993449876 1500 bytes is out-of-order; expected seq:1993385636. Reason: TCP reassembly queue overflow - session 10.10.10.128:55548 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:26: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1993451336 1500 bytes is out-of-order; expected seq:1993387096. Reason: TCP reassembly queue overflow - session 10.10.10.128:55548 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:29: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998717942 1500 bytes is out-of-order; expected seq:1998653702. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:33: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998719402 1500 bytes is out-of-order; expected seq:1998655162. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:42: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998720862 1500 bytes is out-of-order; expected seq:1998656622. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

surgut-inet-2951#

Jul 26 11:13:52: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:1998722322 1500 bytes is out-of-order; expected seq:1998658082. Reason: TCP reassembly queue overflow - session 10.10.10.128:55547 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

parameter-map has default settings, there is only one user on this router (fresh from the box)

Re: ZBF, "match protocol http" makes youtube and some other site

Hello,

I am going to provide you a solution again not a workaround:

1-

came to this workaround myself, but it's "workaround", not a fix.

Why do you think this is a work-around. I mean this is the fix!

Why would you inspect a protocol at layer 7 if you are not doing any kind of DPI, it just does not make sense.

That's why you will need to do it at layer 4 , to make the checks faster or live with the fact the DPI engine will look and inspect the packet altough you do not have any policies for that......

2-Check the logs you are getting man, read them and tell me if they are expected on a network:

out-of-order; expected seq:1998655162. Reason: TCP reassembly queue overflow - !!!

I can see a lot of out of order packets (OOO) this is the down side of you network, no need to blame the ZBFW.

Now for this you have 2 solutions:

1- The real solution: Remove the OOO packets of the network

2- Implement the OOO support that means configure the parameter-map type out-of-order so we can hold this OOO packets in a queue (the default is of 5 mileseconds to receive all of the segments in ooo and we can hold up to 16 OOO packets)

Note: Remember to rate all of my posts, I definetly take my time to write this replies

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

Re: ZBF, "match protocol http" makes youtube and some other site

I am going to provide you a solution again not a workaround:

1-

came to this workaround myself, but it's "workaround", not a fix.

Why do you think this is a work-around. I mean this is the fix!

Why would you inspect a protocol at layer 7 if you are not doing any kind of DPI, it just does not make sense.

That's why you will need to do it at layer 4 , to make the checks faster or live with the fact the DPI engine will look and inspect the packet altough you do not have any policies for that......

This is a basic feature, and it should work out of the box. Youtube isn't a rare service. I might want to filter http access by URIs later and how i will guarantee normal operation to users?

1- The real solution: Remove the OOO packets of the network

They are not out of order, there is queue shortage.

2- Implement the OOO support that means configure the parameter-map type out-of-order so we can hold this OOO packets in a queue (the default is of 5 mileseconds to receive all of the segments in ooo and we can hold up to 16 OOO packets)

What reasonable parameters are?

Re: ZBF, "match protocol http" makes youtube and some other site

Hello Utair,

So you basically say that I am wrong with the information I have provided before,

This is a basic feature, and it should work out of the box. Youtube isn't a rare service. I might want to filter http access by URIs later and how i will guarantee normal operation to users?

They are not out of order, there is queue shortage.

Well.. Then I wish you the best with this deployment, maybe someone else you really trust will answer here,

For Networking Posts check my blog at

http://www.laguiadelnetworking.com/category/english/


Cheers,


Julio Carvajal

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

Re: ZBF, "match protocol http" makes youtube and some other site

So you basically say that I am wrong with the information I have provided before

I say, that "match protocol http", should match HTTP protocol and not cause downloads to stall and youtube not working.

Out of the box, without fine-tuning.

Back on topic:

I tuned parameter-map type ooo global, maxed up queue length am set memory limit up to 128M. Everything started to work.

Re: ZBF, "match protocol http" makes youtube and some other site

Hello Utair,

Back on topic:

I tuned parameter-map type ooo global, maxed up queue length am set memory limit up to 128M. Everything started to work.

So you basically increase the software queue for OOO packets  as I said with the parameter-map tuning.(that should not exist) and increase the memory for that traffic. I am glad that it works but keep an eye on the network as basically your solution promotes latency (It´s just a work-around while you fix the real issue the OOO)

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

Re: ZBF, "match protocol http" makes youtube and some other site

Well, after a while i ran into:

Jul 21 09:44:21 10.96.32.37 13665: tmn-pleh-7201: Jul 21 09:44:20: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:608968433 1500 bytes is out-of-order; expected seq:608842873. Reason: Unable to get a buffer to hold the packet - session x.x.x.x:59934 to 46.61.155.211:80 on zone-pair zbf_LAN2INET class zbfc_LAN2INET

Increased memory limit to 512M. Messages stopped. I wonder for how long.

How do i get memory usage statistics for inspection engine?

Re: ZBF, "match protocol http" makes youtube and some other site

Hello Utair,

Do you understand that those are OOO packets right? Those are not normal nor expected packets?

You are basically wasting resources of the router to support invalid packets.

How do i get memory usage statistics for inspection engine?

There is a limit (you should see it on your platafform if you hit the question mark) I dont think we can increase that as the router need memory for it's other tasks

Any other question? Otherwise mark the question as answered

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

Re: ZBF, "match protocol http" makes youtube and some other site

Do you understand that those are OOO packets right? Those are not normal nor expected packets?

Yes, but who do i complain to? Google? UN?

Increasing memory limit helped for couple hours, then situation repeated.

Removed "match protocol http" completely.

New Member

Re: ZBF, "match protocol http" makes youtube and some other site

Doesn't "match protocol tcp" track for OOO?

Re: ZBF, "match protocol http" makes youtube and some other site

Hello Utair,

Yes, but who do i complain to? Google? UN?

In this case the OOO packets come from the inside of the network, so the problem it's on the LAN side

Doesn't "match protocol tcp" track for OOO?

It does as well but here is the difference:

It's well known that the HTTP layer 7 inspection of the ZBFW has limitations with the OOO handeling this is why I recommended to use a match protocol TCP as long as possible.

If you then need to use DPI for the HTTP protocol then you should fix the OOO packets.

I wrote a post regarding a ZBFW enviroment on my blog www.laguiadelnetworking.com, You are more than welcome to visit it.

Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

Re: ZBF, "match protocol http" makes youtube and some other site

Yes, but who do i complain to? Google? UN?

In this case the OOO packets come from the inside of the network, so the problem it's on the LAN side

It is session direction LAN->INET, but how do you know it's packets from LAN, which caused ooo? Packet size was 1500, which is surely packet with data, not just empty ACK, which is usually sent from client to server on download.

The problem appears on long, downloads and multimedia streams, like youtube or internet radio.

Testing environment was very simple, INET-router-Client PC with fresh Win7 setup with all latest service packs and fixes.

Re: ZBF, "match protocol http" makes youtube and some other site

Because the firewall is lettting you know that......

For Networking Posts check my blog at http://www.laguiadelnetworking.com

Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

ZBF, "match protocol http" makes youtube and some other sites un

Because the firewall is lettting you know that......

Where?

Log message states session details, not single packet direction.

ZBF, "match protocol http" makes youtube and some other sites un

So eventough the router let;s you know the OOO packets are from LAN->INET you still think they are from INET-LAN

Well, I think my time is done here (I have answered like 20 times, not a single point ) but that's okey...


Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

ZBF, "match protocol http" makes youtube and some other sites un

Here is your points, but. Don't get me wrong, i want to get deep into this problem.

So eventough the router let;s you know the OOO packets are from LAN->INET you still think they are from INET-LAN

Router said:

session 10.10.10.128:55548 to 46.61.155.145:80 on zone-pair zp_LAN2INET class zbfc_INSPECT_OUT

It was Session created by zone-pair and class.

But it also stated that:

Dropping TCP Segment: seq:1998720862 1500 bytes

There is no single byte of data being send to youtube except for empty data ACK packets. Which tells us its INET->LAN direction.

ZBF, "match protocol http" makes youtube and some other sites un

Hello Utair,

You told my that the PC is directly connected to the router right?

If that's correct you are right, it does not make sense!!

Looks like the router is showing information incorrectly

For Networking Posts check my blog at http://www.laguiadelnetworking.com/category/english/

Cheers,

Julio Carvajal Segura

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
New Member

ZBF, "match protocol http" makes youtube and some other sites un

Come to URL filtering and stuck with this problem again. URL filtering requires "match protocol http" in class-map and there is no way i could influence our service provider to even try resolving out-of-order packets problem. They are biggest national ISP and has worst helpdesk ever.

Is there a way to fine-tune inspection engine to just ignore out-of-order problem?

ZBF, "match protocol http" makes youtube and some other sites un

Hello Utair,

The only way to ignore those packets it's by no inspecting the traffic (using pass) but if you do that then you will not be able to use a L7 policy-map.

As we have spoke before the parameter-map is the only variable available for such traffic.

Regards,

Jcarvaja

Looking for some Networking Assistance? Contact me directly at jcarvaja@laguiadelnetworking.com I will fix your problem ASAP. Cheers, Julio Carvajal Segura http://laguiadelnetworking.com
1500
Views
40
Helpful
20
Replies