Downloading from Apple Mountain Lion App Store Hangs 2901 router

[johnhart]

Cisoers,

I have a repating 2901 router failure when people attempt to download Apple Mac OS X Moutnain Lion upgrade from App Store.

The 2901 just hangs following getting a series of ZBFW packet drop failures:

001928: Jul 26 22:37:18.783 UTC: %APPFW-4-HTTP_PROTOCOL_VIOLATION: HTTP protocol violation (0) detected - session 192.168.223.109:49310 184.25.254.67:80 on zone-pair ZP-PRIVATE-OUT class ccp-protocol-http appl-class ccp-http-blockparam

001929: Jul 26 22:37:20.871 UTC: %APPFW-4-HTTP_DEOBFUSCATION: Deobfuscation signature (15) detected - session 192.168.223.109:49369 66.235.138.44:80 on zone-pair ZP-PRIVATE-OUT class ccp-protocol-http appl-class ccp-http-blockparam

001930: Jul 26 22:37:22.779 UTC: %FW-6-DROP_PKT: Dropping tcp session 192.168.223.130:49217 184.31.204.244:443 on zone-pair ZP-PRIVATE-OUT class ccp-insp-traffic due to  Stray Segment with ip ident 0

The failure results in the ACT Light stopping to blink and the SYS Light remains on solid Green and the entire router hangs.

I cannot SSH to it, all logging to console stops and the only way I can recover the router is by powering it off and on again.

This is very alarming as this is a very common download site and I am finding router is hanging consistently and repeatly when people go there.

Does anyone have any suggestions?

This looks like a major bug in IOS.

Regards,

John.

[johnhart]

Ciscoers,

as suspected this appears to be a problem with ZBFW.

As a work around I have moved HTTP inspection down to the end of my policy list, so TCP protocol policy take priority of HTTP application policy and now people can download again.

So the work around for the time being appears to be to disable HTTP inspection.

I am very surprised that I appear to be the first person who has reported a problem here, as this is a major web site that is having a problem with HTTP inspection.

I hope that cisco responds with a patch or particular configuration resolution.

Regards,

John.

[paclark01 · Accepted solution]

I had a similar issue with a 2811 router and IDS. Transfers would start fine but would eventually slow down to a crawl. I ended up upgrading to a 15.x IOS version and adding the ooo global parameter map to increase the reassembly buffers. I think that's what ended up fixing it in the end.

parameter-map type ooo global

  tcp reassembly queue length 512

  tcp reassembly momory limit 16384

Hope it helps.

[johnhart]

Hi Peter,

thanks very much for the suggestion.

I did a check on the 2901 and the config has very small (default) allocations:

parameter-map type ooo global

tcp reassembly timeout 5

tcp reassembly queue length 16

tcp reassembly memory limit 1024

tcp reassembly alarm off

As per your suggestion I have update the sizes (the 2901 has 2GB RAM) moved HTTP application inspection back up to it overides straight tcp protocol inspection.

parameter-map type ooo global

   tcp reassembly timeout 5

   tcp reassembly queue length 512

   tcp reassembly memory limit 16384

   tcp reassembly alarm off

I have a couple of further Apple Mac's that need to be updated to Mountain Lion so will test download again when updating these machines.

Cheers,

John.

[#5]

I had similar issue with a Cisco 887VA (C887VA-W-E-K9) running

Cisco IOS 15.1(4)M4 while downloading Mac OS "IOS" 10.8 ;-)

I had to disable the Trend Micro content filtering by removing

service-policy urlfilter ... from the HTTP filter.

policy-map type inspect POM_INSIDE_TO_OUTSIDE

class type inspect CLM_INVALID_SOURCE

  drop log

class type inspect CLM_INSIDE_TO_OUTSIDE_HTTP

  inspect

  service-policy urlfilter POM_INSIDE_TO_OUTSIDE_HTTP

I think there is a bug in IOS trying to deal with HTTP session downloading

big files (>4G). I had similar issue 3 months ago while trying to download

a full movie from Xbox Live Marketplace.

Which version of IOS solved your issue ?

[paclark01]

At least in my experience it wasn't the version of IOS (although 15 solved a couple of other wierd things I was seeing) but massively increasing the OOO buffers that fixed the problem and allowed inspect to be used.

YMMV.

[#7]

Increasing the OOO buffers did not work for me but my small router

have "only" 1G (and is fanless ready to cook eggs when it hangs

I can use "only" inspect (ZBF) to track HTTP session but I cannot use

Trend Micro deep packets inspection while downloading Mac OS X 10.8

dmg 4G file.

Are you also using Trend Micro content filtering ?

( https://supportforums.cisco.com/docs/DOC-8028 )

[paclark01]

Actually, no, I'm just using the standard IOS IDS signatures, not the Trend Micro stuff.

[johnhart]

Hi Peter,

I was waiting for Mountain Lion aware VMWare Fusion release arrived before testing this again.

With the arrival of VMWare Fusion 5, I have retested a Mountain Lion download with HTTP Inspection on and changes to buffer size as per your original note and all goes ok, downloaded 4GB without a hickup.

Thanks very much for providing the corrective configuration.

Regards,

John.

[paclark01]

Glad I could be of service.