Slow Throughput through FWSM, but only after a few hops...

Unanswered Question
May 6th, 2010

I'm experiencing a very big degredation in throughput when I route through my FWSM. But it's a strange issue.

We have a 6513 with a FWSM. If I take that vlan out of the firewall group, and don't route through the FWSM, I can hit 900+Mb/s using PerfSonar/Iperf as a testing tool, to another perfsonar box accross the country/WAN. As soon as I put that vlan in the FWSM group, and test to that same box, I get only 50-100Mb/s at best. Sometimes much lower.

Here is the strange part----We've setup a few test scenerios, one with a test box outside the firewall (but conected to the 6513) and one routing through the FWSM. We get acceptable performance 600Mb/s.

If I test from the outside box to a far away host, I get 900Mb/s

If I test from the inside box to that same far away host, less than 100Mbs.

What would be causing that much drop accross the wan ONLY when going through the FWSM. (like i said, directly in front of the FWSM, it's fine)

We have version 4.0.6 and I've tried the 'sysopt np completion-unit' trick.

we (Cisco and I) are inthe process of analying various traffic caps, but I wanted to throw this out there to see if anyone else has experienced this. This isn't a new issue, my client has had this problem since day one, SCP's accross a WAN max out at 250k in some instances and just drop/fail.

After looking at the configs, Cisco says there is nothing out of the ordinary, and nothing overtaxed.   Buffer problem? one thing we noticed is window size dropped from 180k to ~50k.


I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
jan.nielsen Thu, 05/06/2010 - 14:35

What version of fwsm code are you running ?

Have you tried to use the sysopt np completion-unit command ?

Command History

sysopt np completion-unit

To enable the hardware completion unit in the accelerated path network  processors (NPs), which ensures that packets are forwarded out in the  same order they were received in the ingress queues of the NPs., use the  sysopt np completion-unit command in global  configuration mode. To restore the default setting, use the no form of this command.

sysopt np completion-unit

no sysopt np completion-unit



This command was introduced.

Usage Guidelines

When you enable this command in the admin context, it is enabled for the  whole device. You cannot configure this command separately for each  context.

Because of design constraints:

This command only works for  packets forwarded by the accelerated path. Packets that require  inspection, for example, go through the session management path or the  control path, and are not affected by this command.

This command does not guarantee  that the order of multicast packets are maintained in routed mode

This command does not guarantee  the order of fragmented packets or packets to be fragmented by the FWSM  because of its MTU.


The following example enables the hardware completion unit:

hostname(config)# sysopt np completion-unit 

Panos Kampanakis Fri, 05/07/2010 - 08:13

Except from the "no completion" trick you can use "sysopt connection tcp sack-permitted".

sometimes TCP download issues for specific hosts are due to the hosts using Selective ACKS and the FWSM not allowing them.

I hope it helps.


REUBEN CHIN Wed, 12/08/2010 - 09:32

Hi Rob,

In your environment with the FWSM, are you running the command, 'sysopt np completion-unit'  in a setup where the FWSM has a single context or multiple context?  How were your results thus far?

I am considering using this feature, but have concerns on how this feature will adversely affect multiple contexts.

If you know or have experience with this command and bugs (if any) associated to using this feature, please let me know.

Why would this be a feature allowed on the FWSM but not enabled by default?  I'm wondering if there are Best Practice type articles surrounding this feature.

Any advice or related bugs would be greatly appreciated.



Shobith K Fri, 02/04/2011 - 01:59

Above document works. I have implmented the below options without any outage and the data transfer throughput increased 3 times !!!! I was pretty worried about implmenting sysopt np completion unit in live environment, but nothing went wrong.

Optimised Firewall Configuration

Interface MTU set to 1500


TCP MSS adjusted to

1460 bytes

TCP Windows Scale and

SACK permitted

TCP Sequence Number

Randomization disabled

NP Completion Unit



This Discussion