SA520W - High memory usage, possible fix in 2.2.0 firmware?

florenm1970 · ‎10-09-2012

As suggested by Thomas Watts, I'm starting a new thread to discuss the new SA520W firmware (2.2.0) and a possible resolution to high memory usage I'm experiencing on my network.

My current setup is: 16Mbit DSL > SA520W > SA300-10, all with stock settings (no fancy VLAN's etc.)

I have 4 CentOS 5/6 servers and a Windows 7 Ultimate station connected to the switch. I use CIFS to connect from Windows station to the other Linux servers and send large files. I currently notice the following behavior:

When the file transfer starts, the Intel 1Gbit NIC is nearly saturated, hitting 115MB/sec. After few seconds, the data transfer comes to a halt and the transfer speed drops to around 50MB/sec. If I check the memory usage before the file transfer, it is approximately to 50-60% (on a fresh router reboot). Every time I send large files to other machines, the router memory consumption increases and it does not lower after a reasonable delay. I end-up with high memory near 90% and the only solution I have is to reboot the router in order to bring it back to 50%.

Now, Thomas told me that this is simply a cosmetic issue, the memory is not actually 90% used. Yet, when the memory hits this threshold, I'm not capable to send files are normal LAN speeds I'm used to. Rebooting the router allows me to send only ONCE (and for few seconds) data at the expected LAN speeds.

I would apreciate any input from Cisco engineers as well other users who experience the same issue. I would also like to know if any related work was done into 2.2.0 firmware and when we expect to have it released to users.

Regards,

Floren Munteanu

Tom Watts · ‎10-09-2012

Hi Floren, thanks for moving this over to a different topic. It makes things a bit cleaner especially is a resolution is reached, then this topic can be archived when/if the correct answer is discovered for your scenario.

A couple questions

Are you currently running the 2.1.7.1 code?
Are you using IPS?
Are you using Protectlink services?

I've been working with the SA's for about 2 years now. At first through the firmware progressions there were many cases escalated to the L2 team because the SA is reporting high memory/cpu.

The symptom perceived in our lab environment, on a fresh reboot, the unit running idle with zero configuration, will go to approximately 56%. About 7-10 minutes idle, it will slowly increment to 60-65%, etc. It may take 2-3 days to increment to 90+ without any network traffic. Dave Hornstein and I did some testing a few weeks back, made a SSL connection and some random features. Dave SSL'd in to the router, tinkered a bit. We did note the utilization if I remember right, around 75-80%. The last service request I escalated to L2, on the 2.1.7.1 firmware, there wasn't a resolution, it was a similar conclusion, it appears a cosmetic problem.

Now, to tackle your issue a bit. You do have a visible problem. Obviously you are noting a network performance issue. Naturally, a cpu or memory utilization, indicating a high percentage I would certainly believe it is attributed to these meters. Assuming the utilization meters are a cosmetic issue, lets disect this a different direction.

Can you verify the 3 questions above?

-Tom
Please rate helpful posts

-Tom Please mark answered for helpful posts http://blogs.cisco.com/smallbusiness/

florenm1970 · ‎10-09-2012

Hi Tom,

See below the answers.

Are you currently running the 2.1.71 code?
Yes
Are you using IPS?
No, the LAN is for internal use (no external users allowed)
Are you using Protectlink services?
No

Hardware wise, I did not changed anything on machines. All boxes have dual Intel EXPI9301CT NIC's (LACP was planned) but I currently use single connections for sanity reasons (disks won't allow greater speeds anyway). Previous to Cisco, I used a Netgear ProSafe router + switch which did not encountered the issues I mention. Honestly, at first I thought I'm dealing with some stupid disk issues on Windows. So I ran a quick test and the stats are proper:

> winsat disk -drive c

> Disk Sequential 64.0 Read 109.62 MB/s 6.5

> Disk Random 16.0 Read 2.47 MB/s 4.4

> Responsiveness: Average IO Rate 2.12 ms/IO 6.9

> Responsiveness: Grouped IOs 8.34 units 7.4

> Responsiveness: Long IOs 5.59 units 7.7

> Responsiveness: Overall 46.63 units 7.1

> Responsiveness: PenaltyFactor 0.0

> Disk Sequential 64.0 Write 117.03 MB/s 6.7

> Average Read Time with Sequential Writes 6.977 ms 5.3

> Latency: 95th Percentile 32.720 ms 3.0

> Latency: Maximum 118.231 ms 7.6

> Average Read Time with Random Writes 13.346 ms 3.7

> Total Run Time 00:01:39.50

As I mentioned before, everything is pretty much stock on router/switch settings. If you have any tips that allow me to identify the cause, I would appreciate the input. What puzzles me is the speed drop and quick memory usage increase. It occurs 7-10 seconds after the transfers begins. It looks like the data transfer hangs for a very short period of time (less than half of second) and the transfer speed decreases from 110-115MB/sec to 50-60MB/sec. The transfer is completed at this speed. No matter how many other files I try to transfer after, the speed won't go higher than 60MB/sec. If I reboot the router, I get the same cycle.

doug_counsil · ‎10-09-2012

We use an SA540 as our gateway/VPN router and have a 24-port Netgear switch connected to it. We have dozens of devices connected to the Netgear switch, including three WAPs (which have dozens of users connected to them at any one time), but only the Netgear is attached to the SA540.

We regularly achieve 950+ Mbps throughput from device to device if they all have gigabit connections. Wirelessly we can easily achieve ~300 Mbps throughput depending on the device, wireless NIC, number of users, load, etc. We have an 1812+ Synology NAS connected to the Netgear using Link Aggregation. We easily achieve 160+ MB/s throughput with that guy.

We use Trend Micro ProtectLink Web and IPS and our RAM usage stays right around 80 - 85%. We reboot monthly, but we have had our SA540 run for over two months without any issues. 2.1.78 proved to be quite stable for us. We are running 2.2.0.7 right now because we are trialing the Verisign VIP functionality.

Sorry to hear that your SA520W is giving you grief. Our SA540 is a work horse. Our Netgear switch does all the work though regarding file transfers.

florenm1970 · ‎10-09-2012

Hi Curtis,

I'm aware of SA540 rock solid performance. I was reading on the forums that others experienced the same behaviour.

doug_counsil · ‎10-09-2012

Oh, our SA540 has been running for four days now and it is at 80% memory usage. We deployed the 2.2.0.7 firmware just this weekend. I'll let you know if we encounter any memory spikes, but we have always hovered around 80 - 85%. I don't see the 2.2.0.x firmware universally decreasing memory usage, but I can see it fixing some bugs that might cause some memory leaks.