High CPU utilization one-way

cisco24x7 · ‎03-23-2008

LinuxES-1----(F0/1)Cisco7120_Router(F0/0)----LinuxES-2

LinuxEs_1 IP is 192.168.1.10

LinuxES_2 IP is 4.2.2.3

C7140 IOS version: c7100-jo3s-mz.123-12e.bin

interface F0/1

ip address 192.168.1.1 255.255.255.0

ip nat inside

speed 100

duplex ful

interface F0/0

ip address 4.2.2.2 255.255.255.0

ip address outside

speed 100

duplex full

ip nat outside

access-list 100 permit ip 192.168.1.0 0.0.0.255 any

ip nat inside source list 100 interface F0/0 overload

ip nat inside source static tcp 192.168.1.10 22 interface FastEthernet0/0 22

Everything is connected to a Catalyst 3750 FastEthernet switch.

If I download a large file from LinuxES_1 to LinuxES_2 via SCP and I do it from

LinuxES_1, I get about 8.6MBytes/sec upload. CPU on the router is:

LAB>sh process cpu

CPU utilization for five seconds: 40%/39%; one minute: 40%; five minutes: 40%

If I uploaded a large file from LinuxES_2 to LinuxES_1 via SCP and I do it

from LinuxES_2, I get about 5.2MBytes/sec upload. CPU on the router is:

LAB>sh process cpu

CPU utilization for five seconds: 99%/98%; one minute: 98%; five minutes: 98%

Basically, I am getting half the transfer speed because the router CPU is

at 99% CPU utilization. I don't understand why direction is important in

this case.

Both of these Linux servers are Dell with dual quad-core processors with

8GB RAM and 10k RPM so read/write latencies is not a factor here.

If I put both servers on the same subnet and perform scp, I get about

11.8MBytes/sec transfer each way.

Anyone know why the router behaves this way. Thanks.

Joseph W. Doherty · ‎03-23-2008

In this instance, I would suspect NAT is sensitive to flow direction.

It isn't clear whether you're "pushing" or "pulling" from each host to the other, or "pushing" and "pulling" from the same host. (i.e. former like LinuxES-1 sends/gets from LinuxES-2 followed by LinuxES-2 does same to LinuxES-1 vs. LinuxES-1 sends to LinuxES-1 followed by LinuxES-1 gets from LinuxES-2.) It might make a difference. I would also wonder about the impact of "overload".

[edit]

Rereading your post, I see LinuxES-1 gets from LinuxES-2 and LinuxES-2 sends to LinuxES-1. Correct?

cisco24x7 · ‎03-23-2008

That's correct.

I am on the LinuxES-1 console, I performed

"scp root@4.2.2.3:/var/ftp/large .", I get

8.6MBytes/sec (68.8Mbits/sec) download,

pulling or getting data

I am on the LinuxES-2 console, I peformed

"scp large root@4.2.2.2:/var/ftp/.", I get

5.2MBytes (41.6Mbits/sec) upload, pushing or

sending data.

I noticed that this is also true with other

traffics as well such as FTP and Iperf. In

other words, anything that leaving the

router, I get very good wire-speed and CPU

never goes beyond 50% utilization. Anything

coming into the router, the traffics never

goes beyond 5.2MBytes/sec and CPU is at 99%

utilization.

Any ideas on how to improve this gentlemen?

Thanks.

cisco24x7 · ‎03-23-2008

As a follow-up, I replaced the Cisco 7140

with a VXR7204 and the traffic is about the

same as before. No improvement. The IOS

version on the VXR7204 is 12.4(16) Advanced

IP Services.

I then replaced the VXR7204 with a Checkpoint

Firewall NGx R65 Secureplatform running on a

Dell Server 2550, dual processors 1.0Ghz

with 2GB RAM. The NAT setup on the

Checkpoint is identical to the NAT on the

Cisco 7140 and VXR7204. My throughput goes

up to 11.1MBytes/sec sending and receiving.

Apprently, NAT works differently between

Checkpoint and Cisco. Apparently, from

what I can see, it consumes more CPU on

Cisco than on Checkpoint. Then again,

the architecture is different,

the dell box has much more memory, 2GB RAM

versue 256MB RAM on the Cisco VXR7204, so my

assumption may not be a correct one.

I am going to test this tomorow with a

Cisco Pix525 and see if I see the same

result.

will keep everyone posted.

CCIE Security