Additional info from Data sheet.

If using VPN module:

The Cisco 2800 Series Module (AIM-VPN/SSL-2) can provide hardware-based IPSec encryption services of 30 and 90 Mbps in the Cisco 2801, 35 and 100 Mbps in the Cisco 2811, 90 and 125 Mbps in the Cisco 2821, and 100 and 150 Mbps in the Cisco 2851 (IPSec IMIX and 1400-byte packets).

The VPN module is enable. Anymore ideas?

R2851-3#sh crypto engine accelerator statistic

Device: AIM-VPN/EPII-PLUS

Location: AIM Slot: 0

Virtual Private Network (VPN) Module in slot : 0

Statistics for Hardware VPN Module since the last clear

of counters 1192084 seconds ago

97506087 packets in 97506087 packets out

90174632666 bytes in 90915789436 bytes out

81 paks/sec in 81 paks/sec out

605 Kbits/sec in 610 Kbits/sec out

58251169 packets decrypted 39254918 packets encrypted

87145415176 bytes before decrypt 3770374260 bytes encrypted

83726767962 bytes decrypted 6447864704 bytes after encrypt

0 packets decompressed 0 packets compressed

0 bytes before decomp 0 bytes before comp

0 bytes after decomp 0 bytes after comp

0 packets bypass decompr 0 packets bypass compres

0 bytes bypass decompres 0 bytes bypass compressi

0 packets not decompress 0 packets not compressed

0 bytes not decompressed 0 bytes not compressed

1.0:1 compression ratio 1.0:1 overall

119919 commands out 119919 commands acknowledged

Last 5 minutes:

700980 packets in 700980 packets out

2336 paks/sec in 2336 paks/sec out

17346343 bits/sec in 17501119 bits/sec out

608470038 bytes decrypted 14480060 bytes encrypted

16445136 Kbits/sec decrypted 391352 Kbits/sec encrypted

1.0:1 compression ratio 1.0:1 overall

HSP details:

hsp_operations : 119935 hsp_sessions : 4

R2851-3#

I am thinking if there is a fragmentation issue here. Since if haredware vpn engine is enabled, the router's CPU should not be that high. Please check the following link to see if it can be a help.

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

" the router's CPU should not be that high."

It should not but it is. I am not sure if the link you provided can resolve my issue since I am not using GRE/IPSec, just straight forward IPSec.

I can say that if I replace the Cisco 2851 with another Checkpoint firewall, I can easily push 500+ mbps IPSec traffics with IPerf. As soon as I put the Cisco 2851 back in place, I am stuck at 8Mbps throughput and that the CPU stays contanst at 96% CPU utilization.

Hi David, per data sheet, 2851 should be able to handle around 100M bps vpn traffic with vpn hardware module. If it is stuck at 8Mbps and have a high cpu, it looks like the packet was processed by CPU instead of VPN module for certain reason.

Can you try packet size 1400 byte when using iperf to do bandwidth testing?

well I did one step better. I set the MTU on both the Cisco 2851 and the Checkpoint firewall to 1400. I also run iperf at 1400 byte packet size. CPU is till 98% at between 8mbps and 9mbps. The bandwidth varies betwen 8mbps and 20mbps but CPU is constant at 99% utilization.

don't change MTU on cisco2851 and checkpoint. Keep them as default 1500. Just use 1400 byte on your iperf testing.

I am not sure what you're trying to achieve here but I changed the MTU on both Checkpoint and Cisco back to 1500 and 1400 bytes on my iperf. Same issue, CPU on 2851 hits 99% utilization at 8mbps.

I even set the mtu on my linux client to 1400. Furthermore, I have full MTU path discovery end-to-end but no luck.

anymore ideas?

Performance testing in data sheet is done by using 1400 bytes packet size. If 1400 bytes packet size is used and all interface MTU on the path is 1500 bytes, packet should not be fragmentated.

Based on your testing, it does not look like a fragmentation issue here.

By the way, which process uses most CPU? Can you post "show process cpu sort"?

R2851-3#sh process cpu sorted

CPU utilization for five seconds: 97%/16%; one minute: 97%; five minutes: 72%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

220 15115924 5603399 2697 34.47% 33.98% 19.34% 0 Crypto Support

6 11381916 3267358 3483 25.11% 24.76% 14.05% 0 Pool Manager

119 48853064 26696407 1829 17.83% 18.10% 18.08% 0 IP Input

252 4201672 8522276 493 2.23% 2.23% 2.11% 0 Crypto PAS Proc

184 404544 353832475 1 0.55% 0.53% 0.45% 0 HQF Shaper Backg

305 524 520 1007 0.47% 0.07% 0.05% 514 Virtual Exec

19 1332452 7192843 185 0.31% 0.28% 0.26% 0 ARP Input

18 2096 48605 43 0.07% 0.00% 0.00% 0 Environmental mo

185 23888 14560169 1 0.07% 0.03% 0.02% 0 RBSCP Background

192 3444 2846848 1 0.07% 0.00% 0.00% 0 Inspect process

37 10656 770731 13 0.07% 0.01% 0.00% 0 Net Background

Hi David, I could not think of anything else. I would like to suggest you to open a TAC case for further assistance.

I am not sure if VPN module is bad. If you want, you can try a new one.

Capture "show ip traffic" and "show buffer" before and after you testing to see if there is anything suspected.

Hi,

Many thanks for your feedback. I will open a TAC case in a few days.

For what it worths, I replaced this router with another identical 2851 but still run into the same issue.

Thanks again.