877 Performance Question(All help is greatly appreciated!)

[snetherland]

We have a newly deployed 877 with 12.4(20)T1 at a customer site. A quick look at my packet switching stats showed me an alarming process-to-fast switching ratio. In fact, all of the LAN interfaces seem to be process-switching the vast majority of inbound and outbound packets. I almost laughed when I saw it because it doesn't seem to be mathematically possible. Below is a sample of what I ran into:

FastEthernet0

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 47562941 2338167514 81068047 1119642

Route cache 0 0 0 0

Total 47562941 2338167514 81068047 1119642

FastEthernet1

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 218243 40857597 558405 121564049

Route cache 0 0 1732 103920

Total 218243 40857597 560137 121667969

FastEthernet2

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 300189 97530100 725119 402520970

Route cache 0 0 150754 9045240

Total 300189 97530100 875873 411566210

FastEthernet3

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 80783709 3000191334 47789058 3211307848

Route cache 0 0 148954 8937240

Total 80783709 3000191334 47938012 3220245088

FastEthernet4

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 516433 69563498 948405 312457764

Route cache 3621645 3230936579 2475650 40662

Total 4138078 3300500077 3424055 312498426

Vlan1

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 794162 287957855 293319 40560793

Route cache 2748817 723858560 3777120 3458134548

Total 3542979 1011816415 4070439 3498695341

As you can probably decipher from the text above, all LAN interfaces are kept in the default Vlan1.

Also, because I could see some of the sharp engineers on here asking me for it, here's some cef stats:

IPv4 CEF Packets passed on to next switching layer

Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag

RP 0 0 1022382 4079 30071 0 63520 0

Packet inspection statistics [process switch:fast switch]

tcp packets: [464206:5422471]

udp packets: [122407:21086]

icmp packets: [8183:100828]

smtp packets: [36:96113]

http packets: [7177:1898505]

ftp packets: [16:0]

pop3 packets: [172:264402]

im-yahoo packets: [0:20]

im-msn packets: [0:252]

IPv4 CEF input features:

Feature Drop Consume Punt Punt2Host Gave route

QoS Drop 79257 0 0 0 0

Virtual Fragment 26 0 0 0 0

Access List 78374 0 0 63832 0

IPSec input clas 511 246882 0 0 0

NAT Outside 0 0 0 247842 0

Total 158168 246882 0 311674 0

IPv4 CEF output features:

Feature Drop Consume Punt Punt2Host New i/f

Post-routing NAT 0 0 0 545754 0

Firewall (inspec 2864 0 0 160405 0

Total 2864 0 0 706159 0

I really appreciate everyone's time and assistance.

[John Blakley]

How many users do you have behind this router? Have you looked at the processor utilization?

--John

[snetherland]

John thanks so much for responding. At the most you will see 40 concurrent users behind this router. Which I completely aggree is borderline at best. In the event that our router becomes overloaded, I have a 1721 ready for deployment.

This was not taken at peak usage but just after:

CPU utilization for five seconds: 2%/1%; one minute: 4%; five minutes: 8%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 413408 22427 18433 0.00% 0.74% 0.93% 0 Check heaps

8 8068 26560 303 0.00% 0.01% 0.00% 0 ARP Input

32 704 366 1923 0.32% 0.06% 0.05% 2 SSH Process

45 11632 712 16337 0.00% 0.01% 0.00% 0 Per-minute Jobs

47 114852 210763 544 0.73% 0.47% 0.57% 0 COLLECT STAT COU

75 50736 55999 906 0.16% 0.34% 0.98% 0 IP Input

82 9468 42252 224 0.00% 0.01% 0.00% 0 Spanning Tree

160 8692 10835 802 0.00% 0.10% 0.27% 0 Crypto PAS Proc

FastEthernet0 is up, line protocol is up

5 minute input rate 342000 bits/sec, 49 packets/sec

5 minute output rate 170000 bits/sec, 39 packets/sec

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output errors, 0 collisions, 2 interface resets

FastEthernet1 is up, line protocol is up

5 minute input rate 1000 bits/sec, 7 packets/sec

5 minute output rate 5000 bits/sec, 5 packets/sec

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output errors, 0 collisions, 2 interface resets

FastEthernet2 is up, line protocol is up

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 1000 bits/sec, 2 packets/sec

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output errors, 0 collisions, 2 interface resets

FastEthernet3 is up, line protocol is up

5 minute input rate 14000 bits/sec, 26 packets/sec

5 minute output rate 636000 bits/sec, 58 packets/sec

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output errors, 0 collisions, 2 interface resets

FastEthernet4 is up, line protocol is up

5 minute input rate 578000 bits/sec, 65 packets/sec

5 minute output rate 141000 bits/sec, 44 packets/sec

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output errors, 0 collisions, 2 interface resets

Vlan1 is up, line protocol is up

5 minute input rate 132000 bits/sec, 43 packets/sec

5 minute output rate 574000 bits/sec, 64 packets/sec

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output errors, 1 interface resets

I will be able to pull stats closer to peak useage in a few hours.

John thanks for your help.

[John Blakley]

Here's a link to give you a couple of things to look at. Although it talks about high cpu utilization, it may give you a few things to look for:

http://supportwiki.cisco.com/ViewWiki/index.php/Troubleshooting_High_CPU_Utilization_in_IP_Input_Process

If you can get the stats during peak hours, I'd be interested to see what the cpu utilization is.

You can also post your config if you want, and we can look through it.

Thanks,

John

[John Blakley]

Oh, and I believe the recommended max amount of users on any 870 series is 20, so if you have 40 users on it at any one time, it's really going to tax that device. You may want to look at moving to the 1700 series that you have.

--John