Latency Issue

WILLIAM STEGMAN · ‎07-31-2009

I'm having latency issue on an Internet router that is only 2 hops away from the LAN and the host monitoring the latency. It's gigabit all the way through and no duplex or speed errors are apparent on any of the interfaces in between. The path crosses my ASA before it gets to the Internet router, and that segment appears to be the issue. I can ping across other interfaces of the ASA to DMZ segments and get less than 1ms response times, but when pinging the Internet router, I get the results below.

--- x.x.x.x ping statistics ---

207 packets transmitted, 207 packets received, 0% packet loss

round-trip min/avg/max/stddev = 1.556/107.754/674.219/162.069 ms

The firewall's int and Internet router's interfaces on this segment show full duplex with no crc or input errors. The ASA is running at 93% of its memory being used, which has pretty much always been the case since I plugged it in, and the Internet router has a somewhat high proc utilization of 60%. I'm not sure where else to look for the cause. I've attached some int stats. Does anyone have a direction I could start focusing on?

thank you,

Bill

Joseph W. Doherty · ‎07-31-2009

You might start with why your Internet router's 61% is 51% non-interrupt (i.e. process) CPU.

WILLIAM STEGMAN · ‎08-01-2009

The latency issue does appear to go away when usage is light. I didn't think a 3640 would have trouble keeping up with one default route to the ISP, but we did recently disable split tunneling for remote access users. I also upgraded the IOS on this in case it was an IOS bug.

--- x.x.x.x ping statistics ---

421 packets transmitted, 421 packets received, 0% packet loss

round-trip min/avg/max/stddev = 1.542/1.965/66.489/4.078 ms

vishwancc · ‎08-01-2009

Hi Bill,

What is the ping responce from the LAN to the asa inside and ASA outside,and the ping responce from the ASA outside to router inside and from router inside to ASA inside and outside.

If the ping is good from ASA inside from LAn and having siginificant drop to ASA outside you have to check the ASA .

If it is fine to asa in and out and problemis from ASA out to router we have to check the router try to use any other LAN port on router for testing if you have any.

How about the CPU utilization on ASA.

Chao

Vishwa

WILLIAM STEGMAN · ‎08-01-2009

I can ping through the ASA to a DMZ host hanging off one of its interfaces, and those replies are typical of a LAN, around 1 ms. I don't have any additional interfaces available on my Internet router, and it's a little difficult to ping the ASA's interfaces due to inherent security. I followed a doc that was supposed to allow for that very thing, but it didn't work out. The ASA's cpu is low, but memory is high. However, the mem usage is always high, even when no one is in the office.

vishwancc · ‎08-02-2009

Hi Bill,

Could you provide network connectivity diagram .

Chao

Vishwa

WILLIAM STEGMAN · ‎08-02-2009

The problem segment appears to the be segment between the ASA and INTERNETGW.

LAN->RTR->ASA->INTERNETGW

|

DMZ

vishwancc · ‎08-04-2009

Hi Bill,

Could you post the show process cpu on the router and do a extented ping test for LAN and WAN ip source to internet.

Chao

Vishwa

WILLIAM STEGMAN · ‎08-05-2009

Vishwa, so far I haven't seen the processor hit above 50% since Friday. I took a look at the history, and it seems to happen once every week or so. I'll have to try capture the running processes again next time it happens.

Thank you,

Bill

mbroberson1 · ‎08-05-2009

Hi Bill,

Another thing to check is your logging levels.

On the router check "sh processes cpu" and see where the culprit is.

For instance make sure you are not logging debugging or something of that nature.

HTH,

Brandon

WILLIAM STEGMAN · ‎08-06-2009

It's started again.

HBG-L3-RTR#sh proc cpu sorted 1

CPU utilization for five seconds: 38%/33%; one minute: 36%; five minutes: 38%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

31 54408 1583821 34 0.49% 0.48% 0.43% 0 Per-Second Jobs

57 3879688 3318639 1169 0.49% 0.45% 0.37% 0 IP Input

3 8376 1199 6985 2.04% 0.44% 0.30% 130 SSH Process

5 2630556 189708 13866 1.22% 0.22% 0.17% 0 Check heaps

91 84856 15470845 5 0.16% 0.16% 0.16% 0 RBSCP Background

96 71484 2320977 30 0.08% 0.08% 0.08% 0 CEF process

75 44148 6182957 7 0.08% 0.06% 0.08% 0 SSS Feature Time

48 961068 26723 35964 0.00% 0.05% 0.05% 0 Per-minute Jobs

2 49304 316769 155 0.00% 0.04% 0.05% 0 Load Meter

183 940840 105422 8924 0.00% 0.03% 0.05% 0 BGP Scanner

169 30220 3213227 9 0.08% 0.02% 0.00% 0 BGP Router

59 3004 496689 6 0.00% 0.01% 0.00% 0 NTP

101 15888 1578042 10 0.00% 0.01% 0.00% 0 RUDPV1 Main Proc

178 695456 262910 2645 0.00% 0.01% 0.00% 0 SNMP ENGINE

30 13912 1578040 8 0.00% 0.01% 0.00% 0 TTY Background

23 11192 1583815 7 0.00% 0.01% 0.00% 0 GraphIt

166 9952 1566181 6 0.00% 0.01% 0.00% 0 trunk conditioni

27 13704 349347 39 0.00% 0.01% 0.00% 0 Net Background

47 7940 316770 25 0.00% 0.00% 0.00% 0 Compute load avg

76 111028 657692 168 0.00% 0.00% 0.00% 0 TCP Timer

68 7652 1578042 4 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

vishwancc · ‎08-06-2009

Hi Bill,

High CPU is not due to process but due to

Interrupt switching.

38%/33% (33%) check the interface for input and output packet ,check this link.

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

Chao

Vishwa