cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
593
Views
9
Helpful
10
Replies

Needing advice on how to troubleshoot this

John Blakley
VIP Alumni
VIP Alumni

All,

I have a location that has a 2800 series router, packet shaper, l3 switch (dell), 1231bg APs, and several l2 switches (dell) spread over 2 buildings.

I noticed over the last week that if I ping the inside interface of the router from my side (public -> private), my access times can come back as high as 200ms. The serial interface on the router stays consistent within 12ms-70ms. When I ping any of the switches behind the router, I've seen latency as high as 1200ms.

It's not a broadcast storm, at least according to all of the switch statistics. Broadcast packets are incrementing at a rate of about 15-30 packets every 15 sec. I would think it would be much more if it was a storm.

CPU and memory is fine on the router, and this was my first thought as to why the traffic would jump from a 70ms public to a 200ms private address within the same router. I'm honestly at a loss.

I can put a sniffer on the network, but I'll only catch broadcasts. I could try to mirror the port on the dell that connects to the router and see what's going through that port, but I'm not sure if this would be necessary. Does this situation call for netflow running on the inside interface?

Thanks,

John

HTH, John *** Please rate all useful posts ***
1 Accepted Solution

Accepted Solutions

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

View solution in original post

10 Replies 10

Edison Ortiz
Hall of Fame
Hall of Fame

If you telnet to that remote router and ping back to your network while sourcing from the LAN interface, how bad is the latency?

How about performing a traceroute from your network towards that site? Which hop does the latency increase?

__

Edison.

Edison,

The following is from the router to the core switch that's directly connected to it:

H#ping 10.125.3.5 sour 10.125.3.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.125.3.5, timeout is 2 seconds:

Packet sent with a source address of 10.125.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 108/131/144 ms

The next one is from the remote side to my side:

H#ping 10.125.100.1 sour 10.125.3.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.125.100.1, timeout is 2 seconds:

Packet sent with a source address of 10.125.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 12/13/16 ms

Here's a trace from my workstation to the core switch on the "problem" side:

Tracing route to 10.125.3.5 over a maximum of 30 hops

1 7 ms 4 ms 5 ms 3750.my.side [10.125.100.5

2 12 ms 3 ms 3 ms it-rt1.my.side [10.125.100.1]

3 9 ms 11 ms 9 ms 172.16.100.2

4 45 ms 43 ms 31 ms 172.20.3.1 <- The "problem" side serial

5 151 ms 225 ms 180 ms 10.125.3.5 <- The "problem" side core switch (dell)

One thing that we noticed on Friday, and I can't attest that this is still happening, is that we have two serial wics in their 2800 router. One comes to us, and the other is on a p2p link across the street. If we shut the interface down that goes to the location across the street, it seems to clear up the latency everywhere. I don't show anything wrong with my routing table though.

Thanks Edison!

John

HTH, John *** Please rate all useful posts ***

125ms avg to a directly connected LAN interface?

That does not seem right.

I recommend checking the LAN cables and interfaces at both end of the links.

Perhaps bad cable or duplex mismatch is affecting this connection.

The ping from the remote WAN to your network is 14ms avg, that's pretty good and I don't think the WAN configuration has anything to do with it.

__

Edison.

Yeah, it's horrible. Here's the output on the interface that connects directly to the core switch. I don't have an CRC errors, and overall the connection seems fine:

FastEthernet0/1 is up, line protocol is up

Hardware is MV96340 Ethernet, address is 0023.5ece.d951 (bia 0023.5ece.d951)

Internet address is 10.126.3.1/24

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 3/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/38/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 288000 bits/sec, 172 packets/sec

5 minute output rate 1493000 bits/sec, 226 packets/sec

19829991 packets input, 3800011820 bytes

Received 794523 broadcasts, 0 runts, 0 giants, 1 throttles

10 input errors, 0 CRC, 0 frame, 0 overrun, 10 ignored

0 watchdog

0 input packets with dribble condition detected

20782931 packets output, 1649380649 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

3 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

The 10.126.x.x address that you see is the primary address on the router interface. The trace was sourced from the secondary address.

Thanks!

John

HTH, John *** Please rate all useful posts ***

Do you have anything on the LAN side within the 10.126.x.x that you can ping from this router?

What's the latency then?

__

Edison.

Here are two hosts:

H#ping 10.126.3.186 sour fa0/1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.126.3.186, timeout is 2 seconds:

Packet sent with a source address of 10.126.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 268/285/296 ms

H#ping 10.126.3.149 sour fa0/1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.126.3.149, timeout is 2 seconds:

Packet sent with a source address of 10.126.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 420/516/600 ms

Here's my routing table:

H#sh ip route 10.126.3.0

Routing entry for 10.126.3.0/24

Known via "connected", distance 0, metric 0 (connected, via interface)

Advertised by bgp 65003

Routing Descriptor Blocks:

* directly connected, via FastEthernet0/1

Route metric is 0, traffic share count is 1

Thanks,

John

HTH, John *** Please rate all useful posts ***

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

Edison,

I've been working on this all day, and I've figured it out.

I used your suggestion of pinging from the switch to a host. I also pinged the same host from the router. The difference: Router 200ms vs Switch < 1ms.

The only difference between the two was that the router was going through the packet shaper. I turned shaping off, and that fixed the problem. I then looked in the shaper's table, and I noticed two hosts were sending 1.6m/sec of data between themselves. Pinging with a -a showed one of the hosts, but I couldn't find the other host. I ran a portscan with nmap, and I found 80 and 23 to be open. I connected to it with my web browser, and it was a camera system.

The other host happened to be the warehouse manager, and he had a connection open to this camera. I had him close it, I turned shaping back on, and everything was nice and quick. He then connected to another set of cameras, and there were no problems with that one. I'm having them contact the camera company and let them know that it's sending a ton of data out when connecting to it.

Thanks for all of your help!

John

HTH, John *** Please rate all useful posts ***

Edison Ortiz
Hall of Fame
Hall of Fame

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

Edison Ortiz
Hall of Fame
Hall of Fame

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco