Needing advice on how to troubleshoot this

Answered Question
Mar 3rd, 2009

All,

I have a location that has a 2800 series router, packet shaper, l3 switch (dell), 1231bg APs, and several l2 switches (dell) spread over 2 buildings.

I noticed over the last week that if I ping the inside interface of the router from my side (public -> private), my access times can come back as high as 200ms. The serial interface on the router stays consistent within 12ms-70ms. When I ping any of the switches behind the router, I've seen latency as high as 1200ms.

It's not a broadcast storm, at least according to all of the switch statistics. Broadcast packets are incrementing at a rate of about 15-30 packets every 15 sec. I would think it would be much more if it was a storm.

CPU and memory is fine on the router, and this was my first thought as to why the traffic would jump from a 70ms public to a 200ms private address within the same router. I'm honestly at a loss.

I can put a sniffer on the network, but I'll only catch broadcasts. I could try to mirror the port on the dell that connects to the router and see what's going through that port, but I'm not sure if this would be necessary. Does this situation call for netflow running on the inside interface?

Thanks,

John

I have this problem too.
0 votes
Correct Answer by Edison Ortiz about 7 years 9 months ago

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.7 (3 ratings)
Loading.
Correct Answer
Edison Ortiz Tue, 03/03/2009 - 08:47

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

Correct Answer
Edison Ortiz Tue, 03/03/2009 - 08:47

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

Edison Ortiz Tue, 03/03/2009 - 07:56

If you telnet to that remote router and ping back to your network while sourcing from the LAN interface, how bad is the latency?

How about performing a traceroute from your network towards that site? Which hop does the latency increase?

__

Edison.

John Blakley Tue, 03/03/2009 - 08:10

Edison,

The following is from the router to the core switch that's directly connected to it:

H#ping 10.125.3.5 sour 10.125.3.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.125.3.5, timeout is 2 seconds:

Packet sent with a source address of 10.125.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 108/131/144 ms

The next one is from the remote side to my side:

H#ping 10.125.100.1 sour 10.125.3.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.125.100.1, timeout is 2 seconds:

Packet sent with a source address of 10.125.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 12/13/16 ms

Here's a trace from my workstation to the core switch on the "problem" side:

Tracing route to 10.125.3.5 over a maximum of 30 hops

1 7 ms 4 ms 5 ms 3750.my.side [10.125.100.5

2 12 ms 3 ms 3 ms it-rt1.my.side [10.125.100.1]

3 9 ms 11 ms 9 ms 172.16.100.2

4 45 ms 43 ms 31 ms 172.20.3.1 <- The "problem" side serial

5 151 ms 225 ms 180 ms 10.125.3.5 <- The "problem" side core switch (dell)

One thing that we noticed on Friday, and I can't attest that this is still happening, is that we have two serial wics in their 2800 router. One comes to us, and the other is on a p2p link across the street. If we shut the interface down that goes to the location across the street, it seems to clear up the latency everywhere. I don't show anything wrong with my routing table though.

Thanks Edison!

John

Edison Ortiz Tue, 03/03/2009 - 08:21

125ms avg to a directly connected LAN interface?

That does not seem right.

I recommend checking the LAN cables and interfaces at both end of the links.

Perhaps bad cable or duplex mismatch is affecting this connection.

The ping from the remote WAN to your network is 14ms avg, that's pretty good and I don't think the WAN configuration has anything to do with it.

__

Edison.

John Blakley Tue, 03/03/2009 - 08:24

Yeah, it's horrible. Here's the output on the interface that connects directly to the core switch. I don't have an CRC errors, and overall the connection seems fine:

FastEthernet0/1 is up, line protocol is up

Hardware is MV96340 Ethernet, address is 0023.5ece.d951 (bia 0023.5ece.d951)

Internet address is 10.126.3.1/24

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 3/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/38/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 288000 bits/sec, 172 packets/sec

5 minute output rate 1493000 bits/sec, 226 packets/sec

19829991 packets input, 3800011820 bytes

Received 794523 broadcasts, 0 runts, 0 giants, 1 throttles

10 input errors, 0 CRC, 0 frame, 0 overrun, 10 ignored

0 watchdog

0 input packets with dribble condition detected

20782931 packets output, 1649380649 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

3 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

The 10.126.x.x address that you see is the primary address on the router interface. The trace was sourced from the secondary address.

Thanks!

John

Edison Ortiz Tue, 03/03/2009 - 08:31

Do you have anything on the LAN side within the 10.126.x.x that you can ping from this router?

What's the latency then?

__

Edison.

John Blakley Tue, 03/03/2009 - 08:36

Here are two hosts:

H#ping 10.126.3.186 sour fa0/1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.126.3.186, timeout is 2 seconds:

Packet sent with a source address of 10.126.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 268/285/296 ms

H#ping 10.126.3.149 sour fa0/1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.126.3.149, timeout is 2 seconds:

Packet sent with a source address of 10.126.3.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 420/516/600 ms

Here's my routing table:

H#sh ip route 10.126.3.0

Routing entry for 10.126.3.0/24

Known via "connected", distance 0, metric 0 (connected, via interface)

Advertised by bgp 65003

Routing Descriptor Blocks:

* directly connected, via FastEthernet0/1

Route metric is 0, traffic share count is 1

Thanks,

John

Correct Answer
Edison Ortiz Tue, 03/03/2009 - 08:47

How about pinging from workstation <-> workstation within that LAN?

How about pinging from that Dell switch to those workstations?

__

Edison.

John Blakley Tue, 03/03/2009 - 11:19

Edison,

I've been working on this all day, and I've figured it out.

I used your suggestion of pinging from the switch to a host. I also pinged the same host from the router. The difference: Router 200ms vs Switch < 1ms.

The only difference between the two was that the router was going through the packet shaper. I turned shaping off, and that fixed the problem. I then looked in the shaper's table, and I noticed two hosts were sending 1.6m/sec of data between themselves. Pinging with a -a showed one of the hosts, but I couldn't find the other host. I ran a portscan with nmap, and I found 80 and 23 to be open. I connected to it with my web browser, and it was a camera system.

The other host happened to be the warehouse manager, and he had a connection open to this camera. I had him close it, I turned shaping back on, and everything was nice and quick. He then connected to another set of cameras, and there were no problems with that one. I'm having them contact the camera company and let them know that it's sending a ton of data out when connecting to it.

Thanks for all of your help!

John

Actions

This Discussion