Intermittent latency issues with 3560 as router

allenferdinand · ‎01-11-2012

I have a small network with the following configuration:

AT&T 100 MB circuit terminates in their on site switch (I have no visibility)

AT&T switch handoff via ethernet to my 3560 switch

Two firewalls and a soho router connected to the switch on VLAN2

Switch:

gig 0/1 port speed 100, full duplex, routing enabled. My WAN IP is assigned to this port. This port goes to AT&T

VLAN 2 configured as 100, Full, LAN IP is assigned to this port

gig 0/2 port speed 100, full duplex goes to soho router for child company that can't touch my network

gig 0/3 port speed 100, full duplex goes to a 5505 which is used for guest wireless access

gig 0/4 auto/auto (1 GB full duplex) goes to my 5520 which is has internal and DMZ subnets on it

5520 Ethernet 0/0 is set for auto/auto (Full duplex, 1 Gbps)

I am troubleshooting ping spikes. My monitoring system pings the next hop router from AT&T every minute. I am seeing anywhere from 3 ms to 57 ms, which the boss finds unacceptable. If I ping Google DNS, i get a constant 5-7 ms and no loss.

Pinging from 12.107.176.1 to 12.248.81.94, i can see pings as high as 27 ms and if I ping to 12.248.81.93, they can be as high as 75 ms. I'm looking for explanation/cause/whatever help I can get. I've had Cisco bless my config when troubleshooting another issue (AT&T static routes...) and they say it is fine.

Thanks in advance,

Allen

itru-rtr-ext#sh runn
Building configuration...

Current configuration : 4137 bytes
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname rtr-ext
!
boot-start-marker
boot-end-marker
!

!
!
no aaa new-model
system mtu routing 1500
ip routing
!
!
!
!
crypto pki trustpoint HTTPS_SS_CERT_KEYPAIR
enrollment selfsigned
serial-number
revocation-check none
rsakeypair HTTPS_SS_CERT_KEYPAIR
!
!

spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
!
interface FastEthernet0
no ip address
no ip route-cache cef
no ip route-cache
no ip mroute-cache
shutdown
!
interface GigabitEthernet0/1
no switchport
ip address 12.248.81.94 255.255.255.252
speed 100
duplex full
!
interface GigabitEthernet0/2
switchport access vlan 2
!
interface GigabitEthernet0/3
switchport access vlan 2
switchport mode access
speed 100
duplex full
!
interface GigabitEthernet0/4
switchport access vlan 2
switchport mode access
!
interface GigabitEthernet0/5
switchport access vlan 2
switchport mode access
!
interface GigabitEthernet0/6
!
interface GigabitEthernet0/7
!
interface GigabitEthernet0/8
!
interface GigabitEthernet0/9
!
interface GigabitEthernet0/10
!
interface GigabitEthernet0/11
!
interface GigabitEthernet0/12
!
interface GigabitEthernet0/13
!
interface GigabitEthernet0/14
!
interface GigabitEthernet0/15
!
interface GigabitEthernet0/16
!
interface GigabitEthernet0/17
!
interface GigabitEthernet0/18
!
interface GigabitEthernet0/19
!
interface GigabitEthernet0/20
!
interface GigabitEthernet0/21
!
interface GigabitEthernet0/22
!
interface GigabitEthernet0/23
!
interface GigabitEthernet0/24
!
interface GigabitEthernet1/1
!
interface GigabitEthernet1/2
!
interface GigabitEthernet1/3
!
interface GigabitEthernet1/4
!
interface TenGigabitEthernet1/1
!
interface TenGigabitEthernet1/2
!
interface Vlan1
no ip address
!
interface Vlan2
ip address 12.107.176.1 255.255.255.0 secondary
ip address 12.41.96.1 255.255.255.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 12.248.81.93
!
no ip http server
no ip http secure-server
!
ip sla enable reaction-alerts
no cdp advertise-v2
!
!
!
line con 0
login local

!
end

roberto.gagliardi · ‎01-20-2012

Hi,

Send the show interfaces command output.

Thank You

sleepyshark · ‎01-20-2012

Do keep in mind that ICMP is best effort protocol and shouldn't be used as a primary form of quality of communications.

Additionally, I have seen first hand this EXACT same issue with older Cisco equipment... Essentially, the switch's interface ping response times are sluggish, but traffic past it is just fine and not affected. I am not sure what causes this, but is probably either CPU or just the fact that ICMP is a low priority protocol.

If your ISP is seeing no errors and no quality/latency/packet loss issues, you should be fine.

If you want to troubleshoot - to see if it's the swtich, replace your 3560 with a newer switch and re-try your tests.

Thanks,

Sean Brown

http://www.sleepyshark.com

(please rate this post if useful)

allenferdinand · ‎01-20-2012

I'll pull the show interface stats once I get over this cold. I was going to swap out the 3560 (almost brand new), but I don't have any other layer 3 devices. I have an old router I can swap in, but I cant do it until mid Feb when we have our next maintenance window. I finally found that device after a move between buildings. AT&T has come back and said they find nothing wrong. I do have an open ticket with Cisco on it, but am out sick.

Thanks for looking guys, i'll update Monday.