I'm using a MOE circuit here in Phoenix to connect several locations in a mesh configuration with my Main DC. I have one location that has 3-5% packet loss but I see no errors inbound or outbound on the interface. CEF is running and none of the other sites are dropping packets either to eachother or to the DC, However when I ping the troubled site I see the same behavior. Here are the statistics...

NOTE: Each Remote Site has a collapsed Core with a 6509 running the latest IOS and Firmware.

Remote-Site1#p
Protocol [ip]:
Target IP address: [Troubled-Site]
Repeat count [5]: 1000
Datagram size [100]: 1500
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to [Troubled-Site], timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!.!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!.!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!.!.!!!!!!!!!!!!!!.!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!.!!!.!!.!!!!!

!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!.!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!.!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!
Success rate is 96 percent (969/1000), round-trip min/avg/max = 1/3/12 ms

Troubled-Site#sh ip int GigabitEthernet1/24
GigabitEthernet1/24 is up, line protocol is up
  Internet address is [omitted]/28
  Broadcast address is 255.255.255.255
  Address determined by setup command
  MTU is 1500 bytes
  Helper address is not set
  Directed broadcast forwarding is disabled
  Multicast reserved groups joined: 224.0.0.10 224.0.0.1 224.0.0.2 224.0.0.22
      224.0.0.13
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is enabled
  ICMP redirects are always sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP fast switching on the same interface is disabled
  IP Flow switching is enabled
  IP CEF switching is enabled
  IP Flow switching turbo vector
  IP Flow CEF switching turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, Flow cache, CEF, Full Flow
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Probe proxy name replies are disabled
  Policy routing is disabled
  Network address translation is disabled
  WCCP Redirect outbound is disabled
  WCCP Redirect inbound is disabled
  WCCP Redirect exclude is disabled
  BGP Policy Mapping is disabled
  Sampled Netflow is disabled
  IP multicast multilayer switching is enabled

Troubled-Site#sh int GigabitEthernet1/24
GigabitEthernet1/24 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0017.0f97.f000 (bia 0017.0f97.f000)
  Description: QMOE CKT#14/XXXX/XXXXXXX to Central DC
  Internet address is [omitted]/28
  MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
     reliability 255/255, txload 14/255, rxload 33/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 6w4d
  Input queue: 0/75/31/31 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13244000 bits/sec, 2122 packets/sec
  5 minute output rate 5598000 bits/sec, 2011 packets/sec
  L2 Switched: ucast: 17662597 pkt, 1982838385 bytes - mcast: 24075638 pkt, 12600715488 bytes
  L3 in Switched: ucast: 7321139380 pkt, 6113001010979 bytes - mcast: 457819 pkt, 137273358 bytes mcast
  L3 out Switched: ucast: 7154731761 pkt, 2792326024317 bytes mcast: 45289477 pkt, 22070735402 bytes
     7365636806 packets input, 6128170808390 bytes, 0 no buffer
     Received 26647616 broadcasts (5918633 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     7156492485 packets output, 2774754690600 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Please Share your expertise and any insight into this. I've never seen so much packet loss with no Errors whatsoever before. That's just odd...

Sincerely,

Michael