Packet loss on many remote location

[Sandeep Choudhary]

Hello Everyone,

I am facing this issue from long time and still coudn't find the core issue.

specially we are using CAD servers and some applications(e.g. outlook...email...ftp) on the remote location

Every remote location is connected via gre tunnels and dial back technic from HQ.

can any body give me some suggestion to try something on remote router or HQ router to eliminate these problems.

Regards

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

If the tunnels provide less bandwidth than the LAN, it's often quite normal to see drops.  If traffic is TCP, some special purpose traffic shaping appliances might avoid all drops.  On "normal" routers/switches, best you might accomplish is reducing the number of drops.

[Sandeep Choudhary]

Hi Joseph,

We have 100 Mbps line at the HQ and 2 tunnels to each remote location.

And maximum LAN Bandwidth at remote location is 10Mbps.

so it can not be a issue about less bandwidth of tunnels.

Regards

[bbaillie]

Hi,

You may have already covered this but, you mention CAD, Outlook and FTP, all TCP based items, and larger packets will get dropped when the MTU is too large for the full path to support it.

You may want to verify the MTU from the remote site, through the GRE, through the provider network. The MTU may be smaller than you expect. Test this with ping using large packets and don't fragment set "ping -l 1472 -f 10.10.10.5" The IP address substitute yours. The 1472 is the larges IP packet allowed on standard ethernet, anything larger and it should reject it saying can't fragment. Keep moving lower until a few packet fly through just fine.

You can set the MTU at the the router or also simply have the router adjust the MSS to a smaller value. The MSS is smaller than the MTU by 20 Bytes.

Here is more information on the subject

http://www.cisco.com/en/US/docs/ios/12_2t/12_2t4/feature/guide/ft_admss.html

Cheers,

Brian

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

We have 100 Mbps line at the HQ and 2 tunnels to each remote location.
And maximum LAN Bandwidth at remote location is 10Mbps.
so it can not be a issue about less bandwidth of tunnels.

Ah, good then, I'm glad it cannot be an issue.

[paolo bevilacqua]

Sandeep Choudhary wrote:
Hi Joseph,
We have 100 Mbps line at the HQ and 2 tunnels to each remote location.
And maximum LAN Bandwidth at remote location is 10Mbps.
so it can not be a issue about less bandwidth of tunnels.
Regards

It is for sure a bandwidth problem.

proof:

it happens only during busy hours.

your ping time are widely variable, indicating link congestion.

you are not looking at physical itnerface or traffic graphs, as you should.

[Leo Laohoo]

can any body give me some suggestion to try something on remote router or HQ router to eliminate these problems.

Do you see these drops during or after office hours?  Or is this happening 24-hours a day?

In your routers, can you please post the output to the following commands:

1.  sh interface ; and

2.  sh interface

[Sandeep Choudhary]

Hi leolaohoo,

This is happening in official hours..e.g. 8-5pm.

1.

RemCVPN1#sh int fa0/0-------1st provider

FastEthernet0/0 is up, line protocol is up

  Hardware is MV96340 Ethernet, address is 0024.974c.3708 (bia 0024.974c.3708)

  Description: *** Site to Site NET (Internet) ISP1 ***

  Internet address is 195.243.193.34/29

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:02, output 00:00:00, output hang never

  Last clearing of "show interface" counters 20:52:36

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 43000 bits/sec, 16 packets/sec

  5 minute output rate 203000 bits/sec, 109 packets/sec

     3793671 packets input, 2194054105 bytes

     Received 11765 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog

     0 input packets with dribble condition detected

     13647096 packets output, 3450052086 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

RemCVPN1#sh int fa0/1-------------2nd provider

FastEthernet0/1 is up, line protocol is up

  Hardware is MV96340 Ethernet, address is 0024.974c.3709 (bia 0024.974c.3709)

  Description: *** Site to Site NET (Internet) ISP2 ***

  Internet address is 195.243.201.186/29

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:09, output 00:00:00, output hang never

  Last clearing of "show interface" counters 20:54:07

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 118000 bits/sec, 61 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     8258789 packets input, 3277206728 bytes

     Received 10191 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog

     0 input packets with dribble condition detected

     266137 packets output, 98790589 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

This is the output from the remote vpn router(where 2 provider connected to this router ),

I am attaching the logical structure which we have on almost all remote location.(On the first pag i added the visio)

if you need more output then please let me know.

Regards

[Sandeep Choudhary]

Any advice guys??

Regards

[nkarpysh]

I have few questions to understand it in more details first:

- How do you know that packets are dropped? Any evidence of that to look at?

- Do you see slow response of applications? Was it working before or it is a Day1 issue?

- Did you try to test with a ping of different size between servers and end clients - do you see any drops?

I guess we should start with clarification of above and then looking closely into the devices.

Nik

[Sandeep Choudhary]

Hi Nikolay,

1:

When I ping from HQ to remote vpn router  is hows 20% request timeouts, and usually delay is 35ms but it shows 300-400ms.

2.

My other team member from CAD team, he is compaling about packet loss while CAD server data transfer.

3.

No i did not try.

BUt my concern is ..from where i can start my troubleshooting becase.....when i ping from my pc(HQ) to remote router it is also very slow.

If you need furtherinfo then please let me know.

[Lee Smitherman]

Hi,

how many remote sites are we talking about?  Have you checked the utilisation of the CPU on the central router during these issues?

Lee.

[Sandeep Choudhary]

there are 2-3 remote sites......

the cpu usuage of the HQ router is around 15-16%

HQCVPN2#sh process cpu

CPU utilization for five seconds: 16%/14%; one minute: 17%; five minutes: 17%

too much request timeout ....

Normal delay shoud be 30-40ms but ping shows 250-300ms.

Regards

[handoko wiyanto]

hi sandeep,

what type of connection are you using? delivering 100mbs via satelite will take arround that, and some of them are higher.

regards,

[Sandeep Choudhary]

not via satelite ..we are using gre tunnel from HQ to every remote location ....and every remote location have 1 or 2 own provider .

I don understand why ..suddenly ..lots of request timeout while transfering dats from HQ to remote or vice versa.

Regards