DS3 packet loss after upgrade to NPE-G1 on 7206VXR

gregmay7 · ‎05-27-2007

Hello,

We have a 7206VXR that we've been trying to upgrade for the last two weeks to the NPE-G1 processor. After we upgrade the processor and the IOS to 12.4.11T, we experience packet loss on our DS3. The circuit provider has confirmed the packet loss. As soon as we go back to the NPE-300 and older 12.1 IOS, the packet loss problem goes away. We have copied the serial interface configuration line for line. I'm wondering if there is some command that is now a default or vice-versa. Below is the original, working serial interface config and output from sho int s4/0:

interface Serial4/0

description DS3 Frame Relay Circuit

ip address 10.250.2.1 255.255.255.0

encapsulation frame-relay

ip route-cache flow

framing c-bit

cablelength 50

dsu bandwidth 44210

serial restart-delay 0

frame-relay lmi-type ansi

end

___________________________

Serial4/0 is up, line protocol is up

Hardware is M1T-T3+ pa

Description: DS3 Frame Relay Circuit

Internet address is 10.250.2.1/24

MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,

reliability 255/255, txload 3/255, rxload 2/255

Encapsulation FRAME-RELAY, crc 16, loopback not set

Keepalive set (10 sec)

Restart-Delay is 0 secs

LMI enq sent 254, LMI stat recvd 254, LMI upd recvd 0, DTE LMI up

LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0

LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE

FR SVC disabled, LAPF state down

Broadcast queue 1/256, broadcasts sent/dropped 52506/0, interface broadcasts 48769

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 00:42:31

Input queue: 0/75/160/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue :0/40 (size/max)

5 minute input rate 497000 bits/sec, 306 packets/sec

5 minute output rate 636000 bits/sec, 287 packets/sec

1266165 packets input, 235234432 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 1 giants, 0 throttles

0 parity

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

960557 packets output, 240886449 bytes, 0 underruns

0 output errors, 0 applique, 2 interface resets

0 output buffer failures, 0 output buffers swapped out

4 carrier transitions

rxLOS inactive, rxLOF inactive, rxAIS inactive

txAIS inactive, rxRAI inactive, txRAI inactive

Below is the new config we are attempting to use with the new NPE-G1 processor:

interface Serial4/0

description DS3 Frame Relay Circuit

ip address 10.250.2.1 255.255.255.0

encapsulation frame-relay

ip route-cache flow

framing c-bit

cablelength 50

dsu bandwidth 44210

serial restart-delay 0

frame-relay lmi-type ansi

Any help here is appreciated! We are stumped!

Thanks

paolo bevilacqua · ‎05-27-2007

Hi,

Where do you see the packet loss? On input or output? How did you detected the problem ?

gregmay7 · ‎05-27-2007

Hello,

We are seeing alot of output errors. I was testing with ping after the upgrade and noticed that pings were timing out where as before we did not have that problem. It's very sporatic, sometimes we lose up to half of the pings. Also just noticed general slowness when trying to connect (telnet, rdp, etc) to one of the remote frame-relay sites.

Thanks for your help!

paolo bevilacqua · ‎05-27-2007

Hi,

You may want to seek advice from the TAC about this case. They will ask you the usual details like "show tech-support", and the service parameter (CIR basically) for all the PVCs configured. The fact that one of the remotes sites apparently has more problems than the others is interesting and should be investigated through fully. I can't make any hypothesis here as I haven't seen any of needed details.

Good luck!

pciaccio · ‎05-28-2007

The only thing I see here is Input Drops on the interface. Try increasing your queue size on the interface (Hold-queue 125 in ). This should eliminate most of them. These are caused by an over run of the input queue of the interface from the transmit of the remote router. Your issue may lie within the remote device. Check its configuration and check to see if it is getting any errors...Good Luck....

paolo bevilacqua · ‎05-28-2007

Hello pciaccio,

the show interface above is referred to the working situation with NPE-300.

We have not seen any show commands after the change to NPE-G1 and related problems.

So I'm not sure that increasing the input queue size has chances of fixing the problem.

gregmay7 · ‎05-28-2007

Hi guys,

We thought we had the problem fixed but during the day today, when we had packet loss again. It seems to only occur during high utilization periods. Below is the sho int output. You'll see that it's a HSSI now. Replacing the serial interface was part of our troubleshooting steps. Now it's showing output drops. Any ideas on why that would be occuring? Thanks!

si4/0 is up, line protocol is up

Hardware is M1T-HSSI-B

Description: DS3 Frame Relay Circuit

Internet address is

MTU 4470 bytes, BW 45000 Kbit, DLY 200 usec,

reliability 255/255, txload 3/255, rxload 4/255

Encapsulation FRAME-RELAY, crc 16, loopback not set

Keepalive set (10 sec)

Restart-Delay is 0 secs

LMI enq sent 1765, LMI stat recvd 1765, LMI upd recvd 0, DTE LMI up

LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0

LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE segmentation inactive

FR SVC disabled, LAPF state down

Broadcast queue 0/256, broadcasts sent/dropped 30372/0, interface broadcasts 4023

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 04:54:19

Input queue: 0/75/0/10 (size/max/drops/flushes); Total output drops: 6350

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 750000 bits/sec, 730 packets/sec

5 minute output rate 682000 bits/sec, 512 packets/sec

10680922 packets input, 1873226472 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 parity

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

9434954 packets output, 3146947290 bytes, 0 underruns

0 output errors, 0 applique, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions LC=down CA=up TM=down LB=down TA=up LA=down

paolo bevilacqua · ‎05-29-2007

Hi,

you have about 0.06% packet drops from the statistics above. That may very well be due to normal high utilization of the circuit. Please observe you traffic pattern with a graphical monitoring application and chances are that you will see the drop happening at times of high traffic.

Hope this helps, please rate post if it does!

pciaccio · ‎05-29-2007

Greg:

My prior response was to let you know that the information that you had originally sent does not show any issues with the output, however does show some input drops. I explained what may cause that.. However looking at your new results from the interface I now see output drops. These can be caused by high traffic times. But looking at your utilization it is at a minimum value. Possibly you are flooding the interface in bursts. These drops are caused by the FIFO register being overrun from the output traffic. You may want to increase your Hold queue for the output FIFO register. from 75 to 125 or even higher. This should eliminate your issues. If you continue to have this problem, then you may need to set LLQ and QOS to your configuration to prioritize your traffic to create certain queues for your traffic...

rwest · ‎09-26-2007

I wanted to see if you ever got your problem resolved. We experienced the same exact problem and after 2 hours with TAC we found the problem to be an MTU setting on the DS3 needed changing. On our remote site the serial interfaces all had an MTU of 1500 but our NPE-300 had 4470. After replacing it with the NPE-G1 the TAC guy changed the MTU from 4470 to 1500 and all our frame relay sites came back up. We dont know why this worked so I was just curious on how you fixed your problem.

gregmay7 · ‎09-27-2007

We "resolved" this by turning off eigrp for all WAN connections and reverting back to statics. The packet loss was stemming from the eigrp neighbors bouncing up and down constantly, a problem that we could not solve. Unfortunately we had to put this problem on the back-burner since we had a workaround. I'm wondering now if the MTU setting will help solve the eigrp issue. I will be testing that in our lab now! Thanks!

paolo bevilacqua · ‎09-28-2007

Then try a most normalized routing protocol. ospf is known to behave nicely in most cases.

rwest · ‎09-28-2007

The TAC engineer added static routes and like you worked as a workaround but we did not want it that way and wanted it to learn it's route via eigrp so thats why we kept pressing the issue. After he changed the MTU and it worked this was his reasoning:

MTU is used by eigrp as part of the computation to do neighbor adjacency with other eigrp routers. If there are 2 routers with different MTU sizes, tendency of the other router with lower MTU size will drop the packet. For example, router A(remote site) has MTU of 1500 and router B(7206VXR NPE-G1) has MTU of 4470. If router B sends packet to router A, router A will drop the packet since it can only take 1500 bytes. Different modules have also different MTU sizes.

I hope this may shed some light on the problem but so far all our frame sites have not had any issues since this change.