11-02-2008 12:45 PM - edited 03-04-2019 12:09 AM
We have a head office and 2 branch offices. Router1 in head office got 2 separate DS3 cards and got two DS3 links to two remote sites Router2 and Router3, each one on one DS3 card. Router1 DS3 card which is connected to Router2 gone faulty recently and we have changed it with new one. But since then link was taking heavy CRC errors at only Router1 side and Router2 is no errors.
Somany times has had troubleshooting session with telco, but no problem found when they give loop each side at a time. Changed cable between Router1 and demarc point and also changed Link from one DS3 card to second port of another DS3 card on Router1 (as link is taking errors at Router1 side only).
We did not change any thing at Router side.
Here below are the show controller, show run config of the interface & show interface commands output of both Router1 and Router2.
Pls help, as this issue is running more than 10 days. Thanks in advance.
11-02-2008 12:46 PM
Config on Router1 Se5/1
interface Serial5/1
dampening
bandwidth 44736
ip address xxx.xxx.xxx
ip ospf cost 200
mpls ip
dsu bandwidth 44210
framing c-bit
cablelength 25
clock source internal
serial restart-delay 0
service-policy output QoS-Out
end
Router1#sh contro se5/1
M2T-T3+ pa: show controller:
PAS unit 2, subunit 1, f/w version 3-93, rev ID 0x2800001, version 3
idb = 0x6404B414, ds = 0x6404C5E4, ssb=0x6404C9A0
Clock mux=0x30, ucmd_ctrl=0x0, port_status=0x1
Serial config=0x8, line config=0x1B0202
maxdgram=4584, bufpool=128Kb, 256 particles
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
line state: up
cable type : T3 cable, received clockrate 44201430
base0 registers=0x3E820000, base1 registers=0x3E822000
mxt_ds=0x65396EE8, rx ring entries=124, tx ring entries=254
statring=0xE580260, statr shadow=0x6404F084, stat_head=70
rxring=0xE580020, rxr shadow=0x6404EA50, rx_head=56
txring=0xE5808A0, txr shadow=0x6404FCB8, tx_head=141, tx_tail=141, tx_count=0
throttled=0, enabled=0
halted=0, last halt reason=0
Microcode fatal errors=0
rx_no_eop_err=0, rx_no_stp_err=0, rx_no_eop_stp_err=0
rx_no_buf=0, rx_soft_overrun_err=1729, dump_err= 0, bogus=0, mxt_flags=0x2C
tx_underrun_err=0, tx_soft_underrun_err=0, tx_limited=1(64)
tx_fullring=11664, tx_started=18230392, mxt_flush_count=0
rx_int_count=107101046, tx_int_count=28496935
Framing is c-bit, Clock Source is Internal
Bandwidth limit is 44210, DSU mode 0, Cable length is 25
rx FEBE since last clear counter 0, since reset -597803264
Data in current interval (420 seconds elapsed):
0 Line Code Violations, 66 P-bit Coding Violation
66 C-bit Coding Violation
3 P-bit Err Secs, 1 P-bit Sev Err Secs
0 Sev Err Framing Secs, 0 Unavailable Secs
3 Line Errored Secs, 3 C-bit Errored Secs, 1 C-bit Sev Err Secs
No alarms detected.
Last FEAC code received: activate/deactivate DS3 line loopback
11-02-2008 12:48 PM
Router1#sh int se5/1
Serial5/1 is up, line protocol is up
Hardware is M2T-T3+ pa
Internet address is xxx.xxx.xxx.xxx
MTU 4470 bytes, BW 44736 Kbit, DLY 200 usec,
reliability 255/255, txload 24/255, rxload 24/255
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Restart-Delay is 0 secs
Last input 00:00:00, output 00:00:01, output hang never
Last clearing of "show interface" counters 00:08:23
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/2/256 (active/max active/max total)
Reserved Conversations 2/2 (allocated/max allocated)
Available Bandwidth 11184 kilobits/sec
5 minute input rate 4380000 bits/sec, 154 packets/sec
5 minute output rate 4381000 bits/sec, 154 packets/sec
100459 packets input, 362506326 bytes, 0 no buffer
Received 59 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
2358 input errors, 2358 CRC, 0 frame, 0 overrun, 5 ignored, 0 abort
100762 packets output, 362535096 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
11-02-2008 12:48 PM
Config on Router2 Se3/0
interface Serial3/0
dampening
bandwidth 44736
ip address xxx.xxx.xxx.xxx
ip ospf cost 200
load-interval 30
mpls ip
dsu bandwidth 44210
framing c-bit
cablelength 25
serial restart-delay 0
service-policy output QoS-Out
end
Router2#sh contro se3/0
M1T-T3+ pa: show controller:
PAS unit 0, subunit 0, f/w version 3-93, rev ID 0x2800001, version 3
idb = 0x63936E58, ds = 0x63938028, ssb=0x639383E4
Clock mux=0x30, ucmd_ctrl=0x0, port_status=0x1
Serial config=0x8, line config=0x1B0202
maxdgram=4584, bufpool=128Kb, 256 particles
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
line state: up
cable type : T3 cable, received clockrate 44202795
base0 registers=0x3D800000, base1 registers=0x3D802000
mxt_ds=0x6513BC90, rx ring entries=124, tx ring entries=254
statring=0xE32CE60, statr shadow=0x6393B3C0, stat_head=147
rxring=0xE32CC20, rxr shadow=0x6393AD8C, rx_head=60
txring=0xE32D4A0, txr shadow=0x6393BFF4, tx_head=171, tx_tail=171, tx_count=0
throttled=0, enabled=0
halted=0, last halt reason=0
Microcode fatal errors=0
rx_no_eop_err=0, rx_no_stp_err=0, rx_no_eop_stp_err=0
rx_no_buf=0, rx_soft_overrun_err=30962, dump_err= 0, bogus=0, mxt_flags=0x2C
tx_underrun_err=0, tx_soft_underrun_err=0, tx_limited=1(64)
tx_fullring=6463679, tx_started=1296076828, mxt_flush_count=0
rx_int_count=-819775045, tx_int_count=1689927652
Framing is c-bit, Clock Source is Line
Bandwidth limit is 44210, DSU mode 0, Cable length is 25
rx FEBE since last clear counter 140, since reset 65649022
Data in current interval (423 seconds elapsed):
0 Line Code Violations, 0 P-bit Coding Violation
0 C-bit Coding Violation
0 P-bit Err Secs, 0 P-bit Sev Err Secs
0 Sev Err Framing Secs, 0 Unavailable Secs
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
No alarms detected.
Router2#sh int se3/0
Serial3/0 is up, line protocol is up
Hardware is M1T-T3+ pa
Internet address is xxx.xxx.xxx.xxx
MTU 4470 bytes, BW 44736 Kbit, DLY 200 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Restart-Delay is 0 secs
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:08:56
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/93/256 (active/max active/max total)
Reserved Conversations 2/2 (allocated/max allocated)
Available Bandwidth 11184 kilobits/sec
30 second input rate 151000 bits/sec, 4 packets/sec
30 second output rate 151000 bits/sec, 3 packets/sec
100796 packets input, 362538624 bytes, 0 no buffer
Received 62 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
100500 packets output, 362528197 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
11-03-2008 01:25 AM
Hello Jaleelahmed,
>>100459 packets input, 362506326 bytes, 0 no buffer
Received 59 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
2358 input errors, 2358 CRC, 0 frame, 0 overrun, 5 ignored, 0 abort
100762 packets output, 362535096 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
There is still some problem in the local side of Router1
Hope to help
Giuseppe
11-03-2008 04:59 AM
Hi Giuseppe,
Thanks for your prompt reply.
But, while troubleshooting serveral times with local telco at Router1 side, we changed all the cables between router & demarc point and also changed link on to different port.
Telco also tested by giving hard loop from different segments towards each side of the Router1 and Router2 throught the path and link was clean every time tested.
The only thing we found that, when we give 'loopback remote' on Router1 interface we can ping to the self-interface, but heavy CRC errors. But at the same time, when we give 'loopback remote' on Router2 interface and ping self interface, we see heavy packet drops and CRC errors.
Hope this helps to understand the problem.
Telco denies to accept this as their problem and we have tested with all our possibilties, but still no go.
11-03-2008 11:50 AM
Hello Jaleelahmed,
examining again the show controller serial on the two routers I see that you have set clock internal on router R1 and clock line.
This probably is a legitimate configuration I wonder if the clock could be provided by the service provider on both ends.
Another thought is that when T3 is carried over SDH actually it is carried in the payload of an STM-1.
I would suggest to try to use
clock line also on R1 if the provider hasn't given this instruction to you.
Some years ago for a mobile operator that was using T3 links that were carried inside SDH hierarchy we made some tests because there were some errors from time to time.
The analysis gave the results that every time SDH was rerouting we got errors on the T3.
I've checked the router's configurations and we had clock source line on both ends of the same link.
Hope to help
Giuseppe
11-04-2008 02:57 AM
i had similar problem some time back and it was due to clock source. I had to setup clock source line on both side and all worked fine.
As you have isolated physical components i guess clocking is the only thing you might have to look in to.
Regards
Abid
11-04-2008 05:41 AM
Thanks Abid & Giuseppe for your postings.
Well, I've enquired with telco several times on clocking issue. Provider always says, they don't provide any clock at either side. So I configured 'clock source internal' at Router1 side and 'clock source line' at other side. But still no help.
Yesterday, we have had a very long tele-conferene with all providers (4 providers are involved in this link) and provider(s) changed their MUX Port, LIU patchpanel and cables at demarc point. But still issue persists.
The only new thing we got now that, provider was testing the link with some testgear equipment at ISP end nearer to the Router2 and he could see some errors, when we gave software loop on Router2 interface.
So, again today I'm going to have conf call with them all. Let us see, may be it could be resolved today.
If any one have more ideas, certainly are welcome.
Thanks
Jaleel Ahmed
11-10-2008 03:39 AM
Hi,
Now issue solved. Router2 end provider's equipment problem. Strange thing is that, we see errors at Router1 always, but problem is identified at Router2 end.
Thnx for the postings.
Rgds,
Jaleel Ahmed
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide