CRC Errors

Unanswered Question
May 11th, 2010
User Badges:

Hello all,


I'm bringing up a new DS3 line, and i've got some questions.  Basically the circut shows up / up, and the carrier is saying there are no problems on their end of the circut.  I'm having a very hard time believing them because we've had issues all along bringing up this line, and they keep telling me the problem is not on their end, but thusfar, the problems have been at the carrier.


One of my serial interfaces has ALOT of CRC / framing problems:


Serial1/0 is up, line protocol is up
  Hardware is DSXPNM Serial
  Internet address is X.X.X.X
  MTU 4470 bytes, BW 44210 Kbit/sec, DLY 200 usec,
     reliability 200/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, crc 16, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 08:26:52
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 11
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 37000 bits/sec, 8 packets/sec
  5 minute output rate 25000 bits/sec, 7 packets/sec
     254192 packets input, 101865695 bytes, 0 no buffer
     Received 3546 broadcasts, 0 runts, 4022 giants, 0 throttles
     343934 input errors, 343934 CRC, 205200 frame, 156868 overrun, 0 ignored, 185754 abort
     249527 packets output, 89821686 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     6 carrier transitions
DSU mode 0, bandwidth 44210, real bandwidth 44210, scramble 0


The other interface looks good (I cleared the counters, but there were very few CRC errors on this side):


Serial4/0 is up, line protocol is up
  Hardware is DSXPNM Serial
  Internet address is X.X.X.X
  MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, crc 16, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:08, output 00:00:01, output hang never
  Last clearing of "show interface" counters 01:04:40
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     289 packets input, 39491 bytes, 0 no buffer
     Received 89 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     134 packets output, 12036 bytes, 0 underruns
     0 output errors, 0 collisions, 105 interface resets
     0 output buffer failures, 0 output buffers swapped out
     1 carrier transitions

Any help would be very appreciated, because now it's turning into a blame game, where they are blaming our equipment / cabling, even though it is all brand new.


Thanks


Jon

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (4 ratings)
Loading.
paolo bevilacqua Wed, 05/12/2010 - 04:10
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Please send "show controllers T3" taken on both sides.


However, when there are such debates, ask SP to bring in their own testing equipment on premises in front of your eyes, do a 48h BER test (must give 1 * 10^-9 or better), and be sure to understand how is conducted and personally observe the results.


Otherwise, do not accept circuit delivery, and no not pays the bills.

paolo bevilacqua Wed, 05/12/2010 - 05:19
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

But the point is exactly not using the router, beside such an high number of errors makes the test unnecessary.


The point is having telco to use their own equipment so they must stop blaming the router.

francisco_1 Wed, 05/12/2010 - 05:28
User Badges:
  • Gold, 750 points or more

I guess you are right. telco should take full responsiblity and use their own equipment to test and rectify the issue..


Francisco.

Jkloza_2 Wed, 05/12/2010 - 09:24
User Badges:

I'm sorry for not getting back to you folks sooner, been having conference calls with my carrier to no avail.  Basically what I've been doing is running through different types of ping tests, sending different types of data patterns & such, so far my findings have been pretty interesting.


If I try to ping through the link using data patterns of 0x0000, I get 95 - 98% loss of the pings, barely any go through.  But any other data patterns such as 0xFFFF, 0x5555, 0xaaaa, I get 100% response on the pings.  I've found a very interesting article that was written by someone having almost the exact same issue as myself, and their fix was that the alcatel mux's that some carriers use, by default, don't pass all zero's patterns, and this causes translation & linecode errors, thus producing CRC errors on the interface of the Cisco boxes.


Just an FYI, controller output is listed below.


Controller from the router I'm seeing all the CRC errors on.


T3 1/0 is up.
  Applique type is Subrate T3
  Description: DS3 Controller
  No alarms detected.
  MDL transmission is disabled

  FEAC code received: No code is being received
  Framing is C-BIT Parity, Line Code is B3ZS, Clock Source is Internal
  Data in current interval (260 seconds elapsed):
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 1:
     0 Line Code Violations, 247 P-bit Coding Violation
     5 C-bit Coding Violation, 89 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     5 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 2:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 3:
     0 Line Code Violations, 225 P-bit Coding Violation
     6 C-bit Coding Violation, 86 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     5 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 4:
     0 Line Code Violations, 146 P-bit Coding Violation
     2 C-bit Coding Violation, 57 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     2 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 5:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Total Data (last 5 15 minute intervals):
     0 Line Code Violations, 618 P-bit Coding Violation,
     13 C-bit Coding Violation, 232 P-bit Err Secs,
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs,
     0 Unavailable Secs, 0 Line Errored Secs,
     12 C-bit Errored Secs, 0 C-bit Severely Errored Secs


Controller from the router I'm not seeing any errors on:


T3 4/0 is up.
  Applique type is Subrate T3
  Description: APG - Lakehurst Controller
  No alarms detected.
  MDL transmission is disabled

  FEAC code received: No code is being received
  Framing is C-BIT Parity, Line Code is B3ZS, Clock Source is Internal
  Data in current interval (290 seconds elapsed):
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 1:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 2:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 3:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 4:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 5:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 6:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Data in Interval 7:
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Total Data (last 7 15 minute intervals):
     0 Line Code Violations, 0 P-bit Coding Violation,
     0 C-bit Coding Violation, 0 P-bit Err Secs,
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs,
     0 Unavailable Secs, 0 Line Errored Secs,
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs


Any help or opinions are always appreciated!


Thanks,


Jon

paolo bevilacqua Wed, 05/12/2010 - 09:47
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

You have clock source internal on both sides, that is never correct.


If circuit provides clock, set both sides to external.

If it doesn't, set one side only to internal.

If SP can't reliably tell if clock is provided or not (happens often), try and compare the two modes above.


Note, circuit must be clean of any C-bit and P-bit errors to be acceptable.

Clock-breaking tests come after that.

Jkloza_2 Wed, 05/12/2010 - 11:12
User Badges:

Thanks!


I've got the clocking set to line on both sides, because the carrier cannot seem to tell me if they provide clock.  The controller statistics look ok now.  Also, just fyi, it seems they may have a bad cross connect, they're dispatching someone to take a look at the line.


Also, why would ping times be so slow on a ds3 link, they aren't rate limiting, it's a point to point line, but serial interface to serial interface, this is what i'm seeing...  On my other DS3 lines it's much faster - 4/4/4.


ping
Protocol [ip]:
Target IP address: X.X.X.2
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to X.X.X.X, timeout is 2 seconds:


Success rate is 100 percent (1000/1000), round-trip min/avg/max = 16/19/28 ms


ping
Protocol [ip]:
Target IP address: X.X.X.1
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to X.X.X.1, timeout is 2 seconds:


Success rate is 100 percent (1000/1000), round-trip min/avg/max = 16/19/24 ms


Any thoughts?


Thanks!

paolo bevilacqua Wed, 05/12/2010 - 11:18
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

How long is the circuit? SP can be using some funky transport like microwave, handing off to other carriers that influence delay.


Please remember to rate useful posts clicking on the stars below.

Jkloza_2 Wed, 05/12/2010 - 11:27
User Badges:

Of course I'll rate ..  Thanks again for all of your help.  The circuit goes from NJ, to Maryland, but it bounces around a few different places beforehand.


I guess I'll attribute the slow ping times to how many hops data is actually traveling through..


Jon

paolo bevilacqua Wed, 05/12/2010 - 13:31
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Well, in theory delay in TDM circuits is not influenced by number of hops or nodes, but by overall length only.

Not even a change in transport technologu, eg circuit emaulation would explaing such an high delay.

Anyway, it should not cause you any problem.


Thanks for the rating.

Kimberly Adams Wed, 05/12/2010 - 15:06
User Badges:
  • Silver, 250 points or more

Jon,


If you are running a point-to-point circuit, then the carrier usually DOESN'T provide clocking.  At this point set the host side to internal and the remote side to line to pull from the host and any clocking errors should go away.


Good rule of thumb for clocking:  For Frame Relay and MPLS circuits the Telco usually provides the clocking on the line.  For most to all point-to-point type circuits the Telco will not provide it and the Client Premise Equipment (CPE) will need to provide it.


I thought I would throw in my two cents!


Thanks,



Kimberly

pwenstrand Wed, 05/12/2010 - 14:32
User Badges:

Try having your carrier enable scrambling and do the same on your router for that circuit.  If you are seeing these errors as bursting and not constant that might resolve it for you.

paolo bevilacqua Wed, 05/12/2010 - 14:41
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Try having your carrier enable scrambling and do the same on your router
 for that circuit.


That is a good suggestion. I don't remember if Telco does need to do anything, as it's transparent to them.

Actions

This Discussion