cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3155
Views
0
Helpful
16
Replies

Bonded T1 not working

dtom
Level 1
Level 1

We are running two Cisco 3845 routers, one at our central office and one at a remote site.  We bonded two T1's together and it worked fine (link 1 and link 2).  We now want to add two more T1's and bond the four together but can't seem to get it to work.  The original T1's still work fine (link 1 and link 2) but the new ones do not (link 3 and link 4).  Here is our config does it look right?


***CENTRAL OFFICE***

!
card type t1 0 0
card type t1 0 1
card type t1 0 2
card type t1 0 3
card type t1 2 0
card type t1 2 1
!
network-clock-participate slot 2
network-clock-participate wic 0
network-clock-participate wic 1
network-clock-participate wic 2
network-clock-participate wic 3
network-clock-select 1 T1 0/2/0
network-clock-select 2 T1 0/2/1
network-clock-select 3 T1 0/3/0
network-clock-select 4 T1 0/0/0
!
controller T1 0/1/0
clock source internal
cablelength long 0db
channel-group 1 timeslots 1-24
!
controller T1 0/1/1
clock source internal
cablelength long 0db
channel-group 1 timeslots 1-24
!
controller T1 2/0
!
controller T1 2/0/0
clock source internal
cablelength long 0db
channel-group 1 timeslots 1-24
!
controller T1 2/0/1
clock source internal
cablelength long 0db
channel-group 1 timeslots 1-24
!
interface Multilink1
description Multilink Bundle
ip address 172.19.254.102 255.255.255.252
ppp multilink
ppp multilink group 1
service-policy output LAN-EDGE
!
interface Serial0/1/0:1
  description T1 link 1
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/1/1:1
description T1 link 2
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial2/0/0:1
description T1 link 3
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial2/0/1:1
description T1 link 4
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

sh ppp multilink

Multilink1
  Bundle name: er-VANGPAO-TEMP
  Remote Endpoint Discriminator: [1] er-VANGPAO-TEMP
  Local Endpoint Discriminator: [1] vgectr01
  Bundle up for 3d23h, total bandwidth 3072, load 16/255
  Receive buffer limit 24000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    153 lost fragments, 5819169 reordered
    142/39549 discarded fragments/bytes, 0 lost received
    0xB06B82 received sequence, 0xD182AD sent sequence
  Member links: 2 active, 2 inactive (max not set, min not set)
    Se0/1/0:1, since 3d23h
    Se0/1/1:1, since 3d23h
    Se2/0/0:1 (inactive)
    Se2/0/1:1 (inactive)
No inactive multilink interfaces


***REMOTE SITE***

!
card type t1 0 0
card type t1 0 1
!
network-clock-participate wic 0
network-clock-participate wic 1
!
controller T1 0/0/0
framing esf
linecode b8zs
channel-group 0 timeslots 1-24
!
controller T1 0/0/1
framing esf
linecode b8zs
channel-group 0 timeslots 1-24
!
controller T1 0/1/0
framing esf
linecode b8zs
channel-group 0 timeslots 1-24
!
controller T1 0/1/1
framing esf
linecode b8zs
channel-group 0 timeslots 1-24
!
interface Multilink1
description Multilink Bundle to Ed Center (AT&T 3Mb)
ip address 172.19.254.101 255.255.255.252
ppp multilink
ppp multilink group 1
service-policy output LAN-EDGE
!
interface Serial0/0/0:0
description T1 line 1
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/0/1:0
description T1 line 2
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/1/0:0
description T1 line 3
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/1/1:0
description T1 line 4
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

sh ppp multilink

Multilink1, bundle name is vgectr01
  Endpoint discriminator is vgectr01
  Bundle up for 3d23h, total bandwidth 3072, load 5/255
  Receive buffer limit 24000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    306 lost fragments, 6783707 reordered
    300/184999 discarded fragments/bytes, 0 lost received
    0xD18FEA received sequence, 0xB0775C sent sequence
  Member links: 2 active, 2 inactive (max not set, min not set)
    Se0/0/1:0, since 3d23h
    Se0/0/0:0, since 3d23h
    Se0/1/0:0 (inactive)
    Se0/1/1:0 (inactive)
No inactive multilink interfaces

16 Replies 16

Ivan Shirshin
Cisco Employee
Cisco Employee

Hello,

Are the Serial interfaces in up/up state? Please attach the "show int", "show version", "show inventory" outputs from both routers.

Kind Regards,
Ivan Shirshin

Kind Regards,
Ivan

The serial interfaces show down.  Below are the sh int, sh ver, sh inv:

***CENTRAL OFFICE***

vgectr01#sh int s2/0/0:1
Serial2/0/0:1 is down, line protocol is down
  Hardware is DSX1
  Description: Vang Pao link 3
  MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Closed, multilink Closed, loopback not set
  Keepalive set (10 sec)
  Last input 23:02:19, output never, output hang never
  Last clearing of "show interface" counters 1w0d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1152 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     220 packets input, 4730 bytes, 0 no buffer
     Received 0 broadcasts, 37 runts, 93 giants, 0 throttles
     62 input errors, 62 CRC, 0 frame, 0 overrun, 0 ignored, 17 abort
     220 packets output, 4730 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     5 carrier transitions
vgectr01#sh int s2/0/1:1
Serial2/0/1:1 is down, line protocol is down
  Hardware is DSX1
  Description: Vang Pao link 4
  MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
     reliability 210/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Closed, multilink Closed, loopback not set
  Keepalive set (10 sec)
  Last input 17:29:05, output never, output hang never
  Last clearing of "show interface" counters 1w0d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1152 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     332 packets input, 7138 bytes, 0 no buffer
     Received 0 broadcasts, 833910 runts, 2 giants, 0 throttles
     1905649 input errors, 1905649 CRC, 0 frame, 0 overrun, 0 ignored, 950859 abort
     410 packets output, 9400 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     8 carrier transitions
vgectr01#sh ver
Cisco IOS Software, 3800 Software (C3845-ADVIPSERVICESK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 19-Oct-09 21:46 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

vgectr01 uptime is 1 year, 11 weeks, 4 days, 12 hours, 50 minutes
System returned to ROM by reload at 16:59:42 PDST Fri Jun 10 2011
System restarted at 17:01:01 PDST Fri Jun 10 2011
System image file is "flash:c3845-advipservicesk9-mz.124-24.T2.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 3845 (revision 1.0) with 482303K/41984K bytes of memory.
Processor board ID FTX1128A37J
2 Gigabit Ethernet interfaces
124 Serial interfaces
11 Channelized/Clear T1/PRI ports
1 Virtual Private Network (VPN) Module
6 Voice FXO interfaces
8 Voice FXS interfaces
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
125440K bytes of ATA System CompactFlash (Read/Write)

Configuration register is 0x2102

vgectr01# sh inv
NAME: "3845 chassis", DESCR: "3845 chassis"
PID: CISCO3845         , VID: V01 , SN: FTX1128A37J

NAME: "c3845 Motherboard with Gigabit Ethernet on Slot 0", DESCR: "c3845 Motherboard with Gigabit Ethernet"
PID: CISCO3845-MB      , VID: V06 , SN: FOC112556LQ

NAME: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1 on Slot 0 SubSlot 0", DESCR: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1"
PID: VWIC2-2MFT-T1/E1  , VID: V01 , SN: FOC11260KQ0

NAME: "MFT-EC-64 - 64 Channel Hardware Echo Canceller on Slot 0 SubSlot 0", DESCR: "MFT-EC-64 - 64 Channel Hardware Echo Canceller"
PID: EC-MFT-64         , VID: V01 , SN: FOC1119231D

NAME: "1000BASE-SX SFP", DESCR: "1000BASE-SX SFP"
PID: FTLF8519P2BCL-CS0 , VID: 0000^C, SN: FNS10510RCD

NAME: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1 on Slot 0 SubSlot 1", DESCR: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1"
PID: VWIC2-2MFT-T1/E1  , VID: V01 , SN: FOC11260LEM

NAME: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1 on Slot 0 SubSlot 2", DESCR: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1"
PID: VWIC2-2MFT-T1/E1  , VID: V01 , SN: FOC11260MA3

NAME: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1 on Slot 0 SubSlot 3", DESCR: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1"
PID: VWIC2-2MFT-T1/E1  , VID: V01 , SN: FOC11260KVJ

NAME: "PVDMII DSP SIMM with four DSPs on Slot 0 SubSlot 4", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11261D0R

NAME: "PVDMII DSP SIMM with four DSPs on Slot 0 SubSlot 5", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11261D55

NAME: "PVDMII DSP SIMM with four DSPs on Slot 0 SubSlot 6", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11261D00

NAME: "PVDMII DSP SIMM with four DSPs on Slot 0 SubSlot 7", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11261D04

NAME: "High Density Voice Module - 8FXS/DID on Slot 1", DESCR: "High Density Voice Module - 8FXS/DID"
PID: EVM-HD-8FXS/DID   , VID: V03 , SN: FOC111933R2

NAME: "Six port FXO voice interface daughtercard on Slot 1 SubSlot 0", DESCR: "Six port FXO voice interface daughtercard"
PID: EM-HDA-6FXO       , VID: V02 , SN: FOC111933HE

NAME: "High Density Voice2 Network module with on board one port interface  on Slot 2", DESCR: "High Density Voice2 Network module with on board one port interface "
PID: NM-HDV2-1T1/E1    , VID: V03 , SN: FOC11376997

NAME: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1 on Slot 2 SubSlot 0", DESCR: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1"
PID: VWIC2-2MFT-T1/E1  , VID: V01 , SN: FOC11380ZEC

NAME: "PVDMII DSP SIMM with four DSPs on Slot 2 SubSlot 2", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11374BZB

NAME: "PVDMII DSP SIMM with four DSPs on Slot 2 SubSlot 3", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11374BZF

NAME: "PVDMII DSP SIMM with four DSPs on Slot 2 SubSlot 4", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11374AC1


vgectr01#


***REMOTE SITE***


er-VANGPAO-TEMP#SH INT S0/1/0:0
Serial0/1/0:0 is down, line protocol is down
  Hardware is GT96K Serial
  Description: T1 line 3
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Closed, multilink Closed, loopback not set
  Keepalive set (10 sec)
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 4d19h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/0/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1152 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
  Timeslot(s) Used:1-24, SCC: 0, Transmitter delay is 0 flags
er-VANGPAO-TEMP#SH INT S0/1/1:0
Serial0/1/1:0 is down, line protocol is down
  Hardware is GT96K Serial
  Description: T1 line 4
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Closed, multilink Closed, loopback not set
  Keepalive set (10 sec)
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 4d19h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/0/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1152 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
  Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags
er-VANGPAO-TEMP#SH VER
Cisco IOS Software, 3800 Software (C3845-ADVENTERPRISEK9-M), Version 12.4(17), R
ELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Fri 07-Sep-07 23:03 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

er-VANGPAO-TEMP uptime is 4 days, 19 hours, 59 minutes
System returned to ROM by power-on
System image file is "flash:c3845-adventerprisek9-mz.124-17.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 3845 (revision 1.0) with 484352K/39936K bytes of memory.
Processor board ID FTX1140A2L7
2 Gigabit Ethernet interfaces
8 Serial interfaces
4 Channelized T1/PRI ports
1 Virtual Private Network (VPN) Module
4 Voice FXO interfaces
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
125440K bytes of ATA System CompactFlash (Read/Write)

Configuration register is 0x2102

er-VANGPAO-TEMP#SH INV
NAME: "3845 chassis", DESCR: "3845 chassis"
PID: CISCO3845         , VID: V01 , SN: FTX1140A2L7

NAME: "c3845 Motherboard with Gigabit Ethernet", DESCR: "c3845 Motherboard with
Gigabit Ethernet"
PID: CISCO3845-MB      , VID: V06 , SN: FOC11372RHR

NAME: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1", DESCR: "VWIC2-2
MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1"
PID: VWIC2-2MFT-T1/E1  , VID: V01 , SN: FOC11380Z6S

NAME: "1000BASE-SX SFP", DESCR: "1000BASE-SX SFP"
PID: SFBR-5766PZ       , VID:    , SN: AGM125111WW

NAME: "VWIC2-2MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1", DESCR: "VWIC2-2
MFT-T1/E1 - 2-Port RJ-48 Multiflex Trunk - T1/E1"
PID: VWIC2-2MFT-T1/E1  , VID: V01 , SN: FOC11362XDN

NAME: "2nd generation four port FXO voice interface daughtercard", DESCR: "2nd g
eneration four port FXO voice interface daughtercard"
PID: VIC2-4FXO         , VID: V02 , SN: FOC11384B93

NAME: "PVDMII DSP SIMM with four DSPs", DESCR: "PVDMII DSP SIMM with four DSPs"
PID: PVDM2-64          , VID: V01 , SN: FOC11382ZG9


er-VANGPAO-TEMP#

Hello Dtom,

In this case, you will need to isolate the issue. First at all, I recommend performing a hardware loopback test to make sure the new ports are working fine. You can also connect one working T1 to one of the new ports to determine if the interface and T1 controller are coming up. If that happens, probably the issue is related to the provider. On the other hand, I can see that you are participating the data T1s that are part of the multilink and this is not recommended for data circuits for it causes clocking issues. The "network-clock-participate" and "network-clock-select" are used for IMAs and PRIs so I recommend getting rid of those commands related to the cards that are part of the multilink.

Please gather the following outputs for me too:

  • show controllers T1
  • show ppp multilink
  • show interface multilink 1

I request these commands for a previous "show ppp multilink 1" shows lost fragments and this can be caused by the "participate" commands you are using. Thus, I need to check if you are being affected by clocking issues.

Regards.

The clock source should be line if the provider is "providing" clocking somewhere in between. Otherwise, one side should be line and the other should use internal. Please check clocking settings before.

Hello,,

Pls do HArd Loop back test, but I guess it will fail.

As per the Output of show interface serial at Central Router, the port status is down, check out the cable connectivity at the serial port. As per my experiance the port should not show down untill unless there is any physical connectivity issue. Line Protocol shows down for varius issue at network but not port status.

Regards

Rahul Mishra

vmiller
Level 7
Level 7

First thing I see, your clock source should not be internal, set it to line at both ends.

I will take a look at the clock settings (not very familuar with it thought).  I will also move the lines around to see if anything changes.  Here is the information requested:

***CENTRAL OFFICE***

vgectr01#sh controllers t1
T1 0/0/0 is up.
  Applique type is Channelized T1
  Cablelength is long 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (673 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     673 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     673 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     86400 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     86400 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/0/1 is up.
  Applique type is Channelized T1
  Cablelength is long 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (674 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     674 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     674 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     86400 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     86400 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/0 is up.
  Applique type is Channelized T1
  Cablelength is long 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Internal.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (709 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/1 is up.
  Applique type is Channelized T1
  Cablelength is long 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Internal.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (709 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/2/0 is up.
  Applique type is Channelized T1
  Cablelength is long 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (589 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/2/1 is up.
  Applique type is Channelized T1
  Cablelength is long 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (590 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/3/0 is up.
  Applique type is Channelized T1
  Cablelength is long 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (705 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/3/1 is down.
  Applique type is Channelized T1
  Cablelength is long 0db
  Transmitter is sending remote alarm.
  Receiver has loss of signal.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (705 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 705 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs
T1 2/0 is down.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Transmitter is sending remote alarm.
  Receiver has loss of signal.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 255, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  Current port master clock:recovered from backplane
  Data in current interval (732 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 732 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs
T1 2/0/0 is down.
  Applique type is Channelized T1
  Cablelength is long 0db
  Receiver has remote alarm.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Internal.
  Current port master clock:recovered from backplane
  Data in current interval (667 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 667 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs
T1 2/0/1 is down.
  Applique type is Channelized T1
  Cablelength is long 0db
  Receiver has remote alarm.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20090408, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Internal.
  Current port master clock:recovered from backplane
  Data in current interval (667 seconds elapsed):
     0 Line Code Violations, 22 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 667 Unavail Secs
  Total Data (last 24 hours)
     497 Line Code Violations, 2641 Path Code Violations,
     2 Slip Secs, 1 Fr Loss Secs, 0 Line Err Secs, 8 Degraded Mins,
     13 Errored Secs, 9 Bursty Err Secs, 1 Severely Err Secs, 85355 Unavail Secs
vgectr01#

vgectr01#sh ppp multilink

Multilink1
  Bundle name: er-VANGPAO-TEMP
  Remote Endpoint Discriminator: [1] er-VANGPAO-TEMP
  Local Endpoint Discriminator: [1] vgectr01
  Bundle up for 5d02h, total bandwidth 3072, load 88/255
  Receive buffer limit 24000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    218 lost fragments, 8378853 reordered
    202/57474 discarded fragments/bytes, 0 lost received
    0x6D69C2 received sequence, 0xC536EB sent sequence
  Member links: 2 active, 2 inactive (max not set, min not set)
    Se0/1/0:1, since 5d02h
    Se0/1/1:1, since 5d02h
    Se2/0/0:1 (inactive)
    Se2/0/1:1 (inactive)
No inactive multilink interfaces
vgectr01#
vgectr01#sh int multilink 1
Multilink1 is up, line protocol is up
  Hardware is multilink group interface
  Description: Multilink Bundle to Dakota Campus (AT&T 3Mb)
  Internet address is 172.19.254.102/30
  MTU 1500 bytes, BW 3072 Kbit/sec, DLY 100000 usec,
     reliability 255/255, txload 87/255, rxload 15/255
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP, CDPCP, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 2 seconds on reset
  Last input 00:00:04, output never, output hang never
  Last clearing of "show interface" counters 27w6d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1118063
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/0 (size/max total/drops)
  5 minute input rate 190000 bits/sec, 180 packets/sec
  5 minute output rate 1050000 bits/sec, 241 packets/sec
     128764117 packets input, 351983636 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     2250 input errors, 0 CRC, 543 frame, 0 overrun, 0 ignored, 529 abort
     157268319 packets output, 3836912367 bytes, 0 underruns
     0 output errors, 0 collisions, 12 interface resets
     2 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
vgectr01#


***REMOTE SITE***

er-VANGPAO-TEMP#sh controllers t1
T1 0/0/0 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20070725, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (87 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     2 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     2 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     1432 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     1432 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/0/1 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20070725, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (87 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     1 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     1 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     1433 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     1433 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/0 is down.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Transmitter is sending remote alarm.
  Receiver has loss of signal.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20070725, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (87 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 87 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs
T1 0/1/1 is down.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Transmitter is sending remote alarm.
  Receiver has loss of signal.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20070725, FPGA: 13, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (87 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 87 Unavail Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs
er-VANGPAO-TEMP#
er-VANGPAO-TEMP#sh ppp multilink

Multilink1, bundle name is vgectr01
  Endpoint discriminator is vgectr01
  Bundle up for 5d02h, total bandwidth 3072, load 16/255
  Receive buffer limit 24000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    483 lost fragments, 10451510 reordered
    473/296250 discarded fragments/bytes, 0 lost received
    0xC5869C received sequence, 0x6D9DC7 sent sequence
  Member links: 2 active, 2 inactive (max not set, min not set)
    Se0/0/1:0, since 5d02h
    Se0/0/0:0, since 5d02h
    Se0/1/0:0 (inactive)
    Se0/1/1:0 (inactive)
No inactive multilink interfaces
er-VANGPAO-TEMP#
er-VANGPAO-TEMP#sh int multilink 1
Multilink1 is up, line protocol is up
  Hardware is multilink group interface
  Description: Multilink Bundle to Ed Center (AT&T 3Mb)
  Internet address is 172.19.254.101/30
  MTU 1500 bytes, BW 3072 Kbit, DLY 100000 usec,
     reliability 255/255, txload 16/255, rxload 83/255
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP, CDPCP, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 2 seconds on reset
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters 5d02h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 766
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/766 (size/max total/threshold/drops)
     Conversations  0/4/16 (active/max active/max total)
     Reserved Conversations 2/2 (allocated/max allocated)
     Available Bandwidth 616 kilobits/sec
  5 minute input rate 1000000 bits/sec, 238 packets/sec
  5 minute output rate 198000 bits/sec, 173 packets/sec
     25664477 packets input, 3097761567 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     956 input errors, 0 CRC, 244 frame, 0 overrun, 0 ignored, 229 abort
     24232579 packets output, 331162312 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
er-VANGPAO-TEMP#

Hello Dtom,

In this case, we need to take a look at the status of the controllers T1 2/0/0 and 2/0/1. We are observing a remote alarm:

T1 2/0/0 is down.

  Applique type is Channelized T1

  Cablelength is long 0db

  Receiver has remote alarm.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info Firmware: 20090408, FPGA: 13, spm_count = 0

  Framing is ESF, Line Code is B8ZS, Clock Source is Internal.

  Current port master clock:recovered from backplane

  Data in current interval (667 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 667 Unavail Secs

  Total Data (last 24 hours)

     0 Line Code Violations, 0 Path Code Violations,

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs

T1 2/0/1 is down.

  Applique type is Channelized T1

  Cablelength is long 0db

Receiver has remote alarm.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info Firmware: 20090408, FPGA: 13, spm_count = 0

  Framing is ESF, Line Code is B8ZS, Clock Source is Internal.

  Current port master clock:recovered from backplane

  Data in current interval (667 seconds elapsed):

     0 Line Code Violations, 22 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 667 Unavail Secs

  Total Data (last 24 hours)

     497 Line Code Violations, 2641 Path Code Violations,

     2 Slip Secs, 1 Fr Loss Secs, 0 Line Err Secs, 8 Degraded Mins,

     13 Errored Secs, 9 Bursty Err Secs, 1 Severely Err Secs, 85355 Unavail Secs

This means that the far-end equipment has a problem with the signal. I recommend getting your Telco involved to check if the new T1s were already activated. On the other hand, you can move the line around to gather more evidence if you need to talk to the provider again. Furthermore, it is important to focus on some other details such as clocking (participate commands and clock source) and a potential differential delay issue for there are input errors in the interface multilink while lost fragments were increasing. Input errors in the new T1s would be expected due to the interface status.

Regards.

I disagree with Jorge on one point.

If these circuits are physically from your HQ to a Remote, then all timing should be network.

you should only use internal if you are running a back to back link.

Hello,

Jose posting on 'Aug 30, 2012 9:23 AM' is the correct place to start t-shoot on this issue. When you have Point to point links, the carrier should not provide any clocking. One end should be 'line' and other end should be 'internal'. Do that first.

Second- Remove all multilink configs from new T1s and First try to bring up individual T1s. If the T1s still down after 'Clock' adjustment, then check on 'show controllers' and try to run soft loopback tests.

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a754b.shtml

If you have 'hard' loopback plug that would be great. Right away you can prove where the issue is. As this is new T1s- 99.99% its a carrier issue. Push the carrier to send techs to both ends with Test sets. That proves where the issue is.

hth

MS

Jose posting on 'Aug 30, 2012 9:23 AM' is the correct place to start t-shoot on this issue. When you have Point to point links, the carrier should not provide any clocking. One end should be 'line' and other end should be 'internal'. Do that first.

That is not always the case. Altough common in the US, I have seen circuits that provide clock on both ends. Anywhere else in the world, clock is provided to both ends.

My own experience (US): Any telco ordered circut (unless specified) even if its point to point will go through a telco CO.

While the circuit is point to point, at the physical layer, it may be comprised of multiple segements. I always take clock from the line.

In some rare cases you just get copper from the telco (LADS circuits). In this case you slave one clock off the line and one clock internal. Haven't done this in years.

Three questions:

1) I remove the following statements from tthe central office router:

network-clock-participate slot 2

network-clock-participate wic 1

The "good" t1 line went down.  Do I need to do this on both the central office and remote office at the same time?

2) Do you suggest I do the following:

***Central Office Router***

Remove:

network-clock-participate slot 2

network-clock-participate wic 1

On the following interfaces:

S0/1/0:1

S0/1/1:1

S2/0/0:1

S2/0/1:1

Add:

clock source internal

***Remote Site***

Remove:

network-clock-participate wic 0

network-clock-participate wic 1

On the following interfaces:

S0/0/0:0

S0/0/1:0

S0/1/0:0

S0/1/1:0

Add:

clock source line

3) Does anyone know of any good documents that explains clocking (for dummies)?

network-clock-participate slot 2

network-clock-participate wic 1

"

To allow the ports on a specified network module or voice/WAN interface card (VWIC) to use the network clock for timing, use the

network-clock-participate

command in global configuration mode. To restrict the device to use only its own clock signals, use the

no form of this command.

To allow the ports on a specified network module or voice/WAN interface card (VWIC) to use the network clock for timing, use the network-clock-participate command in global configuration mode. To restrict the device to use only its own clock signals, use the no form of this command. "

So, not a good idea IMHO

try clock source line on everything

http://oreilly.com/catalog/t1survival/chapter/ch05.html

Review Cisco Networking products for a $25 gift card