MLPPP-4 T1 bundle, 1721 router

Unanswered Question
Mar 21st, 2008

We have a customer who is using three bonded T1 circuits on a 1721 router. He recently purchased a fourth line with the intention of adding it to his bundle. I attempted to replace his current 1721 router, which contains one WIC-1DSU-T1 card and one VWIC-2MFT-T1 card, with another 1721, which contains two VWIC-2MFT-T1 cards. The current router functions properly with any three--or fewer--of the four circuits, however, the new router is unable to maintain a steady connection on any of the four circuits; all ports intermittently come "up,up," but almost immediately return to the "reset, down." Is this, perhaps, an issue with the hardware, or is there a portion of the configuration which needs to be modified?

New 1721 Config:

!

version 12.3

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname XXXXXXXXXXX

!

boot-start-marker

boot-end-marker

!

logging buffered 8192 errors

enable secret xxx

!

memory-size iomem 25

no aaa new-model

ip subnet-zero

!

!

!

ip cef

!

!

!

controller T1 0

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

controller T1 1

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

controller T1 2

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

controller T1 3

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

!

!

interface MultilinkGROUP

ip address XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX

ppp multilink

ppp multilink group GROUP

!

interface FastEthernet0

ip address XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX

no ip redirects

no ip unreachables

no ip proxy-arp

speed 100

full-duplex

no cdp enable

!

interface Serial0:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

interface Serial1:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

interface Serial2:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

interface Serial3:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

ip default-gateway XXX.XXX.XXX.XXX

ip classless

ip route 0.0.0.0 0.0.0.0 MultilinkGROUP

no ip http server

!

!

line con 0

line aux 0

line vty 0 4

password 7

login

!

end

Sample Error Messages:

(taken from a 2600; the only difference is the controller IDs)

*Mar 1 00:29:00.649: %CONTROLLER-5-UPDOWN: Controller T1 0/3, changed state to up

*Mar 1 00:29:02.652: %LINK-3-UPDOWN: Interface Serial0/3:0, changed state to up

*Mar 1 00:29:04.647: %CONTROLLER-5-UPDOWN: Controller T1 0/3, changed state to down (LOF detected)

*Mar 1 00:29:06.651: %LINK-5-CHANGED: Interface Serial0/3:0, changed state to reset

*Mar 1 00:29:07.648: %CONTROLLER-5-UPDOWN: Controller T1 0/3, changed state to up

*Mar 1 00:29:09.648: %LINK-3-UPDOWN: Interface Serial0/3:0, changed state to up

*Mar 1 00:29:12.148: %CONTROLLER-5-UPDOWN: Controller T1 0/1, changed state to up

*Mar 1 00:29:14.151: %LINK-3-UPDOWN: Interface Serial0/1:0, changed state to up

*Mar 1 00:29:15.645: %CONTROLLER-5-UPDOWN: Controller T1 0/2, changed state to up

*Mar 1 00:29:17.649: %LINK-3-UPDOWN: Interface Serial0/2:0, changed state to up

*Mar 1 00:29:21.647: %CONTROLLER-5-UPDOWN: Controller T1 0/3, changed state to down (LOF detected)

*Mar 1 00:29:23.647: %LINK-5-CHANGED: Interface Serial0/3:0, changed state to reset

*Mar 1 00:29:24.648: %CONTROLLER-5-UPDOWN: Controller T1 0/2, changed state to down (LOF detected)

*Mar 1 00:29:24.652: %CONTROLLER-5-UPDOWN: Controller T1 0/3, changed state to up

*Mar 1 00:29:26.648: %LINK-5-CHANGED: Interface Serial0/2:0, changed state to reset

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Richard Burts Fri, 03/21/2008 - 12:39

Joe

I believe that the problem is that the 1721 can not handle the load of 4 T1s doing multilink. I have seen a router performance document that indicates that the best case (CEF/Fast Switching with no services enabled) for the 1721 is about 6 Mbps. And the 4 T1s would give you that amount. I am not sure what the overhead is of running multilink, but I suspect that it is pushing the 1721 slightly over the edge.

HTH

Rick

cari.net Mon, 03/24/2008 - 08:53

Thank you very much for the reply. Do you think that a 2811 would be able to easily support the given configuration? Also, do you have a link to the technical documentation regarding the 1700's bandwidth limitation?

Richard Burts Mon, 03/24/2008 - 09:20

Joe

I feel pretty confident that a 2811 would easily support the 4 T1 members of the multilink. Where the max for the 1721 (running with NO services configured) was right at 6 Mbps the max for the 2811 is about 61 Mbps. So the 2811 has about a 10 times increase in performance and should handle the 4 T1s in multilink fine.

The link to the documentations is:

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

note that it requires partner level access. I did a quick look for public level access but have not found anything.

HTH

Rick

cari.net Tue, 03/25/2008 - 14:32

I set our customer up with a 2821, but I still had no luck getting the four circuits to run together. The 2821's configuration is nearly identical to that provided above (except for the controller IDs). I am assuming that there is something I am missing in the configuration(?). The only thing that I can think of is the "network-clock participate wic x" setting, but I imagine that is similar to the "clock source" command on the 1721. Thanks in advance...

Paolo Bevilacqua Tue, 03/25/2008 - 15:43

By all means please set network-clock-participate, but that alone doesn't explain the bundle to not come up.

A bit more troubleshooting would be needed including show intefaces, debug ppp negotiation, just to begin with.

Richard Burts Wed, 03/26/2008 - 05:44

Joe

In your original post there were error indication of Loss of Frame and they were spread over various members of the multilink bundle. I am not clear whether the symptoms are the same on the 2811 or are different. Perhaps a new description of the symptoms would be helpful. If you can post some of the specific log messages and perhaps the new config, then perhaps we would be better able to figure out the problem.

HTH

Rick

cari.net Fri, 03/28/2008 - 10:26

Here is a copy of the configuration from the 2821 that I am attempting to use. It may also be important to note that the other side of the connection is a 7206VXR (NPE 300).

Current configuration : 1791 bytes

!

version 12.4

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname GROUP

!

boot-start-marker

boot-end-marker

!

logging buffered 8192 errors

enable secret xxx

!

no aaa new-model

!

resource policy

!

network-clock-participate wic 0

network-clock-participate wic 1

ip subnet-zero

!

!

ip cef

!

!

no ip domain lookup

!

!

voice-card 0

no dspfarm

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

controller T1 0/0/0

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

controller T1 0/0/1

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

controller T1 0/1/0

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

controller T1 0/1/1

framing esf

linecode b8zs

channel-group 0 timeslots 1-24

!

!

!

!

!

interface MultilinkGROUP

ip address XXX.XXX.XXX.XXX

ppp multilink

ppp multilink group GROUP

!

interface GigabitEthernet0/0

no ip address

shutdown

duplex auto

speed auto

!

interface GigabitEthernet0/1

ip address XXX.XXX.XXX.XXX 255.255.255.248

duplex full

speed 100

!

interface Serial0/0/0:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

interface Serial0/0/1:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

interface Serial0/1/0:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

interface Serial0/1/1:0

no ip address

encapsulation ppp

ppp multilink

ppp multilink group GROUP

!

ip default-gateway XXX.XXX.XXX.XXX

ip classless

ip route 0.0.0.0 0.0.0.0 MultilinkGROUP

!

!

no ip http server

no ip http secure-server

!

!

!

!

!

control-plane

!

!

!

!

!

!

!

!

!

!

line con 0

line aux 0

line vty 0 4

password 7

login

!

scheduler allocate 20000 1000

!

end

Richard Burts Fri, 03/28/2008 - 10:54

Joe

Thanks for posting the config of the new router. This config looks fine and I do not see any obvious issues with it.

A 2821 with this config should certainly be able to handle multilink with 4 T1s. And depending on what else it is configured to do a 7206VXR with NPE300 should be ok with this load. So while I suspect you were overloading the 1721, I believe the current hardware is adequate.

Can you give us a description of the current problems and perhaps copies of relevant error messages?

HTH

Rick

cari.net Mon, 03/31/2008 - 17:06

I tried testing the line using the single available circuit while running "debug ppp negotiation." The following is the logged debugging result:

*Mar 31 21:28:53.383: %CONTROLLER-5-UPDOWN: Controller T1 0/1/1, changed state to up

*Mar 31 21:28:55.383: %LINK-3-UPDOWN: Interface Serial0/1/1:0, changed state to up

*Mar 31 21:28:55.387: Se0/1/1:0 PPP: Using default call direction

*Mar 31 21:28:55.387: Se0/1/1:0 PPP: Treating connection as a dedicated line

*Mar 31 21:28:55.387: Se0/1/1:0 PPP: Session handle[F000000A] Session id[1]

*Mar 31 21:28:55.387: Se0/1/1:0 PPP: Phase is ESTABLISHING, Active Open

*Mar 31 21:28:55.387: Se0/1/1:0 LCP: O CONFREQ [Closed] id 1 len 24

*Mar 31 21:28:55.387: Se0/1/1:0 LCP: MagicNumber 0x13C72D76 (0x050613C72D76)

*Mar 31 21:28:55.387: Se0/1/1:0 LCP: MRRU 1500 (0x110405DC)

*Mar 31 21:28:55.387: Se0/1/1:0 LCP: EndpointDisc 1 GROUP (0x130A01576562734F776E)

*Mar 31 21:28:57.371: Se0/1/1:0 LCP: TIMEout: State REQsent

*Mar 31 21:28:57.371: Se0/1/1:0 LCP: O CONFREQ [REQsent] id 2 len 24

*Mar 31 21:28:57.371: Se0/1/1:0 LCP: MagicNumber 0x13C72D76 (0x050613C72D76)

*Mar 31 21:28:57.371: Se0/1/1:0 LCP: MRRU 1500 (0x110405DC)

*Mar 31 21:28:57.371: Se0/1/1:0 LCP: EndpointDisc 1 GROUP (0x130A01576562734F776E)

*Mar 31 21:28:59.387: Se0/1/1:0 LCP: TIMEout: State REQsent

*Mar 31 21:28:59.387: Se0/1/1:0 LCP: O CONFREQ [REQsent] id 3 len 24

*Mar 31 21:28:59.387: Se0/1/1:0 LCP: MagicNumber 0x13C72D76 (0x050613C72D76)

*Mar 31 21:28:59.387: Se0/1/1:0 LCP: MRRU 1500 (0x110405DC)

*Mar 31 21:28:59.387: Se0/1/1:0 LCP: EndpointDisc 1 GROUP (0x130A01576562734F776E)

*Mar 31 21:29:01.403: Se0/1/1:0 LCP: TIMEout: State REQsent

*Mar 31 21:29:01.403: Se0/1/1:0 LCP: O CONFREQ [REQsent] id 4 len 24

*Mar 31 21:29:01.403: Se0/1/1:0 LCP: MagicNumber 0x13C72D76 (0x050613C72D76)

*Mar 31 21:29:01.403: Se0/1/1:0 LCP: MRRU 1500 (0x110405DC)

*Mar 31 21:29:01.403: Se0/1/1:0 LCP: EndpointDisc 1 GROUP (0x130A01576562734F776E)

*Mar 31 21:29:02.635: %CONTROLLER-5-UPDOWN: Controller T1 0/1/1, changed state to down (10 SES)

*Mar 31 21:29:03.419: Se0/1/1:0 LCP: TIMEout: State REQsent

*Mar 31 21:29:03.419: Se0/1/1:0 LCP: O CONFREQ [REQsent] id 5 len 24

*Mar 31 21:29:03.419: Se0/1/1:0 LCP: MagicNumber 0x13C72D76 (0x050613C72D76)

*Mar 31 21:29:03.419: Se0/1/1:0 LCP: MRRU 1500 (0x110405DC)

*Mar 31 21:29:03.419: Se0/1/1:0 LCP: EndpointDisc 1 GROUP (0x130A01576562734F776E)

*Mar 31 21:29:04.635: %LINK-3-UPDOWN: Interface Serial0/1/1:0, changed state to down

*Mar 31 21:29:04.635: Se0/1/1:0 PPP: Sending Acct Event[Down] id[3]

*Mar 31 21:29:04.635: Se0/1/1:0 LCP: State is Closed

*Mar 31 21:29:04.635: Se0/1/1:0 PPP: Phase is DOWN

*Mar 31 21:41:18.411: %CONTROLLER-5-UPDOWN: Controller T1 0/1/0, changed state to down (LOS detected)

*Mar 31 21:41:18.411: %CONTROLLER-5-UPDOWN: Controller T1 0/1/1, changed state to down (LOS detected)

The log entries on both the 2821 and the VXR are almost identical. In both cases it seems that neither router responds to the other's LCP Requests.

Richard Burts Mon, 03/31/2008 - 19:29

Joe

I am a bit confused. I thought we were talking about routers with a multilink with 4 T1s in the multilink bundle. But your comment is:

testing the line using the single available circuit

are we talking a single circuit or multiple circuits. Also I note in the debug posted that 1 T1 controller comes up at the beginning of the debug and that 2 controllers go down at the end. Did the second controller come up just before the place where you selected your debug output to post?

I agree that the debug would seem to indicate that the routers are not responding to each other. I wonder if there might be an issue about having the correct cable connected to the correct port?

HTH

Rick

websown Mon, 03/31/2008 - 19:40

He attempted to use one line while the three remaining lines were in production use. The three T1s currently in use are working fine, and have done so for multiple years. Converting to four t1s... it all falls apart.

Richard Burts Mon, 03/31/2008 - 19:59

Robert

Do you work with Joe? If so can you tell me whether the debug output represents attempting to use the 4th circuit in the multilink bundle or was it operating independently?

HTH

Rick

websown Mon, 03/31/2008 - 20:06

No. I am the customer. So... my answers may be brief and very off base.

I believe he was attempting to use it in a multilink bundle with only one live line.

The problem occurs by just starting the system ATTEMPTING to run four lines wether or not all four lines are connected. He was testing if with one line connected from the bundle setup for four lines would work. This debug is the result.

He only had one T1 connected durring these tests I believe.

Richard Burts Mon, 03/31/2008 - 20:28

Robert

OK so you do have some insight into what is going on with this issue. I believe that the debug seems to show some issue with that circuit. I would then ask what happens if you do not try the multilink but just run that circuit as a serial link on its own? Does it function correctly? Or does it have problems?

Do the neighbors on each end come up properly and communicate with each other? (based on the debug I would guess that they do not - but it would be good to know this).

HTH

Rick

websown Mon, 03/31/2008 - 20:34

We are planning on testing that starting tomorrow.

I know that we did try breaking into four links, all seperate and failed to have connectivity.

What gets me is that any combination of 3 T1s work but just change the settings and try to use four and they all go down.

cari.net Wed, 04/02/2008 - 09:30

Rick

Sorry for the confusion, I changed my troubleshooting strategy. Instead of trying to work with all four lines at once, I decided that I should try getting one line to work and build off of that success.

Yesterday I did some testing with the available circuit and came across both predictable and unpredictable results. I first used a 1721 and successfully built the connection up from a Layer-2 HDLC connection to a Layer-3 PPP connection. Next I put the 2821 in place and attempted the same test but failed at the first step (Layer-2 HDLC [Note: The 2821 had 2 VWIC-2MFT-T1 cards in HWIC slots 0&1]).

Since the 2821 did not seem to work "as is" I decided to use hardware from the 1721 to validate the test. I installed a VWIC-2MFT-T1 card from the 1721 into HWIC slot 2 on the 2821 and I was able to replicate my results from the 1721 on both controllers. I then moved a pre-installed card into HWIC slot 3, however, the testing that followed yielded identical results to previous tests with the 2821: LOF messages, link bouncing, etc.

Is it possible that there is a particular hardware orientation required for this configuration to function properly?

Richard Burts Sun, 04/06/2008 - 18:22

Joe

I am not clear whether you have made progress in resolving this? I find myself a bit confused about which cards worked in which slots? And whether the symptoms point more at a problem with a particular card or with a particular slot. Can you clarify this?

I found this thread interesting in that it talks about success with multilink on more interfaces that you have been trying to get to work:

http://forums.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=WAN%2C%20Routing%20and%20Switching&topicID=.ee71a06&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cc005cb

HTH

Rick

cari.net Mon, 04/07/2008 - 11:19

Rick,

Thanks for all of your help. In the end it turned out to be an issue with the WICs that I was using. After accepting that the configuration was not the issue, I moved on to the hardware itself. Once I replaced the VWIC-2MFT-T1 cards that I was using with some others that we had in the office the installation became routine again.

The one thing that still has me somewhat frustrated is the fact that two cards that met all requirements during testing were completely useless in the field. I suppose the next issue is discerning useful from useless equipment when next we order hardware.

Thanks again.

Anthony Davis

Network Engineer

Cari.net

Richard Burts Mon, 04/07/2008 - 11:29

Anthony

Thanks for posting back and indicating that you had found a solution for your problem. It is interesting that it turned out to be a hardware problem. I think your experience here is one that many of us share - when we encounter a problem we first look for it to be something that we have done, and most especially for it to be a flaw in the configuration. But it turned out to be a hardware problem.

Do you know what the problem with the cards was? Were the cards bad? Is it an issue of compatibility of those cards in that platform? Is it an issue of compatibility of those cards with that version of software?

HTH

Rick

cari.net Mon, 04/07/2008 - 15:12

Rick

The interesting thing about the dysfunctional cards is that when I tested them in a lab environment they appeared to work properly. The closest that I could get to production conditions was to put the 2800 back-to-back with a 1721 and verify that each interface could successfully complete PPP negotiation. Of course, when I attempted to install them in the field they failed to (reliably) register any sort of connection. The notable differences between the test and real-world application were: (1) the circuit providers network and (2) the 7200 VXR router to which the 2800 connected.

Unfortunately, I am ignorant of any particular battery of tests that can be used to ferret our the source of the VWICs inoperability. The most I am able to do at the moment is find out as much as possible about the functioning cards so that we can distinguish them from others when we make purchases.

websown Mon, 03/31/2008 - 20:30

No. I am the customer. So... my answers may be brief and very off base.

I believe he was attempting to use it in a multilink bundle with only one live line.

The problem occurs by just starting the system ATTEMPTING to run four lines wether or not all four lines are connected. He was testing if with one line connected from the bundle setup for four lines would work. This debug is the result.

He only had one T1 connected durring these tests I believe.

Actions

This Discussion