Unstable MPLS circuit using an ATM interface

stephenshaw · ‎06-18-2009

Hi,

here's the scenario ... downsized an office using a 3825 router with an ATM DS3 Network Module and replaced that with a 2811 using a different ATM DS3 module. Putting the original router back is not an option.

Problem is that the ATM interface loses "line protocol" far too often - there is no pattern but seems to be related to traffic volume. i.e this is a 10M MPLS service and when traffic starts getting above 1M is when the line protocol starts dropping. It'll run perfectly clean for extended periods with minimal traffic but once we introduce a higher volume, we start getting packet drops and the ATM interface line protocol starts dropping.

flash:c2800nm-advipservicesk9-mz.124-22.T1.bin

ATM DS3 Port adapter, 1 port

Example from log:

Jun 18 08:33:29.960: %LINK-3-UPDOWN: Interface ATM1/0, changed state to up

Jun 18 08:33:30.960: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM1/0, changed state to up

Jun 18 08:33:33.256: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 69.183.x.173 (ATM1/0.1) is up: new adjacency

Jun 18 08:35:22.960: %LINK-3-UPDOWN: Interface ATM1/0, changed state to down

Jun 18 08:35:22.992: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 69.183.x.173 (ATM1/0.1) is down: interface down

Jun 18 08:35:23.960: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM1/0, changed state to down

Display of "s int atm1/0"

ATM1/0 is up, line protocol is up

Hardware is RS8234 ATM DS3

MTU 4470 bytes, sub MTU 4470, BW 45000 Kbit/sec, DLY 190 usec,

reliability 255/255, txload 1/255, rxload 5/255

Encapsulation ATM, loopback not set

Encapsulation(s): AAL5

1023 maximum active VCs, 1 current VCCs

VC Auto Creation Disabled.

VC idle disconnect time: 300 seconds

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 14

Queueing strategy: Per VC Queueing

5 minute input rate 946000 bits/sec, 96 packets/sec

5 minute output rate 38000 bits/sec, 63 packets/sec

2144548 packets input, 1163597874 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

2165984 packets output, 1215084533 bytes, 0 underruns

0 output errors, 0 collisions, 2 interface resets

0 unknown protocol drops

0 output buffer failures, 0 output buffers swapped out

Note the output drops but lack of log entries or errors indicating a problem with the ATM module. The 2 CRC errors are due to the interface resets.

My question is if anyone may have any idea what may be happening here to cause a perfectly good MPLS service to start "acting" up simply due to a hardware change. According to the Cisco site, the ATM module is compatible with the 2811 using the IOS code I have installed. I would expect to see output errors or log entries if the ATM module is defective. And don't see how the ATM module itself would cause the line protocol to drop.

The telco is engaged to do end-end testing to try and determine if it is our "new" equipment or something else causing the problem. They have also confirmed that the ATM configuration/parameters are correct and match what they expect.

Any ideas we can try would be appreciated.

Thanks,

Steve

paolo bevilacqua · ‎06-18-2009

Can you load the very same IOS that was on the 3845 ?

stephenshaw · ‎06-18-2009

Hi,

Cisco do not have the same IOS available. However, the version on the 2811 is a newer one within the same train.

What concerns me is the Cisco info on compatibility between the ATM module, the 2811 and the IOS codes was incorrect - I am leaning towards thinking we have a compatibility issue. No errors, no log entries, debugs only show the following:

Jun 18 11:38:42.019: rs8234_teardown_vc(ATM1/0): vc:1 vpi:15 vci:72

Jun 18 11:38:42.019: Status and ptr is 1800 Status Q is 6

Jun 18 11:38:42.023: ATM(): IP multicast cache invalidated for ATM1/0.1

Jun 18 11:38:42.023: ATM: PVC removed, ATM1/0.1 VCD 1 (15/72)

Jun 18 11:38:44.019: %LINK-3-UPDOWN: Interface ATM1/0, changed state to down

Jun 18 11:38:44.019: atmoc3_cstate(ATM1/0): state=0 atm_lineup=1

Jun 18 11:38:44.019: Resetting ATM1/0

Jun 18 11:38:44.055: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 69.183.x.173 (ATM1/0.1) is down: interface down

Jun 18 11:38:45.019: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM1/0, changed state to down

I notice the statement that says "pvc removed" but nothing to indicate why and this is followed by the ATM line protocol going down. This could be a circuit problem (unlikely) or could be a compatibility issue.

Telco is currently testing and we will try placing this ATM module into the original 3825 router tomorrow morning to see what happens.

Thanks,

Steve

Giuseppe Larosa · ‎06-18-2009

Hello Steve,

first of all checking all cabling and performing some loop tests is something you have done I suppose

interface stats doesn't show great input errors figures and just two interface resets.

both the SW advisor and feature navigator confirm that the NM should be supported in your release but I would give a try to another IOS image to see if the behavior changes.

for example

12.4(24)T

c2800nm-adventerprisek9-mz.12.4-24.T

or depending on what the router does I would give a try to a service provider feature set image.

Hope to help

Giuseppe

stephenshaw · ‎06-18-2009

Thanks,

certainly am thinking the IOS code and compatibility may be at issue. As mentioned, we will try placing this ATM module into the original router (3825) and see if this does or does not solve the problems we've experienced. Unfortunately, the original (known working) module was couriered to a different office and am forced to use a "spare" module (that was working prior to shipping).

I would have to load any IOS code remotely so need to have the Telco finish their tests before doing that.

Thanks for your input.

Steve

stephenshaw · ‎06-19-2009

Hi,

as a follow-up, Telco testing showed both internal wiring and the actual circuit were error free. We placed the ATM module into the 3825 router and re-installed the 3825 onto the circuit - all of our problems have disappeared. From this, it's obvious that the actual ATM module is OK and most likely the IOS code on the 2811 would be the suspect.

Thanks to all that provided input on this.