Re: Replacing ATM Interface Module in 3660

Mike McWethy · ‎11-01-2010

I have been researching the procedure to replace the ATM Interface module in a Cisco 3660, but I am not finding much. Are there any steps I need to perform or any tips I need to keep in mind when replacing an ATM interface module? I have never replaced one, and I wanted to make sure it was done correctly. Is it as simple as shutting down the router and swapping the modules? They are the same direct replacement module, 1A-T3 cards. Any suggestions would be greatly appreciated.

Thanks,
Mike

paolo bevilacqua · ‎11-01-2010

3660 supports OIR, so you don't have to shutdown, just swap the module.

Mike McWethy · ‎11-01-2010

So, I just need to unplug the ATM cables, remove the ATM module, insert the new module, re-insert the ATM cables, and I am good to go since the router supports online insertion and removal (OIR)? There shouldn't be any problems with me doing this?

Thanks,

Mike

paolo bevilacqua · ‎11-01-2010

Are you sure you have a faulty module? If yes, why?

Mike McWethy · ‎11-01-2010

To be honest, I am not 100% sure it is a faulty module. The problem is I have to keep bouncing the interface

daily to re-establish connectivity. I posted here before and it was mentioned to try updating the IOS which I am going to do as well, but I do not currently have enough flash memory to support the new versions. I have 16mb and need a minimum of 32mb. I asked for troubleshooting help but all that was mentioned was to update the IOS first and see. I have ordered the memory but I am needing to try something quickly as I am getting hounded by management about the issue. The logs show the interface bouncing from up -> down all day long. It goes down for like 2 minutes at a time then comes back up. Any other suggestions?

paolo bevilacqua · ‎11-01-2010

More likely uou have a faulty circuit.

I would wait for the memory upgrade, as it's very unlikely that you have a faulty module.

Mike McWethy · ‎11-01-2010

Paolo,

Do you think it is possibly on AT&T's end then? The problem started about a month or so ago briefly. No changes have been made to the frame network at all in this time period. The problem is inconsistent as far as time of day, but the interface status changes from up/down all day long several times a day.

vmiller · ‎11-01-2010

Do this:

Pick a time (say on the hour) log into the router and do a clear counters.

any time 2 to 24 hours after that, log back on the router and do a show interface for the problem circuit.

There will be a few clues in there.

Couple other items:

1. How long has the circuit been installed ?

2. When did the issues start ? is there any change associated with the start of the issues ?

Mike McWethy · ‎11-01-2010

@vmiller,

The circuit has been installed for many years. I am new to the company having only been employed there 6 months. The issues started briefly about a month ago, but it has seemed to have gotten progressively worse. Almost daily, I have to go in and do a shut/no shut on the ATM interface to re-establish connectivity. There have been no config/network changes to our frame network in the 6 months I have been there.

Mike

vmiller · ‎11-02-2010

Ok, so lets get some time stamped show int counters from the interface

as i suggested. You should also consider some testing with the carrier

during a maintenance window. they need to stress test the entire link.

Mike McWethy · ‎11-02-2010

@vmiller,

I am going to do as you suggested for the counters and such. I also opened a ticket to have AT&T do a monitoring session to see if they see anything on their end. If that doesn't prove useful, then I will have them do an intrusive test during our maintenance window. I will let you know what the status of the counters are in the next day or so. Thanks.

Mike

vmiller · ‎11-02-2010

Its a plan, which is better than no plan.

Post the results of the show interface after your time period

along with the show controllers for the same interface.

Mike McWethy · ‎11-03-2010

@vmiller,

I cleared the counters and let it run for 4+ hours. I didn't have any link failures in that 4 hour time span, but here are the results of the items you

requested and maybe you can shed some light on a potential problem(s).

Show int atm2/0:

ATM2/0 is up, line protocol is up

Hardware is RS8234 ATM DS3

MTU 4470 bytes, sub MTU 4470, BW 45000 Kbit, DLY 190 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ATM, loopback not set

Encapsulation(s): AAL5

1023 maximum active VCs, 9 current VCCs

VC idle disconnect time: 300 seconds

Last input 00:00:17, output 00:02:54, output hang never

Last clearing of "show interface" counters 04:10:30

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: Per VC Queueing

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

1775 packets input, 184510 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

2131 packets output, 2835164 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

Show controllers:

Interface ATM2/0 is up

Hardware is RS8234 ATM DS3

LANE client MAC address is 0006.xxxx.xxxx

hwidb=0x6203A168, ds=0x6203BEE8

RS8234 base 3E000000, ds 6203BEE8, PM5346 base 3E400000, slave base 3E400000

SBDs - avail 2048, guaranteed 9, unguaranteed 2039, starved 0 rbds 4098

Seg VCC table 3E00DD80, Shadow Seg VCC Table 62066974, VCD Table 6207C9A0

Schedule table 3E018D80, Shadow Schedule table 620829CC, Size FC1

RSM VCC Table 3E14F180, Shadow RSM VCC Table 6218A380

VPI Index Table 3E04C700, VCI Index Table 3E04F180

Bucket2 Table 3E02C900, Shadow Bucket2 Table 620868FC

MCR Limit Table 3E02CD00, Shadow MCR Table 62088528

ABR template 3E02CF00, Shadow template 61B47160

RM Cell RS Queue 3E04D180

RBD OIR array 6203EC5C, HBD OIR array 6203CC5C

LECID Table 3E04CF00, Host LECID Table 62043628, LECID Table Bitmask 0

Queue TXQ Addr Pos StQ Addr Pos

0 UBR 3E048F00 214 07819820 214

1 UBR PLUS 3E049300 0 0781A020 0

2 UBR TUN1 3E049700 0 0781A820 0

3 UBR TUN2 3E049B00 0 0781B020 0

4 UBR TUN3 3E049F00 0 0781B820 0

5 ABR 3E04A300 0 0781C020 0

6 VBR NRT 3E04A700 0 0781C820 0

7 VBR RT 3E04AB00 0 0781D020 0

8 VBR/ABR TUN1 3E04AF00 0 0781D820 0

9 VBR/ABR TUN2 3E04B300 0 0781E020 0

10 VBR/ABR TUN3 3E04B700 0 0781E820 0

11 SIG 3E04BB00 0 0781F020 0

12 CBR 3E04BF00 0 0781F820 0

13 VPD 3E04C300 0 07820020 0

Queue FBQ Addr Pos RSQ Addr Pos

0 OAM 3E15B180 161 078208E0 162

1 UBR 3E15C180 95 078218E0 235

2 UBR PLUS 3E15D180 0 078228E0 0

3 UBR TUN1 3E15E180 0 078238E0 0

4 UBR TUN2 3E15F180 0 078248E0 0

5 UBR TUN3 3E160180 0 078258E0 0

6 ABR 3E161180 0 078268E0 0

7 VBR NRT 3E162180 0 078278E0 0

8 VBR RT 3E163180 0 078288E0 0

9 VBR/ABR TUN1 3E164180 0 078298E0 0

10 VBR/ABR TUN2 3E165180 0 0782A8E0 0

11 VBR/ABR TUN3 3E166180 0 0782B8E0 0

12 SIG 3E167180 0 0782C8E0 0

13 CBR 3E168180 0 0782D8E0 0

PCI bus err 0, DMA fifo full err 0, RSM parity err 0

RSM sync err 0, RSM/SEG Q full err 0, RSM overflow err 0

RSM stat Q full err 0, no free buff Q err 0, SEG underflow err 0

SEG stat Q full err 0

Framer Chip Type PM7345

Framer Chip ID 0x20

Framer State RUNNING

Defect Status NO ERRORS

Loopback Mode NONE

Clock Source INTERNAL (but the source of this clock is derived from LINE)

DS3 Scrambling ON

Framing DS3 C-bit w/PLCP framing

TX cells 1640139

TX bytes 86927367

Last output time 1y11w

RX cells 404467

RX bytes 21436751

Last input time 1y11w

Line Code Violations (LCV) 0

DS3: F/M-bit errors 0

DS3: parity errors 0

DS3: path parity errors 0

DS3/E3: G.832 FEBE errors 3432

T3/E3: excessive zeros 0

PLCP BIP errors 0

PLCP framing octet errors 0

PLCP FEBE errors 1766

uncorrectable HEC errors 0

idle/unassigned cells dropped 0

LCV errored secs 0

DS3: F/M-bit errored secs 0

DS3: parity errored secs 0

DS3: path parity errored secs 0

T3/E3: excessive zeros errored secs 0

DS3/E3: G.832 FEBE errored secs 62

PLCP BIP errored secs 0

PLCP framing octet errored secs 0

PLCP FEBE errored secs 38

uncorrectable HEC errored secs 0

LCV error-free secs 175680

DS3: F/M-bit error-free secs 175680

DS3: parity error-free secs 175681

DS3: path parity error-free secs 175695

T3/E3: excessive zeros error-free secs 175695

DS3/E3: G.832 FEBE error-free secs 175633

PLCP BIP error-free secs 175695

PLCP framing octet error-free secs 175695

PLCP FEBE error-free secs 175657

uncorrectable HEC error-free secs 175695

PM 7345 registers (base 0x3E400000):

cfgr 0x00, ier 0x08, isr 0x00, ctlr

0x00,

imrr 0x21, dlcr 0x78, rboc_cier 0x00, rboc_isr

0x3F,

t3frmr_cfgr 0x01, t3frmr_ier 0x03, t3frmr_isr 0x00, t3frmr_statr

0x20,

rfdl_cfgr 0xFC, rfdl_esr 0xF8, rfdl_statr 0x87, rfdl_datar

0x87,

pmon_pmr 0x00, pmon_iesr 0x03, pmon_lcvec0r 0x00, pmon_lcvec1r

0x00,

pmon_fbeec0r 0x00, pmon_fbeec1r 0x00, pmon_sezdc0r 0x00, pmon_sezdc1r

0x00,

pmon_peec0r 0x00, pmon_peec1r 0x00, pmon_ppeec0r 0x00, pmon_ppeec1r

0x00,

pmon_febeec0r 0x00, pmon_febeec1r 0x00, t3tran_cfgr 0x01, t3tran_diagr

0x00,

xfdl_cfgr 0x00, xfdl_isr 0x02, xfdl_txdatar 0x00, xboc_coder

0xFF,

splr_cfgr 0x04, splr_ier 0x00, splr_isr 0x7F, splr_statr

0x08,

splt_cfgr 0x0C, splt_ctlr 0x00, splt_diagr 0x00, splt_f1r

0x00,

cppm_locmr 0xFC, cppm_copmr 0x90, cppm_b1ec0r 0x00, cppm_b1ec1r

0x00,

cppm_feec0r 0x00, cppm_feec1r 0x00, cppm_febec0r 0x00, cppm_febec1r

0x00,

cppm_hcsec0r 0x00, cppm_hcsec1r 0x00, cppm_iucc0r 0xFF, cppm_iucc1r

0xFF,

cppm_rcc0r 0x00, cppm_rcc1r 0x00, cppm_tcc0r 0x00, cppm_tcc1r

0x00,

rxcp_ctlr 0x2C, rxcp_frcr 0x00, rxcp_iesr 0x1C, rxcp_iucph1r

0x00,

rxcp_iucph2r 0x00, rxcp_iucph3r 0x00, rxcp_iucph4r 0x01, rxcp_iucmh1r

0xFF,

rxcp_iucmh2r 0xFF, rxcp_iucmh3r 0xFF, rxcp_iucmh4r 0xFF, rxcp_upcph1r

0x00,

rxcp_upcph2r 0x00, rxcp_upcph3r 0x00, rxcp_upcph4r 0x00, rxcp_upcmh1r

0xFF,

rxcp_upcmh2r 0xFF, rxcp_upcmh3r 0xFF, rxcp_upcmh4r 0xFF, rxcp_hcscsr

0xFC,

rxcp_lctctr 0xB4, txcp_ctlr 0xA4, txcp_iesr 0x18, txcp_iucph1r

0x00,

txcp_iucph2r 0x00, txcp_iucph3r 0x00, txcp_iucph4r 0x01, txcp_iucph5r

0x52,

txcp_iucpr 0x00, e3frmr_foptr 0x00, e3frmr_moptr 0x00, e3frmr_fier

0x00,

e3frmr_fiisr 0x01, e3frmr_meier 0x00, e3frmr_meiir 0x00, e3frmr_mesr

0x00,

e3tran_foptr 0x00, e3tran_sdoptr 0x03, e3tran_bip8emr 0x00, e3tran_maoptr

0x00,

ttb_ctlr 0x04, ttb_ttisr 0x00, ttb_iar 0x00, ttb_idr

0x00,

ttb_eptlr 0x00, ttb_ptlcsr 0x00, sffpcsr 0x31, pcr

0x30,

paolo bevilacqua · ‎11-03-2010

Unfortunately not all problems leave a trace in show interface, and I am afraid your is one of these that do not.

vmiller · ‎11-04-2010

I do notice some FEBE error counts.

http://www.cisco.com/en/US/tech/tk39/tk48/technologies_tech_note09186a0080111093.shtml

Little guide on Troubleshooting ATM issues, you 'Might" have an issue at the far end.