I have been researching the procedure to replace the ATM Interface module in a Cisco 3660, but I am not finding much. Are there any steps I need to perform or any tips I need to keep in mind when replacing an ATM interface module? I have never replaced one, and I wanted to make sure it was done correctly. Is it as simple as shutting down the router and swapping the modules? They are the same direct replacement module, 1A-T3 cards. Any suggestions would be greatly appreciated.
So, I just need to unplug the ATM cables, remove the ATM module, insert the new module, re-insert the ATM cables, and I am good to go since the router supports online insertion and removal (OIR)? There shouldn't be any problems with me doing this?
To be honest, I am not 100% sure it is a faulty module. The problem is I have to keep bouncing the interface
daily to re-establish connectivity. I posted here before and it was mentioned to try updating the IOS which I am going to do as well, but I do not currently have enough flash memory to support the new versions. I have 16mb and need a minimum of 32mb. I asked for troubleshooting help but all that was mentioned was to update the IOS first and see. I have ordered the memory but I am needing to try something quickly as I am getting hounded by management about the issue. The logs show the interface bouncing from up -> down all day long. It goes down for like 2 minutes at a time then comes back up. Any other suggestions?
More likely uou have a faulty circuit.
I would wait for the memory upgrade, as it's very unlikely that you have a faulty module.
Do you think it is possibly on AT&T's end then? The problem started about a month or so ago briefly. No changes have been made to the frame network at all in this time period. The problem is inconsistent as far as time of day, but the interface status changes from up/down all day long several times a day.
Pick a time (say on the hour) log into the router and do a clear counters.
any time 2 to 24 hours after that, log back on the router and do a show interface for the problem circuit.
There will be a few clues in there.
Couple other items:
1. How long has the circuit been installed ?
2. When did the issues start ? is there any change associated with the start of the issues ?
The circuit has been installed for many years. I am new to the company having only been employed there 6 months. The issues started briefly about a month ago, but it has seemed to have gotten progressively worse. Almost daily, I have to go in and do a shut/no shut on the ATM interface to re-establish connectivity. There have been no config/network changes to our frame network in the 6 months I have been there.
Ok, so lets get some time stamped show int counters from the interface
as i suggested. You should also consider some testing with the carrier
during a maintenance window. they need to stress test the entire link.
I am going to do as you suggested for the counters and such. I also opened a ticket to have AT&T do a monitoring session to see if they see anything on their end. If that doesn't prove useful, then I will have them do an intrusive test during our maintenance window. I will let you know what the status of the counters are in the next day or so. Thanks.
Its a plan, which is better than no plan.
Post the results of the show interface after your time period
along with the show controllers for the same interface.
I cleared the counters and let it run for 4+ hours. I didn't have any link failures in that 4 hour time span, but here are the results of the items you
requested and maybe you can shed some light on a potential problem(s).
Show int atm2/0:
ATM2/0 is up, line protocol is up
Hardware is RS8234 ATM DS3
MTU 4470 bytes, sub MTU 4470, BW 45000 Kbit, DLY 190 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ATM, loopback not set
1023 maximum active VCs, 9 current VCCs
VC idle disconnect time: 300 seconds
Last input 00:00:17, output 00:02:54, output hang never
Last clearing of "show interface" counters 04:10:30
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Per VC Queueing
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1775 packets input, 184510 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2131 packets output, 2835164 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
Interface ATM2/0 is up
Hardware is RS8234 ATM DS3
LANE client MAC address is 0006.xxxx.xxxx
RS8234 base 3E000000, ds 6203BEE8, PM5346 base 3E400000, slave base 3E400000
SBDs - avail 2048, guaranteed 9, unguaranteed 2039, starved 0 rbds 4098
Seg VCC table 3E00DD80, Shadow Seg VCC Table 62066974, VCD Table 6207C9A0
Schedule table 3E018D80, Shadow Schedule table 620829CC, Size FC1
RSM VCC Table 3E14F180, Shadow RSM VCC Table 6218A380
VPI Index Table 3E04C700, VCI Index Table 3E04F180
Bucket2 Table 3E02C900, Shadow Bucket2 Table 620868FC
MCR Limit Table 3E02CD00, Shadow MCR Table 62088528
ABR template 3E02CF00, Shadow template 61B47160
RM Cell RS Queue 3E04D180
RBD OIR array 6203EC5C, HBD OIR array 6203CC5C
LECID Table 3E04CF00, Host LECID Table 62043628, LECID Table Bitmask 0
Queue TXQ Addr Pos StQ Addr Pos
0 UBR 3E048F00 214 07819820 214
1 UBR PLUS 3E049300 0 0781A020 0
2 UBR TUN1 3E049700 0 0781A820 0
3 UBR TUN2 3E049B00 0 0781B020 0
4 UBR TUN3 3E049F00 0 0781B820 0
5 ABR 3E04A300 0 0781C020 0
6 VBR NRT 3E04A700 0 0781C820 0
7 VBR RT 3E04AB00 0 0781D020 0
8 VBR/ABR TUN1 3E04AF00 0 0781D820 0
9 VBR/ABR TUN2 3E04B300 0 0781E020 0
10 VBR/ABR TUN3 3E04B700 0 0781E820 0
11 SIG 3E04BB00 0 0781F020 0
12 CBR 3E04BF00 0 0781F820 0
13 VPD 3E04C300 0 07820020 0
Queue FBQ Addr Pos RSQ Addr Pos
0 OAM 3E15B180 161 078208E0 162
1 UBR 3E15C180 95 078218E0 235
2 UBR PLUS 3E15D180 0 078228E0 0
3 UBR TUN1 3E15E180 0 078238E0 0
4 UBR TUN2 3E15F180 0 078248E0 0
5 UBR TUN3 3E160180 0 078258E0 0
6 ABR 3E161180 0 078268E0 0
7 VBR NRT 3E162180 0 078278E0 0
8 VBR RT 3E163180 0 078288E0 0
9 VBR/ABR TUN1 3E164180 0 078298E0 0
10 VBR/ABR TUN2 3E165180 0 0782A8E0 0
11 VBR/ABR TUN3 3E166180 0 0782B8E0 0
12 SIG 3E167180 0 0782C8E0 0
13 CBR 3E168180 0 0782D8E0 0
PCI bus err 0, DMA fifo full err 0, RSM parity err 0
RSM sync err 0, RSM/SEG Q full err 0, RSM overflow err 0
RSM stat Q full err 0, no free buff Q err 0, SEG underflow err 0
SEG stat Q full err 0
Framer Chip Type PM7345
Framer Chip ID 0x20
Framer State RUNNING
Defect Status NO ERRORS
Loopback Mode NONE
Clock Source INTERNAL (but the source of this clock is derived from LINE)
DS3 Scrambling ON
Framing DS3 C-bit w/PLCP framing
TX cells 1640139
TX bytes 86927367
Last output time 1y11w
RX cells 404467
RX bytes 21436751
Last input time 1y11w
Line Code Violations (LCV) 0
DS3: F/M-bit errors 0
DS3: parity errors 0
DS3: path parity errors 0
DS3/E3: G.832 FEBE errors 3432
T3/E3: excessive zeros 0
PLCP BIP errors 0
PLCP framing octet errors 0
PLCP FEBE errors 1766
uncorrectable HEC errors 0
idle/unassigned cells dropped 0
LCV errored secs 0
DS3: F/M-bit errored secs 0
DS3: parity errored secs 0
DS3: path parity errored secs 0
T3/E3: excessive zeros errored secs 0
DS3/E3: G.832 FEBE errored secs 62
PLCP BIP errored secs 0
PLCP framing octet errored secs 0
PLCP FEBE errored secs 38
uncorrectable HEC errored secs 0
LCV error-free secs 175680
DS3: F/M-bit error-free secs 175680
DS3: parity error-free secs 175681
DS3: path parity error-free secs 175695
T3/E3: excessive zeros error-free secs 175695
DS3/E3: G.832 FEBE error-free secs 175633
PLCP BIP error-free secs 175695
PLCP framing octet error-free secs 175695
PLCP FEBE error-free secs 175657
uncorrectable HEC error-free secs 175695
PM 7345 registers (base 0x3E400000):
cfgr 0x00, ier 0x08, isr 0x00, ctlr
imrr 0x21, dlcr 0x78, rboc_cier 0x00, rboc_isr
t3frmr_cfgr 0x01, t3frmr_ier 0x03, t3frmr_isr 0x00, t3frmr_statr
rfdl_cfgr 0xFC, rfdl_esr 0xF8, rfdl_statr 0x87, rfdl_datar
pmon_pmr 0x00, pmon_iesr 0x03, pmon_lcvec0r 0x00, pmon_lcvec1r
pmon_fbeec0r 0x00, pmon_fbeec1r 0x00, pmon_sezdc0r 0x00, pmon_sezdc1r
pmon_peec0r 0x00, pmon_peec1r 0x00, pmon_ppeec0r 0x00, pmon_ppeec1r
pmon_febeec0r 0x00, pmon_febeec1r 0x00, t3tran_cfgr 0x01, t3tran_diagr
xfdl_cfgr 0x00, xfdl_isr 0x02, xfdl_txdatar 0x00, xboc_coder
splr_cfgr 0x04, splr_ier 0x00, splr_isr 0x7F, splr_statr
splt_cfgr 0x0C, splt_ctlr 0x00, splt_diagr 0x00, splt_f1r
cppm_locmr 0xFC, cppm_copmr 0x90, cppm_b1ec0r 0x00, cppm_b1ec1r
cppm_feec0r 0x00, cppm_feec1r 0x00, cppm_febec0r 0x00, cppm_febec1r
cppm_hcsec0r 0x00, cppm_hcsec1r 0x00, cppm_iucc0r 0xFF, cppm_iucc1r
cppm_rcc0r 0x00, cppm_rcc1r 0x00, cppm_tcc0r 0x00, cppm_tcc1r
rxcp_ctlr 0x2C, rxcp_frcr 0x00, rxcp_iesr 0x1C, rxcp_iucph1r
rxcp_iucph2r 0x00, rxcp_iucph3r 0x00, rxcp_iucph4r 0x01, rxcp_iucmh1r
rxcp_iucmh2r 0xFF, rxcp_iucmh3r 0xFF, rxcp_iucmh4r 0xFF, rxcp_upcph1r
rxcp_upcph2r 0x00, rxcp_upcph3r 0x00, rxcp_upcph4r 0x00, rxcp_upcmh1r
rxcp_upcmh2r 0xFF, rxcp_upcmh3r 0xFF, rxcp_upcmh4r 0xFF, rxcp_hcscsr
rxcp_lctctr 0xB4, txcp_ctlr 0xA4, txcp_iesr 0x18, txcp_iucph1r
txcp_iucph2r 0x00, txcp_iucph3r 0x00, txcp_iucph4r 0x01, txcp_iucph5r
txcp_iucpr 0x00, e3frmr_foptr 0x00, e3frmr_moptr 0x00, e3frmr_fier
e3frmr_fiisr 0x01, e3frmr_meier 0x00, e3frmr_meiir 0x00, e3frmr_mesr
e3tran_foptr 0x00, e3tran_sdoptr 0x03, e3tran_bip8emr 0x00, e3tran_maoptr
ttb_ctlr 0x04, ttb_ttisr 0x00, ttb_iar 0x00, ttb_idr
ttb_eptlr 0x00, ttb_ptlcsr 0x00, sffpcsr 0x31, pcr
Unfortunately not all problems leave a trace in show interface, and I am afraid your is one of these that do not.
I do notice some FEBE error counts.
Little guide on Troubleshooting ATM issues, you 'Might" have an issue at the far end.
Thanks vmiller. I submitted a trouble ticket to AT&T to do further testing on their end to rule out that possibility. I definitely see the FEBE errors and I will take a look at that guide as I am sure that will help me in the future. Needless to say, the problem hasn't happened in the last few days.