Re: IGX UXM card resets itself constantly

Dapp · ‎11-11-2003

Hello,

We have 2 IGX 8410 interconnected via an E1. It carries frame relay PVCs and interconnects 2 PBX for radio communication all within the same E1. Both are part of a national data/voice network of about 30 IGX.

We've been working for about 2 years without any major problems, but the last 2 weeks the UXM card in one of the sites resets itself, once everyday, and it's been twice today. Our service provider says the E1 is fine, they've made physical tests to it without reporting problems (we implement frame relay over a clear E1).

When the card resets itself it takes the trunk down for aprox. 30 seconds, but the connections made over the trunk take about 30 minutes to get up.

This is the log of the IGX with the problem card:

Clear Comm Break with mxlc4 Cleared 11/10/03 16:57:37

Info Clock switch to CLN of puma via TRK 8.1:1 11/10/03 16:57:31

Clear TRK 8.1 OK 11/10/03 16:57:00

Major TRK 8.1 Communication Failure 11/10/03 16:56:47

Clear PHYSLN 8.1 OK 11/10/03 16:56:47

Clear E1-IMA 8 Inserted - Activated 11/10/03 16:56:47

Major TRK 8.1 Back Card Missing 11/10/03 16:56:46

Major PHYSLN 8.1 Back Card Missing 11/10/03 16:56:46

Info UXM 8 Activated 11/10/03 16:56:46

Info UXM 8 Inserted 11/10/03 16:56:46

Info E1-IMA 8 Removed 11/10/03 16:56:35

Clear Failed UXM 8 Removed 11/10/03 16:56:35

Major TRK 8.1 Front Card Missing 11/10/03 16:56:20

Major PHYSLN 8.1 Front Card Missing 11/10/03 16:56:20

Major UXM 8 Not Responding - No backup available 11/10/03 16:56:20

Info Clock switch to oscillator of SCC 11/10/03 16:54:32

Minor Comm Break with mxlc4 11/10/03 16:54:32

Major TRK 8.1 Communication Failure 11/10/03 16:54:31

The trunk and the line are configured the same in each IGX:

mxlc4 TN c4mxl:2 IGX 8410 9.1.19 Nov. 11 2003 08:41 PST

TRK 8.2 Config E1/32 [4830 cps] UXM slot: 8

Line DS-0 map: 0-31 HCS Masking: Yes

Transmit Trunk Rate: 4830 cps Payload Scramble: Yes

Rcv Trunk Rate: 4830 cps Connection Channels: 256

Pass sync: Yes Gateway Channels: 200

Loop clock: No Valid Traffic Classes:

Statistical Reserve: 200 cps V,TS,NTS,FR,FST,CBR,VBR,ABR

Header Type: NNI Deroute delay time: 0 seconds

VPI Address: 1 VPC Conns disabled: No

Routing Cost: 10

Idle code: 54 hex

Restrict PCC traffic: No

Link type: Terrestrial

Line coding: HDB3

Line recv impedance: 75 ohm

This Command: cnftrk 8.2

mxlc4 TN c4mxl:2 IGX 8410 9.1.19 Nov. 11 2003 08:42 PST

LN 3.2 Config E1/31 UVM slot: 3

Loop clock: No

Line framing: On

Line coding: HDB3

Line CRC: No

Line recv impedance: 75 ohm + gnd

Line E1/J1 signal: CCS

Line encoding: A-LAW

Line 56KBS Bit Pos: msb

Line pct fast modem: 20

Line cnfg: External

Line cnf slot.line: --

Line CAS-Switching: Off

Line SVC-Caching: Off

Traffic Shaping: No

This Command: cnfln 3.2

Months ago we reduced the statistical reserve on the trunk in each side (from 200 to 600), that apparently didn't cause any problems, and this problem is very recent (2 weeks).

Please help, thanks.

Daniel.

dmirlenbrink · ‎11-12-2003

Hi Daniel,

have a look at

dspcderrs 3

dspcd 3

May you see some more details like hw errors.

That's not an E1 line problem. It's the UXM card 8.

I would replace the card.

regards

Dietmar

smoosa · ‎11-12-2003

Daniel,

Please include the dspcd and dspcderrs from this card. This may be a hardware issue or could be one of the software defects on the UXM firmware in the earlier releases ( Pre model B rev U). Once we see the dspcderrs and also the firmware running on the card, we can make a determination. Please also include the dspswlog from the node.

Thanks

Dapp · ‎11-14-2003

Hello,

My user doesn't have privileges for the dspswlog command (I'll ask this to the central network admin), meanwhile here's the output for the rest:

======================================

tijc4 VT hhernandez:0 IGX 8410 9.1.19 Nov. 14 2003 09:01 PST

Slot Failure

Number Records

------ --------

0 None

1 None

2 None

3 None

4 None

5 None

6 6

7 2

8 25

Last Command: dspcderrs

======================================

tijc4 VT hhernandez:0 IGX 8410 9.1.19 Nov. 14 2003 09:02 PST

UXM in Slot 8 : 322207 Rev BFE Failures Cleared: Oct. 23 2001 15:45:52 PST

----------------------------------- Records Cleared: Date/Time Not Set

Self Test Threshold Counter: 0 Threshold Limit: 300

Total Pass: 0 Total Fail: 0 Total Abort: 14

First Pass: Last Pass:

First Fail: Last Fail:

Background Test Threshold Counter: 0 Threshold Limit: 300

Total Pass: 228259 Total Fail: 0 Total Abort: 26

First Pass: Date/Time Not Set Last Pass: Nov. 14 2003 08:58:17 PST

First Fail: Last Fail:

Hardware Error Total Events: 25 Threshold Counter: 10

First Event: Dec. 10 2002 23:14:28 PST Last Event: Nov. 14 2003 01:09:47 PST

UXM in Slot 8 : 322207 Rev BFE

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 14 2003 01:09:47 PST

Hardware Error Event:

0B 02 05 A9 01 00 00 00 08

Error Fcode: 02

Failure Type: Internal (Boot PROM)

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 14 2003 01:09:47 PST

Hardware Error Event:

0B 29 00 A9 01 0D 17 09 81

Error Fcode: 29

Failure Type: Unknown

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 23:12:47 PST

Hardware Error Event:

0B 29 00 A9 01 0D 17 09 82

Error Fcode: 29

Failure Type: Unknown

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 23:12:46 PST

Hardware Error Event:

0B 02 05 A9 01 00 00 00 08

Error Fcode: 02

Failure Type: Internal (Boot PROM)

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 23:12:46 PST

Hardware Error Event:

0B 29 00 A9 01 0D 17 09 C1

Error Fcode: 29

Failure Type: Unknown

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 08:11:16 PST

Hardware Error Event:

0B 29 00 A9 01 0C 05 80 21 38 C4 CB B8 6F C7 00 00 00 00 00 00 FF 00

Error Fcode: 29

Failure Type: Unknown

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 08:11:16 PST

Hardware Error Event:

0B 02 05 A9 01 00 00 00 08

Error Fcode: 02

Failure Type: Internal (Boot PROM)

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 08:11:16 PST

Hardware Error Event:

0B 29 00 A9 01 0D 17 09 88

Error Fcode: 29

Failure Type: Unknown

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 04:10:08 PST

Hardware Error Event:

0B 02 05 A9 01 00 00 00 08

Error Fcode: 02

Failure Type: Internal (Boot PROM)

Processor Number: 01

-----------------------------------

Failure Type: Hardware Error

Failure Time: Nov. 13 2003 04:10:07 PST

Hardware Error Event:

0B 29 00 A9 01 0D 17 09 81

Error Fcode: 29

Failure Type: Unknown

Processor Number: 01

This Command: dspcderrs 8

======================================

tijc4 VT hhernandez:0 IGX 8410 9.1.19 Nov. 14 2003 09:03 PST

Detailed Card Display for UVM in slot 3

Status: Active (Front Card Supports CAS-switching)

Revision: EMH (Front Card Supports td connection type)

Serial Number: 310397

Fab Number: 28-2075-01

Integrated Echo Canceller

Channels: 31

Backplane Installed

Backcard Installed

Type: E1-2

Revision: AB

Serial Number: 321488

Last Command: dspcd 3

======================================

tijc4 VT hhernandez:0 IGX 8410 9.1.19 Nov. 14 2003 09:05 PST

Detailed Card Display for UXM in slot 8

Status: Active (Front Card Supports SIW)

Revision: BFE (Front Card Supports Cell Forwarding)

Serial Number: 322207 (Front Card with GW installed)

Fab Number: 28-2164-02 (# of trunks that can be upped = 8)

Backplane Installed (Front Card Supports Hot Standby)

Backcard Installed (Front Card Supports Traffic Shaping)

Type: E1-IMA

Revision: AC

Serial Number: 275301

Ports: 8

Interface: BNC

Last Command: dspcd 8

======================================

So far we moved the trunk from port 1 to port 2 in the UXM, though I suspect too it is a problem of the whole card. What do you think?

Thanks again,

Daniel.

smoosa · ‎11-14-2003

Hi ,

From the dspcderrs that you have given here, it looks like you hit a known firmware bug. You are running firmware revision Model B revision E which does not have the fix for the bug.

The error string that you see :

Hardware Error Event:

0B 29 00 A9 01 0D 17 09 81

Matches DDTS CSCdw15969. : This causes the card to reset once it has been up for 319 days.

The fix for this problem is available in the model B revision R and later releases.

When upgrading the UXM's it would be a good idea to upgrade them to the model B revision U firmware which is the latest stable image. There have been a lot of bug fixes since the release you are running on the UXM cards right now.