PGW2200: Collecting System log files and Call traces

Muthurani Lavanya Paneerselvam · ‎02-28-2012

Introduction
PGW2200: Collecting System related Information
Handling Alarms in PGW2200
System Log files
PGW2200 Call Trace
Related Information
Related discussion in CSC

Introduction

This document covers the Technotes for collecting the PGW2200 System related information, handling Alarms in PGW2200, collecting system log files and Call Traces. The information in this document will help the users to troubleshoot the PGW2200 system and network related issues easily.

PGW2200: Collecting System related Information

Collecting PGW ground information

Following details to be collected to check the PGW Platform Status

Unix version (uname –a)
Memory (prtconf | grep Memory , uname)
Processor details (psrinfo –v)
Free disk space (df –k)
Interface statistics (ifconfig -a, netstat – irn)
Patch information (pkginfo | grep CSCO)

Collecting MGC Information

Following details to be collected to check the PGW MGC (Media Gateway Controller) Status

System Alarms
System logs
Call traces
MML queries

PGW Health Status Monitoring

To check the PGW, Sun Netra Uptime

# uptime
4:21pm up 221 day(s), 11:52, 2 users, load average: 0.11, 0.10, 0.10

Check the PGW Health Status issue the command

Mml > Rtrv-ne-heath

Shows the Health Status of PGW about

• Platform State : Active / Standby

• Machine Congestion Level

• Current Calls in Progress

• Call Attempts in CPS (ie Calls Per Second)

• CPU Utilization

• Memory & Disk Utilization

Based on the Above Information, one can determine the Running Status of PGW and the CPS.

Check the PGW Platform State issue the command

Mml> rtrv – ne

Shows the Following details about the PGW

Hardware Platform
Vendor
MGC Application Version
Platform State

Check the Current Running Processes on PGW Sun Server Box.

Check the Processes on PGW Sun Server Box

# ps –ef

UID PID PPID C STIME TTY TIME CMD

root 0 0 0 Jan 30 ? 0:11 sched

root 1 0 0 Jan 30 ? 1:11 /sbin/init

root 2 0 0 Jan 30 ? 0:05 pageout

root 3 0 1 Jan 30 ? 2149:08 fsflush

root 229 7 0 Jan 30 ? 0:07 /usr/lib/saf/sac -t 300

root 7 1 0 Jan 30 ? 8:10 /lib/svc/bin/svc.startd

root 1108 1 0 Jan 30 ? 25:09 /opt/CiscoMGC/snmp/mib2agt -d

root 210 1 0 Jan 30 ? 0:02 /usr/sbin/cron

daemon 217 1 0 Jan 30 ? 0:00 /usr/lib/nfs/statd

root 1068 1 0 Jun 11 ? 0:57 /lib/svc/bin/svc.configd

root 118 1 0 Jan 30 ? 17:30 /usr/lib/picl/picld

root 130 1 0 Jan 30 ? 49:21 /usr/sbin/nscd

daemon 115 1 0 Jan 30 ? 0:42 /usr/lib/crypto/kcfd

root 132 1 0 Jan 30 ? 0:00 devfsadmd

root 126 1 0 Jan 30 ? 0:00 /usr/lib/sysevent/syseventd

root 233 229 0 Jan 30 ? 0:05 /usr/lib/saf/ttymon

daemon 214 1 0 Jan 30 ? 0:00 /usr/sbin/rpcbind

root 232 1 0 Jan 30 ? 1:26 /usr/lib/utmpd

root 234 1 0 Jan 30 ? 0:22 /usr/lib/inet/inetd start

root 296 1 0 Jan 30 ? 2:37 /usr/lib/fm/fmd/fmd

root 6110 6103 0 May 27 ? 114:15 /opt/CiscoMGC/snmp/critagt -d

root 261 1 0 Jan 30 ? 0:04 /usr/lib/ssh/sshd

root 255 1 0 Jan 30 ? 4:04 /usr/sbin/syslogd

root 1109 1 0 Jan 30 ? 8:30 /opt/CiscoMGC/snmp/hostagt -d

root 297 1 0 Jan 30 ? 0:00 /usr/sbin/mdmonitord

root 335 334 0 Jan 30 ? 75:09 /opt/TimesTen/tt60/bin/timest

ensubd -verbose -id 0 -facility user

root 334 1 0 Jan 30 ? 100:45 /opt/TimesTen/tt60/bin/timest

end -initfd 13

root 336 334 0 Jan 30 ? 18:12 /opt/TimesTen/tt60/bin/timest

Check the CPU Utilization for Each Process

Checking the CPU Utilization for each Process

# prstat

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP

13989 mgcusr 1324M 804M sleep 59 -19 3:07:36 0.7% engine.smartall/5

13997 mgcusr 55M 18M sleep 59 0 1:03:48 0.2% ioChanMgr/1

13988 mgcusr 70M 66M sleep 59 0 0:53:20 0.2% replicator/4

14001 mgcusr 32M 13M sleep 59 0 0:53:07 0.2% ISDNIP/1

14002 mgcusr 32M 13M sleep 59 0 0:47:45 0.1% ISDNIP/1

13993 mgcusr 153M 138M sleep 59 0 0:47:06 0.1% pom/5

13979 mgcusr 11M 7792K sleep 59 0 1:10:44 0.1% diskmonitor/2

13982 mgcusr 45M 25M sleep 59 0 0:16:02 0.1% measMgr/1

14003 mgcusr 101M 97M sleep 59 0 0:30:59 0.1% SS7/1

14008 mgcusr 97M 93M sleep 59 0 0:29:58 0.1% SS7/1

14004 mgcusr 32M 13M sleep 59 0 0:28:33 0.1% ISDNIP/1

14005 mgcusr 97M 93M sleep 59 0 0:25:24 0.1% SS7/1

14007 mgcusr 32M 13M sleep 59 0 0:24:13 0.1% ISDNIP/1

14009 mgcusr 97M 93M sleep 59 0 0:24:51 0.1% SS7/1

28729 root 4872K 4632K cpu1 59 0 0:00:00 0.1% prstat/1

130 root 3864K 2064K sleep 59 0 0:49:20 0.0% nscd/25

28702 root 8096K 2616K sleep 59 0 0:00:00 0.0% sshd/1

18762 mgcusr 13M 6808K sleep 59 0 0:07:26 0.0% procM/1

13986 mgcusr 43M 23M sleep 59 0 0:03:07 0.0% mmSAgt/1

2590 root 2304K 824K sleep 100 - 0:17:33 0.0% xntpd/1

13983 mgcusr 12M 8264K sleep 59 0 0:00:03 0.0% cfgM/1

28699 root 8464K 4992K sleep 59 0 0:00:00 0.0% sshd/1

18260 mgcusr 4440K 2776K sleep 59 0 0:01:27 0.0% lmgrd/1

338 root 11M 4376K sleep 59 0 0:18:09 0.0% timestensubd/5

336 root 11M 4376K sleep 59 0 0:18:11 0.0% timestensubd/5

297 root 3488K 32K sleep 59 0 0:00:00 0.0% mdmonitord/1

1109 root 3896K 1664K sleep 59 0 0:08:30 0.0% hostagt/1

255 root 3744K 1728K sleep 59 0 0:04:04 0.0% syslogd/13

261 root 3944K 808K sleep 59 0 0:00:03 0.0% sshd/1

296 root 8496K 992K sleep 59 0 0:02:36 0.0% fmd/15

Total: 72 processes, 226 lwps, load averages: 0.10, 0.10, 0.09

#

Check the PGW Sun Server CPU Hardware Status

# psrinfo –v

Status of virtual processor 0 as of: 09/08/2009 16:44:53
on-line since 01/30/2009 04:28:39.
The sparcv9 processor operates at 1503 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 09/08/2009 16:44:53
on-line since 01/30/2009 04:28:38.
The sparcv9 processor operates at 1503 MHz,
and has a sparcv9 floating point processor.
#

Check the PGW Patch Information

# pkginfo | grep CSC

Check the PGW Network and Interface Configuration

This command shows the Bge Interface IP Address configuration

# ifconfig –a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 210.118.72.51 netmask fffffff8 broadcast 210.118.72.55
ether 0:14:4f:8f:c0:d2
bge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 210.118.72.59 netmask fffffff8 broadcast 210.118.72.63
ether 0:14:4f:8f:c0:d3
bge2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 210.118.75.51 netmask ffffffe0 broadcast 210.118.75.59
ether 0:14:4f:8f:c0:d4
#

Handling Alarms in PGW2200

Alarms are generated for Processes, Link sets, SPC, Signaling, Bearer links and the Service Status

The active alarm log file resides in /opt/CiscoMGC/var/log directory. This directory will have the following log files created for all the dates and have history of log files.

Platform_<date>.log
Diskmonitor_<date>.log
Mml_<date>.log

MML for Retrieving alarms from the PGW

rtrv-alms : To Retrieve Alarms
ack-alm : To acknowledge Alarms

Retrieving & Acknowledging alarms in PGW2200

Syntax:

-------

RTRV-ALMS

RTRV-ALMS::CONT

Command Description:

CONT -- Tells the system to display alarm events until Ctrl-C is pressed.

Alarm severity levels (SEV=) are as follows:

- Critical (CR)

- Major (MJ)

- Minor (MN)

Informational

How To Acknowledge Alarms ?

Examples:
---------
The MML command that follows is used in acknowledging a Config Fail alarm for the I/O channel manager process:

mml>ACK-ALM:IOCM-01:"Config Fail"
Media Gateway Controller - MGC-01 2000-01-12 15:19:51
M COMPLD
;

System Log files

The active system log file resides in /opt/CiscoMGC/var/log directory

Can also use UNIX commands to parse through the logs. Following System logs can be found in the directory with the history of logs

Platform_<date>.log
Diskmonitor_<date>.log
Mml_<date>.log

Sample log files from PGW2200

Platform_<date>.log file

(Contains Error on PGW2200)

Mon Aug 17 04:09:29:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538a,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

Mon Aug 17 04:14:39:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538b,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

Mon Aug 17 04:19:49:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538c,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

Mon Aug 17 04:24:59:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538d,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

Mon Aug 17 21:22:55:110 2009 IST | ss7-i-1 (PID 4585) <Error>

TIOS_ERR_RUDP_MAX_RETRANS: Connection 00000000052f9d70, maximum number (2) of retransmission exceeded

Mon Aug 17 21:22:55:110 2009 IST | ss7-i-1 (PID 4585) <Error>

TIOS_ERR_RUDP_MAX_RESYN: Connection 00000000052f9d70, maximum number (1) of soft resets exceeded

Mon Aug 17 21:22:55:110 2009 IST | ss7-i-1 (PID 4585) <Error>

480002, RUDP connection received SOFT_RESET signal

mml_<date>.log file

un Aug 16 04:10:00:901 2009 IST | mml11 (PID 10954) <Info>

user 'mgcusr' invoked mml11 session

Sun Aug 16 04:10:00:902 2009 IST | mml11 (PID 10954) <Info>

/tmp/almM_input: installed time handler, hdlrId = 7

Sun Aug 16 04:10:00:903 2009 IST | mml11 (PID 10954) <Info>

procEventHandler: process state transition beginning (3) -> (5)

Sun Aug 16 04:10:02:074 2009 IST | mml11 (PID 10954) <Info>

MML_INFO_COMMAND: MML Command

mml> sta-aud

MGC-01 - Media Gateway Controller 2009-08-16 04:10:02.073 IST

M COMPLD

"ENGG-01"

;

Sun Aug 16 04:10:02:581 2009 IST | mml11 (PID 10954) <Info>

user 'mgcusr' quit mml11 session

Sun Aug 16 07:52:28:720 2009 IST | mml2 (PID 10726) <Info>

MML_INFO_COMMAND: MML Command

mml> rtrv-c7lnk:all:

MGC-01 - Media Gateway Controller 2009-08-16 07:52:28.680 IST

M RTRV

"c7ip-newyrlink-ny1:ls-sprint-ny1,LID=0:IS"

"c7ip-newyrlink-ny2:ls-sprint-ny2,LID=0:IS"

"c7ip-docotst:lnknset-test,LID=0:IS"

;

diskmonitor_<date>.log file

Tue Sep 1 17:37:52:514 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_DISK_CLEAN: Removed 75 files from /opt/CiscoMGC/etc/CONFIG_LIB/CFG_testdpc

Tue Sep 1 17:37:52:515 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_REMOVE_CONFIG_DIR: Removed config dir /opt/CiscoMGC/etc/CONFIG_LIB/CFG_testdpc

Tue Sep 1 17:50:19:576 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_DISK_CLEAN: Removed 75 files from /opt/CiscoMGC/etc/CONFIG_LIB/CFG_HYDE1CONF

Tue Sep 1 17:50:19:576 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_REMOVE_CONFIG_DIR: Removed config dir /opt/CiscoMGC/etc/CONFIG_LIB/CFG_HYDE1CONF

Viewing logs

Viewing logs

1. Go to the log directory

cd /opt/MGC/var/log

2. View your specific file by using cat

inswpgw02% cat mml_20040310040538.log | more

Tue Mar 9 04:10:00:474 2004 IST | mml11 (PID 9395) <Info>

/tmp/almM_input: installed time handler, hdlrId = 7

Tue Mar 9 04:10:00:476 2004 IST | mml11 (PID 9395) <Info>

procEventHandler: process state transition beginning (3) -> (5)

Tue Mar 9 04:10:00:476 2004 IST | mml11 (PID 9395) <Info>

procEventHandler: process state transition ending (3) -> (5)

Tue Mar 9 04:10:00:983 2004 IST | mml11 (PID 9395) <Info>

Unable to process event

Tue Mar 9 04:10:01:000 2004 IST | mml11 (PID 9395) <Info>

MML_INFO_COMMAND: MML Command

Tue Mar 9 04:10:01:510 2004 IST | mml11 (PID 9395) <Info>

Unable to process event

Exporting Configuration files

Header 1

Exports current configuration of the MGC (Media Gateway Controller)

prov-exp:<component/target>:dirname=<directory name>

Eg. Prov-exp:config:dirname=“data070909”

Syntax description:

Target/Component -- all -- All Configuration Data

Component /Targent Name can be one of the following:

config : Core Configuration Data

routing : Routing Data

numan : Number Analysis Data

trkgrp : Trunk Group Data

trunk : Trunk Data

all : All Configuration Data

Export directory name can be any directory name, in double quotes, which will be created under the cust_specific directory.

PGW2200 Call Trace

A trace is a record of the SS7 ISUP Message flow through the PGW2200 engine.

If a trace is empty, the call hasn’t reached the engine!
Start trace with sig-path details from where the call originates
Start a trace with the following MML command:

STA-SC-TRC:[sig-path]:log=“[name]”,CONFIRM

Stop a trace with the following MML command:

STP-SC-TRC:ALL

How to Enable the Call Trace?

How to enable the PGW2200 Call Trace ?

How To Enable the Call TRACE ?

swpgw1 mml> sta-sc-trc:

Starts a protocol conversion trace on a signal path. The user typically performs a trace when calls are failing, and the engine is the suspected source of the problem.

Examples:

--------------

This command will initiate the Signaling trace on the Specified Signaling Path, the filename would be: <sig path>_yyyymmddhhmmss.btr.

This Trace log is stored in /opt/CiscoMGC/var/trace directory on the PGW

Mml> STA-SC-TRC: < SIGPATH>:CONFIRM

How To Stop the Call TRACE ?

stp-sc-trc:<sig path>

stp-sc-trc:all

Purpose: To Stop One or All Traces

Examples:

---------------

The MML command shown in the following example stops all active capture sessions:

Mml> STP-SC-TRC:all

How to Collect the PGW2200 MDL Trace?

Collecting PGW2200 MDL Trace

How to Collect the PGW2200 MDL Trace ?

Use this procedure in order to collect an MDL trace via the MML command STA−SC−TRC (Start Trace).

Step1. Identify the Originating SS7 SigPath Number or the Originating TrunkGroup Number on which calls are placed.

Step2. Start the MDL trace:

mml>sta−sc−trc:<ss7sigPath name >:CONFIRM

Step3. Perform a test, make few test calls via the SS7 Sigpath.

Step4. Stop the MDL trace:

mml>stp−sc−trc:all

Step5.Identify the Call Id (C:) of the bad call

Note: These files can contain tracings from many calls that are all mixed up together if the capture is

taken on a production PGW. Each tracing record in the file has a specific record type and records

information of a type that relates to that record. Each record has a Call ID that relates it to a specific

call.

Step6: Convert the MDL trace into a readable format:

Go to the /opt/CiscoMGC/var/trace directory.

-bash-3.00# cd /opt/CiscoMGC/var/trace

-bash-3.00# ls

_nasp-norte-ny3_20081212123706.btr

_nasp-norte-ny3_20090604122453.btr

_nasp-norte-ny3_20090604122453_btr_12958689.trc

_nasp-norte-ny3_20090604122453_btr_12958691.trc

_ss7p-norte-ny1_20081221170223.prt

_ss7p-norte-ny1_20081224003732.btr

_ss7p-norte-ny1_20081224003732_btr_4932521.trc

_ss7p-norte-ny1_20081224022732.btr

_ss7p-norte-ny1_20081224022732_btr_4956105.trc

_ss7p-norte-ny2_20090206094100.btr

_ss7p-norte-ny2_20090206094100_btr_10598642.trc

cicss7trace_ss7p-norte-ny1_20081221182221.btr

cictrace_nasp-norte-ny1_20081221180123.btr

finaltrace_ss7p-norte-ny1_20081221184228.btr

naspath2_nasp-norte-ny2_20081221181939.btr

nortenumber_nasp-norte-ny3_20081212124935.btr

ss7p-norte-ny1_20081221170223.prt.gz

trace.tar

Step7: Run this command in mgcusr Login mode.

get_trc.sh <trace file name. btr>

Eg. Get_trc.sh nortenumber_nasp-norte-ny3_20081212124935.btr

For example:

/opt/CiscoMGC/var/trace

swpgw2% get_trc.sh nortenumber_nasp-norte-ny3_20081212124935.btr

get_trc.sh ca/sim/sp Trace File Utility Mistral Version 1.2

The ANALYSIS mdo file is: GENERIC_ANALYSIS.mdo

Retrieving _ss7path_20040116103221.btr trace file Call ID's, please wait...

Enter one of the following commands:

S = Simprint in less

F = Simprint with printing of sent and received Fields in less

D = Display trc trace in less

G = Display trc trace in less (Generated)

C = Convert to trc trace file

A = Display CA file in less

N = Move to Next call ID

P = Move to Previous call ID

L = List call ID's in current file

X = Set SP flags

H = Print Help

Q = Quit get_trc.sh

Or just enter the ID of the call you want if you know it

Use (N)ext and (P)revious to move between the call ID's

_ss7path_20040116103221.btr contains 1 call(s)

==> Working on call 1 ID 23 H = Help [S/F/D/G/C/A/N/P/L/H/Q/id]?

Step8: This get_trc.sh <filename.btr> command gives the complete trace of the SS7 Message.

Step9: Choose option C in order to convert the trace file.

Note: The .btr files are binary trace files that are produced by the PGW tracer function. The main part

of the file name is given in the VSC MML command sta−sc−trc. The PGW always adds a .btr

extension to these files. By using the C option, the file is converted into a text format and the

extension has .trc files that are text trace files. They contain detailed line by line trace information

from the MDO code that is run in the simulation replay that produces the file. Therefore, they contain

MDL traces.

Step10: The trace file is in /opt/CiscoMGC/var/trace.

Step11: Choose Option F inorder to print the SS7 Signaling Message on the Screen.

Step12: Choose Option Q inorder to quit.

Related Information

PGW2200 Softswitch Troubleshooting guide