Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

PGW2200: Collecting System log files and Call traces

 

 

Introduction

 

This document covers the Technotes for collecting the PGW2200 System related information, handling Alarms in PGW2200, collecting system log files and Call Traces. The information in this document will help the users to troubleshoot the PGW2200 system and network related issues easily.

 

PGW2200: Collecting System related Information

 

 

Collecting PGW ground information

 

Following details to be collected to check the PGW Platform Status

  1. Unix version (uname –a)
  2. Memory (prtconf | grep Memory , uname)
  3. Processor details (psrinfo –v)
  4. Free disk space (df –k)
  5. Interface statistics (ifconfig -a, netstat – irn)
  6. Patch information (pkginfo | grep CSCO)


Collecting MGC Information

Following details to be collected to check the PGW MGC (Media Gateway Controller) Status

  1. System Alarms
  2. System logs
  3. Call traces
  4. MML queries

 

PGW Health Status Monitoring

 

To check the PGW, Sun Netra Uptime

# uptime
4:21pm up 221 day(s), 11:52, 2 users, load average: 0.11, 0.10, 0.10

 

  • Check the PGW Health Status issue the command

 

Mml > Rtrv-ne-heath

Shows the Health Status of PGW about

 


• Platform State : Active / Standby

• Machine Congestion Level

• Current Calls in Progress

• Call Attempts in CPS (ie Calls Per Second)

• CPU Utilization

• Memory & Disk Utilization

 

Based on the Above Information, one can determine the Running Status of PGW and the CPS.

 

  • Check the PGW Platform State issue the command

 

Mml> rtrv – ne

Shows the Following details about the PGW

  • Hardware Platform
  • Vendor
  • MGC Application Version
  • Platform State

 

 

  • Check the Current Running Processes on PGW Sun Server Box.

 

Check the Processes on PGW Sun Server Box

# ps –ef

 

UID PID PPID C STIME TTY TIME CMD

root 0 0 0 Jan 30 ? 0:11 sched

root 1 0 0 Jan 30 ? 1:11 /sbin/init

root 2 0 0 Jan 30 ? 0:05 pageout

root 3 0 1 Jan 30 ? 2149:08 fsflush

root 229 7 0 Jan 30 ? 0:07 /usr/lib/saf/sac -t 300

root 7 1 0 Jan 30 ? 8:10 /lib/svc/bin/svc.startd

root 1108 1 0 Jan 30 ? 25:09 /opt/CiscoMGC/snmp/mib2agt -d

root 210 1 0 Jan 30 ? 0:02 /usr/sbin/cron

daemon 217 1 0 Jan 30 ? 0:00 /usr/lib/nfs/statd

root 1068 1 0 Jun 11 ? 0:57 /lib/svc/bin/svc.configd

root 118 1 0 Jan 30 ? 17:30 /usr/lib/picl/picld

root 130 1 0 Jan 30 ? 49:21 /usr/sbin/nscd

daemon 115 1 0 Jan 30 ? 0:42 /usr/lib/crypto/kcfd

root 132 1 0 Jan 30 ? 0:00 devfsadmd

root 126 1 0 Jan 30 ? 0:00 /usr/lib/sysevent/syseventd

root 233 229 0 Jan 30 ? 0:05 /usr/lib/saf/ttymon

daemon 214 1 0 Jan 30 ? 0:00 /usr/sbin/rpcbind

root 232 1 0 Jan 30 ? 1:26 /usr/lib/utmpd

root 234 1 0 Jan 30 ? 0:22 /usr/lib/inet/inetd start

root 296 1 0 Jan 30 ? 2:37 /usr/lib/fm/fmd/fmd

root 6110 6103 0 May 27 ? 114:15 /opt/CiscoMGC/snmp/critagt -d

root 261 1 0 Jan 30 ? 0:04 /usr/lib/ssh/sshd

root 255 1 0 Jan 30 ? 4:04 /usr/sbin/syslogd

root 1109 1 0 Jan 30 ? 8:30 /opt/CiscoMGC/snmp/hostagt -d

root 297 1 0 Jan 30 ? 0:00 /usr/sbin/mdmonitord

root 335 334 0 Jan 30 ? 75:09 /opt/TimesTen/tt60/bin/timest

ensubd -verbose -id 0 -facility user

root 334 1 0 Jan 30 ? 100:45 /opt/TimesTen/tt60/bin/timest

end -initfd 13

root 336 334 0 Jan 30 ? 18:12 /opt/TimesTen/tt60/bin/timest

 

 

 

  • Check the CPU Utilization for Each Process

 

Checking the CPU Utilization for each Process

# prstat

 

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP

13989 mgcusr 1324M 804M sleep 59 -19 3:07:36 0.7% engine.smartall/5

13997 mgcusr 55M 18M sleep 59 0 1:03:48 0.2% ioChanMgr/1

13988 mgcusr 70M 66M sleep 59 0 0:53:20 0.2% replicator/4

14001 mgcusr 32M 13M sleep 59 0 0:53:07 0.2% ISDNIP/1

14002 mgcusr 32M 13M sleep 59 0 0:47:45 0.1% ISDNIP/1

13993 mgcusr 153M 138M sleep 59 0 0:47:06 0.1% pom/5

13979 mgcusr 11M 7792K sleep 59 0 1:10:44 0.1% diskmonitor/2

13982 mgcusr 45M 25M sleep 59 0 0:16:02 0.1% measMgr/1

14003 mgcusr 101M 97M sleep 59 0 0:30:59 0.1% SS7/1

14008 mgcusr 97M 93M sleep 59 0 0:29:58 0.1% SS7/1

14004 mgcusr 32M 13M sleep 59 0 0:28:33 0.1% ISDNIP/1

14005 mgcusr 97M 93M sleep 59 0 0:25:24 0.1% SS7/1

14007 mgcusr 32M 13M sleep 59 0 0:24:13 0.1% ISDNIP/1

14009 mgcusr 97M 93M sleep 59 0 0:24:51 0.1% SS7/1

28729 root 4872K 4632K cpu1 59 0 0:00:00 0.1% prstat/1

130 root 3864K 2064K sleep 59 0 0:49:20 0.0% nscd/25

28702 root 8096K 2616K sleep 59 0 0:00:00 0.0% sshd/1

18762 mgcusr 13M 6808K sleep 59 0 0:07:26 0.0% procM/1

13986 mgcusr 43M 23M sleep 59 0 0:03:07 0.0% mmSAgt/1

2590 root 2304K 824K sleep 100 - 0:17:33 0.0% xntpd/1

13983 mgcusr 12M 8264K sleep 59 0 0:00:03 0.0% cfgM/1

28699 root 8464K 4992K sleep 59 0 0:00:00 0.0% sshd/1

18260 mgcusr 4440K 2776K sleep 59 0 0:01:27 0.0% lmgrd/1

338 root 11M 4376K sleep 59 0 0:18:09 0.0% timestensubd/5

336 root 11M 4376K sleep 59 0 0:18:11 0.0% timestensubd/5

297 root 3488K 32K sleep 59 0 0:00:00 0.0% mdmonitord/1

1109 root 3896K 1664K sleep 59 0 0:08:30 0.0% hostagt/1

255 root 3744K 1728K sleep 59 0 0:04:04 0.0% syslogd/13

261 root 3944K 808K sleep 59 0 0:00:03 0.0% sshd/1

296 root 8496K 992K sleep 59 0 0:02:36 0.0% fmd/15

Total: 72 processes, 226 lwps, load averages: 0.10, 0.10, 0.09

#

 

 

  • Check the PGW Sun Server CPU Hardware Status

 

 

# psrinfo –v

Status of virtual processor 0 as of: 09/08/2009 16:44:53
on-line since 01/30/2009 04:28:39.
The sparcv9 processor operates at 1503 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 09/08/2009 16:44:53
on-line since 01/30/2009 04:28:38.
The sparcv9 processor operates at 1503 MHz,
and has a sparcv9 floating point processor.
#

 

  • Check the PGW Patch Information


# pkginfo | grep CSC

 

  • Check the PGW Network and Interface Configuration

 

This command shows the Bge Interface IP Address configuration

# ifconfig –a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 210.118.72.51 netmask fffffff8 broadcast 210.118.72.55
ether 0:14:4f:8f:c0:d2
bge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 210.118.72.59 netmask fffffff8 broadcast 210.118.72.63
ether 0:14:4f:8f:c0:d3
bge2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 210.118.75.51 netmask ffffffe0 broadcast 210.118.75.59
ether 0:14:4f:8f:c0:d4
#

 

Handling Alarms in PGW2200


Alarms are generated for Processes, Link sets, SPC, Signaling, Bearer links and the Service Status

 

The active alarm log file resides in /opt/CiscoMGC/var/log directory. This directory will have the following log files created for all the dates and have history of log files.

 

  • Platform_<date>.log
  • Diskmonitor_<date>.log
  • Mml_<date>.log


MML for Retrieving alarms from the PGW

 

  • rtrv-alms : To Retrieve Alarms
  • ack-alm : To acknowledge Alarms

 

Retrieving & Acknowledging alarms in PGW2200

Syntax:

-------

RTRV-ALMS

RTRV-ALMS::CONT

 

Command Description:

CONT -- Tells the system to display alarm events until Ctrl-C is pressed.

 

Alarm severity levels (SEV=) are as follows:

- Critical (CR)

- Major (MJ)

- Minor (MN)

Informational

 

 


How To Acknowledge Alarms ?

Examples:
---------
The MML command that follows is used in acknowledging a Config Fail alarm for the I/O channel manager process:

mml>ACK-ALM:IOCM-01:"Config Fail"
Media Gateway Controller - MGC-01 2000-01-12 15:19:51
M COMPLD
;

 

 

System Log files

 

The active system log file resides in /opt/CiscoMGC/var/log directory

 

Can also use UNIX commands to parse through the logs. Following System logs can be found in the directory with the history of logs

 

  • Platform_<date>.log
  • Diskmonitor_<date>.log
  • Mml_<date>.log

 

Sample log files from PGW2200

 

Platform_<date>.log file

(Contains Error on PGW2200)

 

Mon Aug 17 04:09:29:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538a,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

 

Mon Aug 17 04:14:39:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538b,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

 

Mon Aug 17 04:19:49:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538c,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

 

Mon Aug 17 04:24:59:450 2009 IST | engine (PID 4572) <Error>

CP_ERR_OBJ_PTR_IS_NULL: cmgProtocolAdapter::newCall: UCID=0005538d,

OSigPath=00150003, OTG=*NA*, OSPAN=*NA*, OTS/CIC=0,

object ptr cmgProtocolAdapter is NULL

Mon Aug 17 21:22:55:110 2009 IST | ss7-i-1 (PID 4585) <Error>

TIOS_ERR_RUDP_MAX_RETRANS: Connection 00000000052f9d70, maximum number (2) of retransmission exceeded

 

Mon Aug 17 21:22:55:110 2009 IST | ss7-i-1 (PID 4585) <Error>

TIOS_ERR_RUDP_MAX_RESYN: Connection 00000000052f9d70, maximum number (1) of soft resets exceeded

 

Mon Aug 17 21:22:55:110 2009 IST | ss7-i-1 (PID 4585) <Error>

480002, RUDP connection received SOFT_RESET signal

 

 

 

 

mml_<date>.log file

 

 

un Aug 16 04:10:00:901 2009 IST | mml11 (PID 10954) <Info>

user 'mgcusr' invoked mml11 session

 

Sun Aug 16 04:10:00:902 2009 IST | mml11 (PID 10954) <Info>

/tmp/almM_input: installed time handler, hdlrId = 7

 

Sun Aug 16 04:10:00:903 2009 IST | mml11 (PID 10954) <Info>

procEventHandler: process state transition beginning (3) -> (5)

 

Sun Aug 16 04:10:02:074 2009 IST | mml11 (PID 10954) <Info>

MML_INFO_COMMAND: MML Command

mml> sta-aud

MGC-01 - Media Gateway Controller 2009-08-16 04:10:02.073 IST

M COMPLD

"ENGG-01"

;

Sun Aug 16 04:10:02:581 2009 IST | mml11 (PID 10954) <Info>

user 'mgcusr' quit mml11 session

 

Sun Aug 16 07:52:28:720 2009 IST | mml2 (PID 10726) <Info>

MML_INFO_COMMAND: MML Command

mml> rtrv-c7lnk:all:

 

MGC-01 - Media Gateway Controller 2009-08-16 07:52:28.680 IST

M RTRV

"c7ip-newyrlink-ny1:ls-sprint-ny1,LID=0:IS"

"c7ip-newyrlink-ny2:ls-sprint-ny2,LID=0:IS"

"c7ip-docotst:lnknset-test,LID=0:IS"

;

 

 

diskmonitor_<date>.log file

 

Tue Sep 1 17:37:52:514 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_DISK_CLEAN: Removed 75 files from /opt/CiscoMGC/etc/CONFIG_LIB/CFG_testdpc

 

Tue Sep 1 17:37:52:515 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_REMOVE_CONFIG_DIR: Removed config dir /opt/CiscoMGC/etc/CONFIG_LIB/CFG_testdpc

 

Tue Sep 1 17:50:19:576 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_DISK_CLEAN: Removed 75 files from /opt/CiscoMGC/etc/CONFIG_LIB/CFG_HYDE1CONF

 

Tue Sep 1 17:50:19:576 2009 IST | diskmonitor (PID 6030) <Info>

GEN_INFO_REMOVE_CONFIG_DIR: Removed config dir /opt/CiscoMGC/etc/CONFIG_LIB/CFG_HYDE1CONF

 

Viewing logs

 

Viewing logs

1. Go to the log directory

cd /opt/MGC/var/log

 

2. View your specific file by using cat

inswpgw02% cat mml_20040310040538.log | more

 

Tue Mar 9 04:10:00:474 2004 IST | mml11 (PID 9395) <Info>

/tmp/almM_input: installed time handler, hdlrId = 7

Tue Mar 9 04:10:00:476 2004 IST | mml11 (PID 9395) <Info>

procEventHandler: process state transition beginning (3) -> (5)

Tue Mar 9 04:10:00:476 2004 IST | mml11 (PID 9395) <Info>

procEventHandler: process state transition ending (3) -> (5)

Tue Mar 9 04:10:00:983 2004 IST | mml11 (PID 9395) <Info>

Unable to process event

Tue Mar 9 04:10:01:000 2004 IST | mml11 (PID 9395) <Info>

MML_INFO_COMMAND: MML Command

Tue Mar 9 04:10:01:510 2004 IST | mml11 (PID 9395) <Info>

Unable to process event

 

Exporting Configuration files

 

Header 1

Exports current configuration of the MGC (Media Gateway Controller)

 

prov-exp:<component/target>:dirname=<directory name>

 

Eg. Prov-exp:config:dirname=“data070909”

 

Syntax description:

 

Target/Component -- all -- All Configuration Data

 

Component /Targent Name can be one of the following:

 

config : Core Configuration Data

routing : Routing Data

numan : Number Analysis Data

trkgrp : Trunk Group Data

trunk : Trunk Data

all : All Configuration Data

 

Export directory name can be any directory name, in double quotes, which will be created under the cust_specific directory.

 

 

PGW2200 Call Trace

A trace is a record of the SS7 ISUP Message flow through the PGW2200 engine.

 

  • If a trace is empty, the call hasn’t reached the engine!
  • Start trace with sig-path details from where the call originates
  • Start a trace with the following MML command:

STA-SC-TRC:[sig-path]:log=“[name]”,CONFIRM

 

  • Stop a trace with the following MML command:

STP-SC-TRC:ALL

 

How to Enable the Call Trace?

 

How to enable the PGW2200 Call Trace ?

 

How To Enable the Call TRACE ?

 

swpgw1 mml> sta-sc-trc:

 

Starts a protocol conversion trace on a signal path. The user typically performs a trace when calls are failing, and the engine is the suspected source of the problem.

 

Examples:

--------------

This command will initiate the Signaling trace on the Specified Signaling Path, the filename would be: <sig path>_yyyymmddhhmmss.btr.

 

This Trace log is stored in /opt/CiscoMGC/var/trace directory on the PGW

 

 

Mml> STA-SC-TRC: < SIGPATH>:CONFIRM

 

 

How To Stop the Call TRACE ?

 

stp-sc-trc:<sig path>

stp-sc-trc:all

 

Purpose: To Stop One or All Traces

 

Examples:

---------------

 

The MML command shown in the following example stops all active capture sessions:

 

Mml> STP-SC-TRC:all

 

How to Collect the PGW2200 MDL Trace?

 

Collecting PGW2200 MDL Trace

 

How to Collect the PGW2200 MDL Trace ?

 

Use this procedure in order to collect an MDL trace via the MML command STA−SC−TRC (Start Trace).

 

Step1. Identify the Originating SS7 SigPath Number or the Originating TrunkGroup Number on which calls are placed.

 

 

Step2. Start the MDL trace:

mml>sta−sc−trc:<ss7sigPath name >:CONFIRM

 

 

Step3. Perform a test, make few test calls via the SS7 Sigpath.

 

 

Step4. Stop the MDL trace:

mml>stp−sc−trc:all

 

 

Step5.Identify the Call Id (C:) of the bad call

Note: These files can contain tracings from many calls that are all mixed up together if the capture is

taken on a production PGW. Each tracing record in the file has a specific record type and records

information of a type that relates to that record. Each record has a Call ID that relates it to a specific

call.

 

 

Step6: Convert the MDL trace into a readable format:

 

 

Go to the /opt/CiscoMGC/var/trace directory.

 

 

-bash-3.00# cd /opt/CiscoMGC/var/trace

-bash-3.00# ls

-bash-3.00# ls

_nasp-norte-ny3_20081212123706.btr

_nasp-norte-ny3_20090604122453.btr

_nasp-norte-ny3_20090604122453_btr_12958689.trc

_nasp-norte-ny3_20090604122453_btr_12958691.trc

_ss7p-norte-ny1_20081221170223.prt

_ss7p-norte-ny1_20081224003732.btr

_ss7p-norte-ny1_20081224003732_btr_4932521.trc

_ss7p-norte-ny1_20081224022732.btr

_ss7p-norte-ny1_20081224022732_btr_4956105.trc

_ss7p-norte-ny2_20090206094100.btr

_ss7p-norte-ny2_20090206094100_btr_10598642.trc

cicss7trace_ss7p-norte-ny1_20081221182221.btr

cictrace_nasp-norte-ny1_20081221180123.btr

finaltrace_ss7p-norte-ny1_20081221184228.btr

naspath2_nasp-norte-ny2_20081221181939.btr

nortenumber_nasp-norte-ny3_20081212124935.btr

ss7p-norte-ny1_20081221170223.prt.gz

trace.tar

 

 

Step7: Run this command in mgcusr Login mode.

get_trc.sh <trace file name. btr>

 

 

Eg. Get_trc.sh nortenumber_nasp-norte-ny3_20081212124935.btr

 

 

For example:

/opt/CiscoMGC/var/trace

 

 

swpgw2% get_trc.sh nortenumber_nasp-norte-ny3_20081212124935.btr

get_trc.sh ca/sim/sp Trace File Utility Mistral Version 1.2

The ANALYSIS mdo file is: GENERIC_ANALYSIS.mdo

Retrieving _ss7path_20040116103221.btr trace file Call ID's, please wait...

Enter one of the following commands:

S = Simprint in less

F = Simprint with printing of sent and received Fields in less

D = Display trc trace in less

G = Display trc trace in less (Generated)

C = Convert to trc trace file

A = Display CA file in less

N = Move to Next call ID

P = Move to Previous call ID

L = List call ID's in current file

X = Set SP flags

H = Print Help

Q = Quit get_trc.sh

Or just enter the ID of the call you want if you know it

Use (N)ext and (P)revious to move between the call ID's

_ss7path_20040116103221.btr contains 1 call(s)

==> Working on call 1 ID 23 H = Help [S/F/D/G/C/A/N/P/L/H/Q/id]?

 

 

Step8: This get_trc.sh <filename.btr> command gives the complete trace of the SS7 Message.

 

 

Step9: Choose option C in order to convert the trace file.

Note: The .btr files are binary trace files that are produced by the PGW tracer function. The main part

of the file name is given in the VSC MML command sta−sc−trc. The PGW always adds a .btr

extension to these files. By using the C option, the file is converted into a text format and the

extension has .trc files that are text trace files. They contain detailed line by line trace information

from the MDO code that is run in the simulation replay that produces the file. Therefore, they contain

MDL traces.

 

 

Step10: The trace file is in /opt/CiscoMGC/var/trace.

 

 

Step11: Choose Option F inorder to print the SS7 Signaling Message on the Screen.

 

 

Step12: Choose Option Q inorder to quit.

 

Related Information

 

Related discussion in CSC

 

Version history
Revision #:
2 of 2
Last update:
‎08-29-2017 11:28 AM
 
Labels (1)