Solved: No response from 'cgn_ma' cisco ASR9000

kzryumov1 · ‎10-17-2013

Hello,

Cisco ASR9006 IOS XR 4.3.1 doesn't show NAT statistics. For example: show cgn nat44 natXX statistics

RP/0/RSP0/CPU0:Oct 17 11:42:22.964 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat1/stats' from 'show_cgn_nat44_stats' (jid 65900, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 17 11:42:34.527 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat2/stats' from 'show_cgn_nat44_stats' (jid 65900, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 17 11:42:46.096 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat3/stats' from 'show_cgn_nat44_stats' (jid 65899, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 17 11:44:09.403 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/cgn1/stats' from 'show_cgn_nat44_stats' (jid 65898, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 17 11:42:22.964 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM
request for 'oper/service-cgn/gl/nat44/nat1/stats' from 'show_cgn_nat44_stats' (jid 65900, node
0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period
(100 seconds)
RP/0/RSP0/CPU0:Oct 17 11:42:34.527 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM
request for 'oper/service-cgn/gl/nat44/nat2/stats' from 'show_cgn_nat44_stats' (jid 65900, node
0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period
(100 seconds)
RP/0/RSP0/CPU0:Oct 17 11:42:46.096 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM
request for 'oper/service-cgn/gl/nat44/nat3/stats' from 'show_cgn_nat44_stats' (jid 65899, node
0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period
(100 seconds)
RP/0/RSP0/CPU0:Oct 17 11:44:09.403 : sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM
request for 'oper/service-cgn/gl/nat44/cgn1/stats' from 'show_cgn_nat44_stats' (jid 65898, node
0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period
(100 seconds)

Recently, everything works fine. There was no any changes.

Could anyone help me to resolve the issue?

Regards,

Konstantin

somnathr · ‎10-18-2013

Hi Konstantin,

Yes, you can restart cgn_ma process which normally should not have service impact. We provide that as a work-around in similar cases. You can give it a try with that and see if it fixes the issue or not.

Can you pl. send "show process blocked" output as well ? If it shows cgn_ma in the list of blocked processes, process restart is a good way of coming out of it.

If you want detailed level trouble-shooting, you can raise a service request (SR) and Cisco support team can help.

regards,

Somnath.

View solution in original post

somnathr · ‎10-21-2013

Hi Konstantin,

What was before "SYSDB-SYSDB-7-INFO" line ? Can you pl. cut and paste multiple error msgs (as part of "sh logging") ?

I'm not aware of any known issue with this msg.

If you would like to us to do detailed analysis, we would need complete log and the commands that were executed before the issue was seen. Did ISM card go through any reload during that time ?

Does it work fine after 'process restart cgn_ma' ?

regards,

Somnath.

View solution in original post

somnathr · ‎10-21-2013

Those are caused by CSCui39371 (can be ignored) in this context.

View solution in original post

somnathr · ‎10-18-2013

Hi Konstantin,

Can you please send the following output ?

- show run

- show platform summary location

- show process cgn_ma

regards,

Somnath.

kzryumov1 · ‎10-18-2013

Hi Somnath,

#show platform summary location all

Fri Oct 18 13:17:04.992

-------------------------------------------------------------------------------

Platform Node : 0/RSP0/CPU0 (slot 0)

PID : A9K-RSP-4G

Card Type : ASR9K Fabric, Controller, 4G memory

VID/SN : V03 / FOC15078EWY

Oper State : IOS XR RUN

Last Reset : User Initiated reload

Process: reload

: Tue Aug 20 10:25:59 2013

Configuration : Power is enabled

Bootup enabled.

Monitoring enabled

Rommon Ver : Version 1.06(20120210:003513)

IOS SW Ver : 4.3.1

Main Power : Power state Enabled. Estimate power 205 Watts of power required.

Faults : N/A

-------------------------------------------------------------------------------

Platform Node : 0/RSP1/CPU0 (slot 1)

PID : A9K-RSP-4G

Card Type : ASR9K Fabric, Controller, 4G memory

VID/SN : V02 / FOC142683EY

Oper State : IOS XR RUN

Last Reset : User Initiated reload

Process: reload

: Tue Aug 20 15:00:03 2013

Configuration : Power is enabled

Bootup enabled.

Monitoring enabled

Rommon Ver : Version 1.06(20120210:003513)

IOS SW Ver : 4.3.1

Main Power : Power state Enabled. Estimate power 205 Watts of power required.

Faults : N/A

-------------------------------------------------------------------------------

Platform Node : 0/0/CPU0 (slot 2)

PID : A9K-8T-L

Card Type : 8-Port 10GE Low Queue Line Card, Requires XFPs

VID/SN : V03 / FOC153282Z2

Oper State : IOS XR RUN

Last Reset : Reloading managed node

Process: insthelper

: Fri Aug 16 11:48:31 2013

Configuration : Power is enabled

Bootup enabled.

Monitoring enabled

Rommon Ver : Version 1.03(20100211:014208)

IOS SW Ver : 4.3.1

Main Power : Power state Enabled. Estimate power 575 Watts of power required.

Faults : N/A

-------------------------------------------------------------------------------

Platform Node : 0/1/CPU0 (slot 3)

PID : A9K-ISM-100

Card Type : Integrated Services Module

VID/SN : V01 / FOC1706N0P0

Oper State : IOS XR RUN

Last Reset : MBI-HELLO reloading node on receiving reload notification

Process: mbi-hello

: Thu Aug 29 10:55:57 2013

Configuration : Power is enabled

Bootup enabled.

Monitoring enabled

Rommon Ver : Version 1.2(20091201:235620)

IOS SW Ver : 4.3.1

Main Power : Power state Enabled. Estimate power 405 Watts of power required.

Faults : N/A

-------------------------------------------------------------------------------

Platform Node : 0/1/CPU1 (slot 3)

Card Type : Integrated Services Module (Service Engine)

Oper State : APP-READY

Last Start Time : Tue Sep 3 14:37:10 2013

Last Ready Time : Tue Sep 3 14:39:13 2013

Uptime : 1078:37:53

BIOS Ver : 0.17 (Thurley.3.60.18.0033)

SW Ver : 1.1.1.0 (Built on May 08 2013, from services/cgn@r431-support/3)

App Status : 8 CGv6 Application instance(s) is/are running

-------------------------------------------------------------------------------

Platform Node : 0/2/CPU0 (slot 4)

PID : A9K-ISM-100

Card Type : Integrated Services Module

VID/SN : V01 / FOC1704N09P

Oper State : IOS XR RUN

Last Reset : MBI-HELLO reloading node on receiving reload notification

Process: mbi-hello

: Thu Aug 29 11:31:19 2013

Configuration : Power is enabled

Bootup enabled.

Monitoring enabled

Rommon Ver : Version 1.2(20091201:235620)

IOS SW Ver : 4.3.1

Main Power : Power state Enabled. Estimate power 405 Watts of power required.

Faults : N/A

-------------------------------------------------------------------------------

Platform Node : 0/2/CPU1 (slot 4)

Card Type : Integrated Services Module (Service Engine)

Oper State : APP-READY

Last Start Time : Tue Sep 3 14:37:20 2013

Last Ready Time : Tue Sep 3 14:39:46 2013

Uptime : 1078:37:20

BIOS Ver : 0.17 (Thurley.3.60.18.0033)

SW Ver : 1.1.1.0 (Built on May 08 2013, from services/cgn@r431-support/3)

App Status : 8 CGv6 Application instance(s) is/are running

-------------------------------------------------------------------------------

Platform Node : 0/3/CPU0 (slot 5)

PID : A9K-ISM-100

Card Type : Integrated Services Module

VID/SN : V01 / FOC1704N09Q

Oper State : IOS XR RUN

Last Reset : MBI-HELLO reloading node on receiving reload notification

Process: mbi-hello

: Tue Aug 20 10:50:48 2013

Configuration : Power is enabled

Bootup enabled.

Monitoring enabled

Rommon Ver : Version 1.2(20091201:235620)

IOS SW Ver : 4.3.1

Main Power : Power state Enabled. Estimate power 405 Watts of power required.

Faults : N/A

-------------------------------------------------------------------------------

Platform Node : 0/3/CPU1 (slot 5)

Card Type : Integrated Services Module (Service Engine)

Oper State : APP-READY

Last Start Time : Tue Sep 3 14:37:10 2013

Last Ready Time : Tue Sep 3 14:39:13 2013

Uptime : 1078:37:54

BIOS Ver : 0.17 (Thurley.3.60.18.0033)

SW Ver : 1.1.1.0 (Built on May 08 2013, from services/cgn@r431-support/3)

App Status : 8 CGv6 Application instance(s) is/are running

-------------------------------------------------------------------------------

RP/0/RSP0/CPU0:deg-nat01.#

RP/0/RSP0/CPU0:deg-nat01.# show process cgn_ma

Fri Oct 18 13:18:53.717 MSK

Job Id: 1075

PID: 602460

Executable path: /disk0/iosxr-service-4.3.1/0x100000/bin/cgn_ma

Instance #: 1

Version ID: 00.00.0000

Respawn: ON

Respawn count: 1

Max. spawns per minute: 12

Last started: Tue Sep 3 14:35:27 2013

Process state: Run

Package state: Normal

Started on config: cfg/gl/service-cgn/

Process group: central-services

core: MAINMEM

Max. core: 0

Placement: DLRSC Tracker

startup_path: /pkg/startup/cgn_ma.startup

Ready: 1.757s

Process cpu time: 81.527 user, 6.092 kernel, 87.619 total

JID TID CPU Stack pri state TimeInState HR:MM:SS:MSEC NAME

1075 1 1 52K 10 Reply 25:48:11:0377 0:01:27:0579 cgn_ma

1075 2 0 52K 10 Receive 1078:43:26:0449 0:00:00:0001 cgn_ma

1075 3 0 52K 10 Receive 1078:43:26:0427 0:00:00:0000 cgn_ma

1075 4 0 52K 10 Condvar 97:04:59:0192 0:00:00:0025 cgn_ma

1075 5 1 52K 10 Receive 1078:38:43:0702 0:00:00:0013 cgn_ma

-------------------------------------------------------------------------------

RP/0/RSP0/CPU0:deg-nat01.# show process cgn_ma detail location all

Fri Oct 18 13:18:18.805

node: node0_0_CPU0

-------------------------------------------------------------------------------

No such process cgn_ma

-------------------------------------------------------------------------------

node: node0_1_CPU0

-------------------------------------------------------------------------------

No such process cgn_ma

-------------------------------------------------------------------------------

node: node0_2_CPU0

-------------------------------------------------------------------------------

No such process cgn_ma

-------------------------------------------------------------------------------

node: node0_3_CPU0

-------------------------------------------------------------------------------

No such process cgn_ma

-------------------------------------------------------------------------------

node: node0_RSP0_CPU0

-------------------------------------------------------------------------------

Job Id: 1075

PID: 602460

Executable path: /disk0/iosxr-service-4.3.1/0x100000/bin/cgn_ma

Instance #: 1

Version ID: 00.00.0000

Respawn: ON

Respawn count: 1

Max. spawns per minute: 12

Last started: Tue Sep 3 14:35:27 2013

Process state: Run

Package state: Normal

Started on config: cfg/gl/service-cgn/

Process group: central-services

core: MAINMEM

Max. core: 0

Placement: DLRSC Tracker

startup_path: /pkg/startup/cgn_ma.startup

Ready: 1.757s

Running path: /disk0/iosxr-service-4.3.1/0x100000/bin/cgn_ma

Package path: /pkg/bin/cgn_ma

Job-id-link: 1075

group_jid: 8600,0,0,0

fail_count: 0

this pcb: 0xe7ffe838

next pcb: 0x6016f400

jobid on RP: 0

Standby Status: 2

Process cpu time: 81.527 user, 6.092 kernel, 87.619 total

JID TID CPU Stack pri state TimeInState HR:MM:SS:MSEC NAME

1075 1 1 52K 10 Reply 25:47:38:0871 0:01:27:0579 cgn_ma

1075 2 0 52K 10 Receive 1078:42:53:0943 0:00:00:0001 cgn_ma

1075 3 0 52K 10 Receive 1078:42:53:0922 0:00:00:0000 cgn_ma

1075 4 0 52K 10 Condvar 97:04:26:0687 0:00:00:0025 cgn_ma

1075 5 1 52K 10 Receive 1078:38:11:0197 0:00:00:0013 cgn_ma

-------------------------------------------------------------------------------

node: node0_RSP1_CPU0

-------------------------------------------------------------------------------

Job Id: 1075

PID: 606510

Executable path: /disk0/iosxr-service-4.3.1/0x100000/bin/cgn_ma

Instance #: 1

Version ID: 00.00.0000

Respawn: ON

Respawn count: 1

Max. spawns per minute: 12

Last started: Tue Sep 3 14:36:43 2013

Process state: Run

Package state: Normal

Started on config: cfg/gl/service-cgn/cgn1/ord_a/enter

Process group: central-services

core: MAINMEM

Max. core: 0

Placement: DLRSC Tracker

startup_path: /pkg/startup/cgn_ma.startup

Ready: 1.940s

Running path: /disk0/iosxr-service-4.3.1/0x100000/bin/cgn_ma

Package path: /pkg/bin/cgn_ma

Job-id-link: 1075

group_jid: 8600,0,0,0

fail_count: 0

this pcb: 0xe7ffe838

next pcb: 0x6016f400

jobid on RP: 0

Standby Status: 2

role_event_clinfo: 5009c234

Proc role state: backup

Process cpu time: 31.437 user, 12.015 kernel, 43.452 total

1075 1 0 36K 10 Receive 0:00:00:0228 0:00:43:0449 cgn_ma

1075 2 1 36K 10 Receive 1078:41:35:0991 0:00:00:0001 cgn_ma

1075 3 1 36K 10 Receive 1078:41:35:0811 0:00:00:0000 cgn_ma

-------------------------------------------------------------------------------

The config is too long. I absolutely confident that recently it works.

Is it possible to restart cgn_ma process? Would be any service impact?

Regards,

Konstantin

somnathr · ‎10-18-2013

Hi Konstantin,

Yes, you can restart cgn_ma process which normally should not have service impact. We provide that as a work-around in similar cases. You can give it a try with that and see if it fixes the issue or not.

Can you pl. send "show process blocked" output as well ? If it shows cgn_ma in the list of blocked processes, process restart is a good way of coming out of it.

If you want detailed level trouble-shooting, you can raise a service request (SR) and Cisco support team can help.

regards,

Somnath.

kzryumov1 · ‎10-21-2013

Hi Somnath,

Very thanks.

The cgn_ma process was in blocked state.

I noticed that this occured after these message:

%SYSDB-SYSDB-7-INFO :

exec:oper/tty/gl/________/________/vty/3/stats/conn-out/: rc 0x40824200 ('sysdb' detected the

'warning' condition 'A SysDB client attempted to register for verification or as an EDM on a

path which was already registered')

What does it means? Could we avoid this problem in the future?

Regards,

Konstantin

somnathr · ‎10-21-2013

Hi Konstantin,

What was before "SYSDB-SYSDB-7-INFO" line ? Can you pl. cut and paste multiple error msgs (as part of "sh logging") ?

I'm not aware of any known issue with this msg.

If you would like to us to do detailed analysis, we would need complete log and the commands that were executed before the issue was seen. Did ISM card go through any reload during that time ?

Does it work fine after 'process restart cgn_ma' ?

regards,

Somnath.

kzryumov1 · ‎10-21-2013

Hi Somnath.

I've calculated from time when process was blocked in logs were:

LC/0/2/CPU0:Oct 17 04:21:55.946 cgv6_services[140]: core_event_logging_init[835] core 2, Error code 1, Msg: ** i-vrf: 0x11 :: ipv4 addr: 0xae5405d :: port: 0xb0d6 :: ** , 'cgv6_services' detected the 'warning' condition 'No CGN config'

LC/0/2/CPU0:Oct 17 04:21:55.946 MSK: cgv6_services[140]: %PLATFORM-CGV6_SERVICES-7-GENERIC_ERROR : core 2, Error code 1, Msg: ** i-vrf: 0x11 :: ipv4 addr: 0xae5405d :: port: 0xb0d6 :: ** , 'cgv6_services' detected the 'warning' condition 'No CGN config' 0x49418200

R LC/0/2/CPU0:Oct 17 04:21:55.946 cgv6_services[140]: core_event_logging_init[835] core 2, Error code 1, Msg: ** i-vrf: 0x11 :: ipv4 addr: 0xae5405d :: port: 0xb0d6 :: ** , 'cgv6_services' detected the 'warning' condition 'No CGN config'
LC/0/2/CPU0:Oct 17 04:21:55.946 MSK: cgv6_services[140]: %PLATFORM-CGV6_SERVICES-7-GENERIC_ERROR : core 2, Error code 1, Msg: ** i-vrf: 0x11 :: ipv4 addr: 0xae5405d :: port: 0xb0d6 :: ** , 'cgv6_services' detected the 'warning' condition 'No CGN config' 0x49418200

There was no any change in config at this time. What does these messages mean?

Regards,

Konstantin

somnathr · ‎10-21-2013

Those are caused by CSCui39371 (can be ignored) in this context.

kzryumov1 · ‎10-21-2013

Great!

Thanks.

And it is fixed in 4.3.2. Currently, we use 4.3.1.

kzryumov1 · ‎10-22-2013

Hi Somnath,

The problem reproduced again on another ASR and with 4.3.1.

#show processes blocked

Tue Oct 22 11:18:12.172 MSK

Jid Pid Tid Name State TimeInState Blocked-on

65551 16399 1 ksh Reply 1152:52:17:0014 16395 devc-conaux

65554 61458 2 devb-umass Reply 0:00:00:0126 53284 io-usb

98 49187 2 umass-enum Reply 1152:52:21:0728 1 kernel

98 49187 6 umass-enum Reply 1152:52:15:0896 53284 io-usb

98 49187 7 umass-enum Reply 1152:52:18:0625 1 kernel

98 49187 8 umass-enum Reply 1152:52:17:0227 1 kernel

98 49187 9 umass-enum Reply 1152:52:15:0896 1 kernel

52 65573 2 attachd Reply 1152:52:18:0576 53292 eth_server

52 65573 3 attachd Reply 1152:52:18:0574 24595 mqueue

89 65574 6 qnet Reply 0:00:00:0386 53292 eth_server

65584 65584 2 devb-umass Reply 0:00:40:0630 53284 io-usb

51 65590 2 attach_server Reply 1152:52:18:0090 24595 mqueue

65594 77882 2 devb-umass Reply 0:00:04:0484 53284 io-usb

65603 798787 1 exec Reply 672:13:27:0244 16395 devc-conaux

433 237664 1 tftp_server Reply 1152:47:58:0911 24595 mqueue

315 598246 2 lpts_fm Reply 0:00:00:0176 286865 lpts_pa

1075 610601 1 cgn_ma Reply 0:08:25:0003 454777 node 0/1/CPU0 cgn_ea

65893 616034661 1 exec Reply 0:00:00:0164 1 kernel

65894 616063334 1 more Reply 0:00:00:0075 24593 pipe

65895 609853799 1 exec Reply 0:03:35:0428 1 kernel

65897 616063337 1 show_processes Reply 0:00:00:0000 1 kernel

RP/0/RSP0/CPU0:nag-nat01.msk#

RP/0/RSP0/CPU0:nag-nat01.msk#show processes blocked 1075 extended

Tue Oct 22 11:18:44.970 MSK

Jid Pid Tid Name State TimeInState Blocked-on

1075 610601 1 cgn_ma Reply 0:08:57:0723 454777 node 0/1/CPU0 cgn_ea

Following thread 1 to process cgn_ea, pid 454777 on node 0/1/CPU0

139 454777 1 cgn_ea Receive 0:01:06:0352

139 454777 2 cgn_ea Receive 191:03:14:0571

139 454777 3 cgn_ea Receive 0:08:57:0746

Resuming display of process cgn_ma, pid 610601, jid 1075

1075 610601 2 cgn_ma Receive 1152:46:18:0789

1075 610601 3 cgn_ma Receive 1152:46:18:0674

1075 610601 4 cgn_ma Cond 191:03:12:0306 (0x50040608)

1075 610601 5 cgn_ma Receive 1151:56:49:0818

And we see only this message:

%SYSDB-SYSDB-7-INFO :

exec:oper/tty/gl/________/________/vty/1/stats/conn-out/: rc 0x40824200 ('sysdb' detected the

'warning' condition 'A SysDB client attempted to register for verification or as an EDM on a

path which was already registered')

RP/0/RSP0/CPU0:Oct 22 11:10:50.726 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-7-INFO :

exec:oper/tty/gl/________/________/vty/1/stats/conn-out/: rc 0x40824200 ('sysdb' detected the

'warning' condition 'A SysDB client attempted to register for verification or as an EDM on a

path which was already registered')

RP/0/RSP0/CPU0:Oct 22 11:10:51.084 MSK: exec[65895]: %SECURITY-login-6-AUTHEN_SUCCESS :

Successfully authenticated user 'rancid' from '2a02:2168:202:1:77:37:251:71' on 'vty1'

RP/0/RSP0/CPU0:Oct 22 11:11:03.514 MSK: exec[65895]: %SECURITY-login-6-AUTHEN_SUCCESS :

Successfully authenticated user 'rvp' from '2a02:2168:204:0:227:eff:fe02:7350' on 'vty1'

RP/0/RSP0/CPU0:Oct 22 11:11:27.364 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/cgn1/pcpcounters' from 'show_cgn_nat44_pcp_counters'

(jid 65894, node 0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within

the timeout period (100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:11:43.377 MSK: config[65885]: %MGBL-SYS-5-CONFIG_I : Configured from

console by asaveliev on vty0 (2a02:2168:204:0:290:27ff:fe23:f4dd)

RP/0/RSP0/CPU0:Oct 22 11:12:14.581 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat1/stats' from 'show_cgn_nat44_stats' (jid 65894, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:12:19.944 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat1/stats' from 'show_cgn_nat44_stats' (jid 65898, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:12:31.496 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat2/stats' from 'show_cgn_nat44_stats' (jid 65901, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:13:34.229 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat1/stats' from 'show_cgn_nat44_stats' (jid 65893, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:14:30.973 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for

'oper/service-cgn/gl/nat44/nat1/putil/invrf/CGN1/address-rangefilter/b9131474,b91314e1/' from

'show_cgn_ma_nat44_pool_util' (jid 65893, node 0/RSP0/CPU0). No response from 'cgn_ma' (jid

1075, node 0/RSP0/CPU0) within the timeout period (100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:15:42.562 MSK: exec[65900]: %SECURITY-login-6-AUTHEN_SUCCESS :

Successfully authenticated user 'rancid' from '2a02:2168:202:1:77:37:251:71' on 'vty0'

RP/0/RSP0/CPU0:Oct 22 11:15:56.076 MSK: exec[65894]: %SECURITY-login-6-AUTHEN_SUCCESS :

Successfully authenticated user 'rancid' from '2a02:2168:202:1:77:37:251:71' on 'vty0'

RP/0/RSP0/CPU0:Oct 22 11:16:08.669 MSK: exec[65882]: %SECURITY-login-6-AUTHEN_SUCCESS :

Successfully authenticated user 'rancid' from '2a02:2168:202:1:77:37:251:71' on 'vty0'

RP/0/RSP0/CPU0:Oct 22 11:17:23.634 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat1/stats' from 'show_cgn_nat44_stats' (jid 65905, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:17:36.484 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat2/stats' from 'show_cgn_nat44_stats' (jid 65902, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:17:50.271 MSK: sysdb_shared_nc[417]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM

request for 'oper/service-cgn/gl/nat44/nat3/stats' from 'show_cgn_nat44_stats' (jid 65898, node

0/RSP0/CPU0). No response from 'cgn_ma' (jid 1075, node 0/RSP0/CPU0) within the timeout period

(100 seconds)

RP/0/RSP0/CPU0:Oct 22 11:17:50.787 MSK: exec[65893]: %SECURITY-login-6-AUTHEN_SUCCESS :

Successfully authenticated user 'rvp' from '2a02:2168:204:0:227:eff:fe02:7350' on 'vty0'

RP/0/RSP0/CPU0:nag-nat01.msk#

Regards,

Konstantin

kzryumov1 · ‎10-22-2013

Hello Somnath,

We use every 1 and 4 minutes rancid functionaliry to collect counters from #show cgn nat44 natXX stat via CLI becouse of it is impossible to get this information through SNMP.

Could it be root cause of the issue?

Regards,

Konstantin

somnathr · ‎10-22-2013

Hi Konstantin,

I think you are hitting the following:

CSCuh61188 cgn_ma process blked/show cmds fail on executing show cgn nat44 mapping

Do not include the following command in your script:

show cgn ... pcpCounters

Try it after that and let me know if it helps.

regards,

Somnath.

e_tizyaev · ‎10-22-2013

Hello Somnath,

Could you explain this bug, CSCuh61188.

I can't view the details.

Is there a workaround for this issue? Or we shall upgrade IOS-XR?

Thanks.

somnathr · ‎10-22-2013

Hi Egor,

Looks like Release-note tag was not set properly - that could be reason why is it not visible.

Here are the details:

Symptom:

After execution of "show cgn <> pcpCounters" command cgn_ma gets reply blocked on cgn_ea. Afterwards no other "show cgn" cli responds. The mentioned CLI is not supported on ISM and not handled gracefully. Hence, the process gets blocked.

Conditions:

Execution of "show cgn <> pcpCounters"

Workaround:

"process restart cgn_ma" to unblock the cgn_ma process.

More Info:

It is present in 4.3.1 release.

regards,

Somnath.

e_tizyaev · ‎10-22-2013

Thanks a lot, Somnath!

This is exactly our issue.

But one little question: is this issue resolved in 4.3.2 or later IOS-XR?

Thanks.