05-31-2007 05:50 AM
Hi,
We have a couple 6513, all IOS Version 12.2(18)SXD7b, with NAM modules installed. After the firmware upgrade of one of the NAM system from 3.4 to 3.6(1)we get the error:
Cannot open config file /nvram/config/protocoldir_v2.dat.
Also the refresh rate is VERY slow when using <Monitor><Overview> in compare with the other NAM system running with NAM_VERSION="3.4(1a-Patch-2)".
Tech-Support output:
cat: /nvram/config/iscsi.conf: No such file or directory
Could not open database file /nvram/config/rmon/remoteFs.dba (File open error)!
MP upgrade.log
Upgrading the line card on Thu Apr 26 14:12:45 UTC 2007
ERROR: Image download failed. Try downloading the image again.
Upgrade aborted .
Upgrading the line card on Thu Apr 26 14:13:13 UTC 2007
Downloaded upgrade image ftp://name:passwd@10.128.115.50/Cisco/c6svc-nam.3-6-1.bin.gz
Extracted the downloaded file
Proceeding with image upgrade.
Thu Apr 26 14:13:30 2007 : NAM upgrade: type 1, imageFile /tmp/cdisk.gz
Thu Apr 26 14:13:30 2007 : T: 70820477, E: 464814, I: 70820429
Thu Apr 26 14:13:30 2007 : Creating NAM application image file...
Thu Apr 26 14:13:31 2007 : startNAMAppUpgrade:Image: /tmp/cdisk.gz
Thu Apr 26 14:13:31 2007 : startNAMAppUpgrade:Root partition: /dev/hdc1
Thu Apr 26 14:13:31 2007 : startNAMAppUpgrade:Install type: 1
Thu Apr 26 14:13:31 2007 : Partition '/dev/hdc1' unmounted.
Thu Apr 26 14:13:31 2007 : Initializing the application image partition.
Thu Apr 26 14:13:31 2007 : This process may take several minutes...
Thu Apr 26 14:14:15 2007 : Ext2/3 FileSystem created on '/dev/hdc1'.
Thu Apr 26 14:14:15 2007 : Partition '/dev/hdc1' mounted.
Thu Apr 26 14:14:15 2007 : Applying the image, this process may take several minutes...
Thu Apr 26 14:14:15 2007 : Directory changed to '/mnt/hd'.
Thu Apr 26 14:14:24 2007 : Performing post install, please wait...
Thu Apr 26 14:14:24 2007 : File /mnt/hd//usr/local/nam/bin/post-install copied to /tmp/post-install.
Thu Apr 26 14:14:24 2007 : Directory changed to '/tmp'.
Thu Apr 26 14:15:17 2007 : Directory changed to '/tmp'.
Application image upgrade complete. You can boot the image now.
Partition upgraded successfully
Any ideas?
Thx,
bt
05-31-2007 07:21 AM
What does the top section look like from your show tech? What model of NAM is this? How much traffic are you spanning to the NAM? Are you currently running any capture sessions or using the RTP features of the NAM?
05-31-2007 07:48 AM
Well as you can see in the topic message it's a NAM2 module hosted in a 6513.
Date
Thu May 31 16:29:24 CEST 2007
NAM settings
NAM_TARGET=NAM2
NAM_TARGET_ID=3
CARD_IDX=95
MP=no
NAM_PROD_NO=WS-SVC-NAM-2
NAM_PROD_DESCR="Cisco Network Analysis Module (WS-SVC-NAM-2)"
NAM_VERSION="3.6(1)"
NAM_PID=WS-SVC-NAM-2
NAM_SN=SAD094503E8
Switch IP Address
127.0.0.71
We are running some captures but that is what we also do on the other NAM-2 system.
Capture Sessions 300 MB total buffer memory 223.2 MB allocated 76.8 MB available
Name Owner Start Time Buffer Size Packets Status
PASWEB LocalMgr 24 May 2007, 15:04:04 10 MB 953 Running
Capture2 LocalMgr 24 May 2007, 15:03:59 10 MB 35715 Running
(Wrapped)
db_189_110 LocalMgr 24 May 2007, 15:04:00 10 MB 41145 Running
(Wrapped)
db_189_100 LocalMgr 24 May 2007, 15:04:01 10 MB 36890 Running
(Wrapped)
DDOGM04 LocalMgr 24 May 2007, 15:03:59 10 MB 45198 Locked
CONVS_10_128_122_37and10_128_110_127 LocalMgr 24 May 2007, 15:03:59 10 MB 0 Running
dysondhcp LocalMgr 24 May 2007, 15:04:00 10 MB 47287 Locked
Dyson_AMS5073 LocalMgr 24 May 2007, 15:04:00 10 MB 0 Running
LBams168 LocalMgr 24 May 2007, 15:04:00 10 MB 44009 Locked
nat LocalMgr 24 May 2007, 15:04:01 10 MB 33956 Running
(Wrapped)
CM2AMS LocalMgr 24 May 2007, 15:04:00 30 MB 104064 Running
(Wrapped)
cogbox LocalMgr 24 May 2007, 15:04:00 10 MB 33393 Running
(Wrapped)
all_traffic LocalMgr 24 May 2007, 15:04:04 60 MB 304514 Running
(Wrapped)
Automatic_Capture NAM Alarm
(not set) 24 May 2007, 15:04:00 10 MB 51728 Running
(Wrapped)
Both NAM systems are running 24/7 Monitor/Overview for over a year now and never had this problem till we upgraded 1. We didn't want to do both at the same time to see how the upgrade went. Well not good.
Can I downgrade the NAM from 3.6 to 3.5? Is this possible since version 3.6 is very annoying.
thx,
bt
05-31-2007 08:28 AM
By "top section" of the show tech, I meant the section labeled "top". I want to see the CPU usage and process CPU usage for the NAM. You can downgrade the NAM using the same procedure you used to upgrade it.
06-01-2007 12:03 AM
Oh sorry, You mean the Linux (unix) command top. ;-)
Is there any way to hack directly on this Red Hat Linux box? This could make life much easier then those "?$% Cisco commands on this NAM box.
Thx,
bt
Here you go:
top
top - 10:01:48 up 7 days, 19:01, 2 users, load average: 1.08, 1.05, 1.01
Tasks: 56 total, 1 running, 54 sleeping, 0 stopped, 1 zombie
Cpu(s): 48.0% us, 2.4% sy, 0.0% ni, 49.4% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 969244k total, 590952k used, 378292k free, 140696k buffers
Swap: 0k total, 0k used, 0k free, 56028k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
363 root 15 0 466m 320m 1772 S 99.9 33.9 11210:17 rmond
452 www 15 0 12788 9680 3720 S 3.9 1.0 6:13.31 httpd
1 root 15 0 1408 484 428 S 0.0 0.0 0:00.36 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.39 migration/0
3 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:03.56 desched/0
5 root RT 0 0 0 0 S 0.0 0.0 4:45.20 migration/1
6 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
7 root 5 -10 0 0 0 S 0.0 0.0 0:02.02 desched/1
8 root -2 -20 0 0 0 S 0.0 0.0 0:00.33 events/0
9 root -2 -5 0 0 0 S 0.0 0.0 0:00.27 events/1
10 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
12 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
13 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/1
15 root 15 0 0 0 0 S 0.0 0.0 0:00.16 pdflush
16 root 15 0 0 0 0 S 0.0 0.0 0:02.78 pdflush
17 root 16 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
18 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
19 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 aio/1
20 root 17 0 0 0 0 S 0.0 0.0 0:00.42 kseriod
21 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
22 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 ata/1
23 root 15 0 0 0 0 S 0.0 0.0 0:04.18 kjournald
125 root 15 0 1448 692 488 S 0.0 0.1 0:07.90 scpd
135 root 15 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
139 root 15 0 0 0 0 S 0.0 0.0 0:00.31 kjournald
165 bin 16 0 1500 420 344 S 0.0 0.0 0:00.00 portmap
250 root 16 0 1448 552 464 S 0.0 0.1 0:01.80 syslogd
258 root 16 0 1408 440 380 S 0.0 0.0 0:00.00 klogd
267 root 16 0 1628 568 492 S 0.0 0.1 0:00.00 inetd
295 root 17 0 1680 752 556 S 0.0 0.1 0:00.26 smartd
313 daemon 16 0 1440 536 468 S 0.0 0.1 0:00.00 atd
321 root 16 0 1496 640 540 S 0.0 0.1 0:00.00 cron
06-01-2007 12:04 AM
Had to cut top in parts since this board only allows 4000 char.... :-(
326 root 16 0 1408 440 384 S 0.0 0.0 0:00.00 getty
330 root 16 0 0 0 0 Z 0.0 0.0 0:00.00 bash
367 root 25 0 2312 1232 936 S 0.0 0.1 0:15.48 rd_wd
388 root 15 0 9728 5772 3000 S 0.0 0.6 0:00.87 httpd
390 root 16 0 9872 880 640 S 0.0 0.1 1:27.75 polld
393 root 16 0 1592 836 612 S 0.0 0.1 0:05.76 swpolld
396 www 16 0 11284 8176 3728 S 0.0 0.8 6:27.69 httpd
397 www 15 0 12848 9752 3740 S 0.0 1.0 6:34.43 httpd
398 www 16 0 12996 9908 3744 S 0.0 1.0 6:50.41 httpd
399 www 16 0 12852 9764 3736 S 0.0 1.0 6:59.18 httpd
451 www 15 0 12928 9864 3756 S 0.0 1.0 6:43.36 httpd
481 root 16 0 11912 2348 832 S 0.0 0.2 1:52.86 trendd
490 root 19 0 1444 516 428 S 0.0 0.1 0:00.00 configd
491 root 21 0 2972 1500 852 S 0.0 0.2 0:00.00 cli
494 root 25 0 2236 1132 912 S 0.0 0.1 1:05.63 config_monitor
977 www 23 0 11852 8668 3656 S 0.0 0.9 6:05.12 httpd
1150 www 16 0 12788 9660 3712 S 0.0 1.0 6:45.39 httpd
1151 www 15 0 11692 8604 3740 S 0.0 0.9 6:24.01 httpd
18184 www 15 0 12788 9576 3628 S 0.0 1.0 6:19.98 httpd
30802 root 25 0 1648 480 400 S 0.0 0.0 0:00.00 sleep
30833 root 25 0 1648 480 400 S 0.0 0.0 0:00.00 sleep
30842 www 22 0 1800 836 660 R 0.0 0.1 0:00.00 top
06-01-2007 05:22 AM
login: root
Password:
Maintenance image version: 2.1(3)
xxx:xxx@10.128.115.50/Cisco/c6svc-nam.3-5-1b.bin.gz
Downloading the image. This may take several minutes...
ftp://xxx:xxx@10.128.115.50/Cisco/c6svc-nam.3-5-1b.bin.gz (62642K)
/tmp/upgrade.gz [########################] 62642K | 26416.90K/s
64145977 bytes transferred in 2.37 sec (26415.61k/sec)
Upgrade file ftp://xxx:xxx@10.128.115.50/Cisco/c6svc-nam.3-5-1b.bin.gz is downloaded.
Upgrading will wipe out the contents on the storage media.
Do you want to proceed installing it [y|N]: y
Proceeding with upgrade. Please do not interrupt.
If the upgrade is interrupted or fails, boot into
Maintenance image again and restart upgrade.
Creating NAM application image file...
Initializing the application image partition.
This process may take several minutes...
Applying the image, this process may take several minutes...
After 1 hour i have this result:
AR05Z1#sho module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 6 Firewall Module WS-SVC-FWM-1 SAD09300528
2 8 Intrusion Detection System WS-SVC-IDSM-2 SAD0944029J
3 8 Network Analysis Module WS-SVC-NAM-2 SAD094503E8
5 8 8 port 1000mb GBIC Enhanced QoS WS-X6408A-GBIC SAL09295QPJ
7 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL0942342C
9 8 FRU type (0x6003, 0x3F8(1016)) WS-X6708-10GE SAL1101D5F1
11 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL09326WE5
12 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL09454QAD
13 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL094236UE
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0014.a90c.9e32 to 0014.a90c.9e39 3.0 7.2(1) 2.3(2) Ok
2 0015.c694.f794 to 0015.c694.f79b 6.0 7.2(1) 5.0(2) Ok
3 0012.80f1.9118 to 0012.80f1.911f 4.0 7.2(1) 8.3(0.156)RO PwrDown
5 0014.f253.53f0 to 0014.f253.53f7 3.1 5.4(2) 8.3(0.156)RO Ok
7 0013.c42e.b648 to 0013.c42e.b64b 4.4 8.1(3) 12.2(18)SXD7 Ok
9 0018.b966.ea78 to 0018.b966.ea7f 1.3 Unknown Unknown PwrDown
11 000f.2429.0a9c to 000f.2429.0acb 1.5 12.2(14r)S5 12.2(18)SXD7 Ok
12 0015.f997.b6c4 to 0015.f997.b6f3 2.3 12.2(14r)S5 12.2(18)SXD7 Ok
13 0015.c62c.4828 to 0015.c62c.4857 2.3 12.2(14r)S5 12.2(18)SXD7 Ok
06-01-2007 07:12 AM
With:
conf t
power enable module 3
end
hw-module module 3 reset cf:1
I can get back in the maintenance image.
root@AR05NM01.codemasters.nl#show log upgrade
Upgrading the line card on Thu Apr 26 14:13:13 UTC 2007
Downloaded upgrade image ftp://xxx:xxx@10.128.115.50/Cisco/c6svc-nam.3-6-1.bin.gz
Extracted the downloaded file
Proceeding with image upgrade.
Thu Apr 26 14:13:30 2007 : NAM upgrade: type 1, imageFile /tmp/cdisk.gz
Thu Apr 26 14:13:30 2007 : T: 70820477, E: 464814, I: 70820429
Thu Apr 26 14:13:30 2007 : Creating NAM application image file...
Thu Apr 26 14:13:31 2007 : startNAMAppUpgrade:Image: /tmp/cdisk.gz
Thu Apr 26 14:13:31 2007 : startNAMAppUpgrade:Root partition: /dev/hdc1
Thu Apr 26 14:13:31 2007 : startNAMAppUpgrade:Install type: 1
Thu Apr 26 14:13:31 2007 : Partition '/dev/hdc1' unmounted.
Thu Apr 26 14:13:31 2007 : Initializing the application image partition.
Thu Apr 26 14:13:31 2007 : This process may take several minutes...
Thu Apr 26 14:14:15 2007 : Ext2/3 FileSystem created on '/dev/hdc1'.
Thu Apr 26 14:14:15 2007 : Partition '/dev/hdc1' mounted.
Thu Apr 26 14:14:15 2007 : Applying the image, this process may take several minutes...
Thu Apr 26 14:14:15 2007 : Directory changed to '/mnt/hd'.
Thu Apr 26 14:14:24 2007 : Performing post install, please wait...
Thu Apr 26 14:14:24 2007 : File /mnt/hd//usr/local/nam/bin/post-install copied to /tmp/post-install.
Thu Apr 26 14:14:24 2007 : Directory changed to '/tmp'.
Thu Apr 26 14:15:17 2007 : Directory changed to '/tmp'.
Application image upgrade complete. You can boot the image now.
Partition upgraded successfully
Upgrading the line card on Fri Jun 1 12:11:17 UTC 2007
Downloaded upgrade image ftp://xxx:xxx@10.128.115.50/Cisco/c6svc-nam.3-5-1b.bin.gz
Extracted the downloaded file
Proceeding with image upgrade.
Fri Jun 1 12:11:34 2007 : argv1 = 0, argv2 = 0, argv3 = 2, argv4 = 1
Fri Jun 1 12:11:34 2007 : T: 65522544, E: 462621, I: 65522496
Fri Jun 1 12:11:34 2007 : Creating NAM application image file...
Fri Jun 1 12:11:34 2007 : partition: /dev/hdc1
Fri Jun 1 12:11:34 2007 : startNAMAppUpgrade:Image: /tmp/cdisk.gz
Fri Jun 1 12:11:34 2007 : startNAMAppUpgrade:Device: /dev/hdc1
Fri Jun 1 12:11:34 2007 : startNAMAppUpgrade:Install type: 1
Fri Jun 1 12:11:34 2007 : Partition '/dev/hdc1' unmounted.
Fri Jun 1 12:11:35 2007 : Initializing the application image partition.
Fri Jun 1 12:11:35 2007 : This process may take several minutes...
Upgrading the line card on Fri Jun 1 16:05:27 UTC 2007
Downloaded upgrade image ftp://xxx:xxx@10.128.115.50/Cisco/c6svc-nam.3-6-1.bin.gz
Extracted the downloaded file
Proceeding with image upgrade.
Fri Jun 1 16:05:39 2007 : NAM upgrade: type 1, imageFile /tmp/cdisk.gz
Fri Jun 1 16:05:39 2007 : T: 70820477, E: 464814, I: 70820429
Fri Jun 1 16:05:39 2007 : Creating NAM application image file...
Fri Jun 1 16:05:39 2007 : startNAMAppUpgrade:Image: /tmp/cdisk.gz
Fri Jun 1 16:05:39 2007 : startNAMAppUpgrade:Root partition: /dev/hdc1
Fri Jun 1 16:05:39 2007 : startNAMAppUpgrade:Install type: 1
Fri Jun 1 16:05:40 2007 : Partition '/dev/hdc1' unmounted.
Fri Jun 1 16:05:40 2007 : Initializing the application image partition.
Fri Jun 1 16:05:40 2007 : This process may take several minutes...
06-01-2007 08:09 AM
I'm not sure I understand this post. It looks like you went back to 3.6(1)? Given your CPU stats and rmond taking up so much time, you should probably open a TAC service request so that development can get engaged if necessary. We have seen at least one other report of slow 3.6 response time, but that was attributed to a lot of VLAN spanning and voice monitoring Since you said nothing has changed in terms of data collection, this might be a new problem with 3.6.
06-04-2007 01:28 AM
No, you didn't understand my post. At the moment the NAM application image is dead. When booting in the application image the 6513 powers it off!!! Can only get access back to the NAM module to power it on and reboot it back in maintenance image.
This morning I have tried (successful) to use the option --install.
upgrade ftp://xxx:xxx@10.128.115.50/Cisco/c6svc-nam.3-5-1b.bin.gz --install
Now the application images is back on-line, BUT I have no idea with witch password I cannot logon. Both default password as our 'old' passwords are not working!!! So I can't logon this image...
I can still going back into the maintenance images so I will try to do another upgrade. I have to stress here that we having this gear in a live environment and not a 'Lab Test' setup or something...
Sorted the password problem. Used this command> clear module pc-module 3 password
06-04-2007 02:27 AM
All sorted.
This 3.5(1) version is as fast as the 3.4 version on the other 6513.
My advise here, Stay away from 3.6 and wait for a patch or update.
Regards,
bt
06-08-2009 03:18 AM
.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide