I was looking for some main topic which would have a list of issues so far found in LMS 4.0. Here are few first minor bumps on the road.
1) Missing Navigation Panel on several pages
2) Device configuration not being backed up even though all devices are supposed to be managed (LMS didn't even try to download the config. In the configuration overview there was just "Total 1" instead of 2.
I would like to add more and more issues into this thread or if one of the moderators could create a separate folder just for this kind of topics I think it would make the search and life much easier.
P.S.: Any hints or fixes are appreciated
Solved! Go to Solution.
Just tested it out on our lab server the navigation pages appear to work fine and using Firefox 4.0 RC2. Verify that you did not have any installation issues during the setup and that you have added LMS server to your trusted sites in IE.
Post the installation logs if adding the lms server to your trusted site or using a different browser does not display the navigation headers. Logs should start with C:\Ciscoworks_install_date.logs or /var/tmp directory.
For problem 2:
Goto Inventory --> Job Browsers --> Device Credential Verification --> Create Job for devices in question.
Goto Inventory --> Job Browsers --> Inventory Collection ---> Create job for both polling and collection.
Get the logs from CSCOpx/log dcmaclient.log and dcmaservice.log along with the IP address or hostname of test device.
Thanks for the suggestion, Martin. However, there is a difference between known issues, and localized problems. I think you're seeing the latter, and those should be analyzed as new issues. For future issues, please start new threads, so we can properly distinguish individual problems.
For the first issue, how did you reproduce this exactly? Is it something you can reliably reproduce? If so, try clearing your browser cache, and see if that corrects the problem.
For the second issue, more information is required. What device type is not being archived? If you go to Configuration > Configuration Archive > Summary, what do you see? If you go to the Inventory > Device Status dashboard, what do you see in the Collection Summary portlet?
Thanks for the replies. The issue with disappearing navigation menu is fine now. (even though I didn't do anything, so I guess it was related to the browser).
The other issue is a bit strange. Devices are now flagged as managed, but LMS can't seem to be able to use any credentials. I mean it reports telnet credentials as incorrect. I have tried several working logins and I have tried to change the protocol usage as well (SSH, Telnet) and nothing seems to work. I remember that we had an issue like this back with 3.2 and it was related to the TACACS prompts. The strange thing is that our Cisco ACS server doesn't even pick up any login attempts. When I use Putty from the LMS server it works all fine (SSH, Telnet ... several different credentials).
I will try to check the TACACS prompts on the LMS server.
If anyone has any suggestions, please share them with me.
Well, if the username/password prompts are unknown to LMS, it won't even try to login. It will just stare at the prompt. You can add any prompt customization to NMSROOT/objects/cmf/data/TacacsPrompts.ini.
That said, if SSH is not working, I don't think this is your problem. I would first go to Inventory > Device Administration > Add / Import / Manage Devices, and edit credentials to make absolutely sure you've entered the correct values. Do not assume that just because • characters are there that the credentials are entered correctly.
If things still fail. Go back to Reports > Inventory > Management Status > Credential Verification and run a report for telnet credentials. At the same time, start a sniffer trace filtering on tcp/23 traffic to the test device. Check the trace after the job fails to see what LMS is seeing and sending.
I have sorted out the TACACS prompts (or at least I would think so ). Here is what we have got in the TacacsPrompts.ini file
USERNAME_PROMPT=username: ,Username: ,
PASSWORD_PROMPT=password: ,Password: ,
So here is what is happening right now. The archiving job fails and the reported reason is "Command failed". (See the attached screenshot "Job result")
I have done some digging and sometimes it might be an issue with job timeouts. The job timeouts have been extended. (See attached screenshots "Job timeout" and "Telnet timeout")
And what is really strange is that if I go into our ACS server and check AAA Accounting I can see the "copy vlan.dat" command being executed by the LMS server. The devices, which are currently in the LMS 4.0 are 3560, 3750 and also 6509.
Also, how does it work with support from Cisco? Do we need to use the forum or is there an official source for support (something like TAC for evaluation period)?
Many thanks for the help,
So after packet sniffing and reconstructing the Telnet stream we can see, instead of pile of packets, the actual commands sent to the device and the replies sent to LMS server.
The IP addresses have been replaced and the hostname as well. You can see X --> Y and vice versa. So the first line says that device sent to LMS server info which you can see on the next line and so forth. The whole data stream has been cut to this most important part and in the end the device says that it cannot write into LMS' TFTP server. So at least we know what the problem is. Any reason why? ( I have checked the IPs before I replaced them and the device really is targeting LMS4.0 server)
A permission denied error implies there was a problem writing to the NMSROOT/tftpboot directory. LMS should have created the file in that directory prior to copying it, but check to make sure casuser has full control, and check the output of "netstat -a -n -o -b" to make sure it is crmtftp bound to udp/69.
Please see the attached screenshot of the permissions. Nobody touched them so I guess it is setup the way LMS did during the install. Casuser is a member of casusers group and as you can see that has full control. In the top you can see crmtftp.exe is on UDP:69
Yeah, this looks okay. I wonder if perhaps the problem is being caused by some host-base IPS or firewall software. Gather a new sniffer trace filtering on all UDP traffic between the device and the server when doing a config collection. It looks like the first udp/69 packet is working, but the subsequent data stream may not be.
Sorry for the delay in reply, I got overwhelmed with lots of stuff in the last week. I have got the TFTP packet sniffing results. And LMS doesn't seem to send any packets back. See for yourselfs in the screenshot. We have also came across some issues with job statuses (another screenshot), that was "fixed" by rebooting the server (not just daemon manager service .. we did that and that didn't help).
The HW spec is based on Windows 2003 Server LMS4.0 license for 1500 devices (we currently have 3.2 for 1500 devices). The reason for this is that we wanted to move to LMS 4.0 asap, but as you can see we are kinda struggling
The AntiVirus is switched off on the LMS server and there is nothing on the network (security-wise) what could cause the issues. LMS 3.2 is doing just fine (using the same path and traversing the same devices as 4.0)
This indicates that udp/69 is blocked between the device and the server. Run the packet capture from the LMS server itself (LMS includes a built-in packet capture utility in Device Center). If you see the udp/69 packets arriving on the LMS server, check to see if Windows Firewall is enabled. If so, disable it. If Windows Firewall is disabled, check for other security software on the server (IPS-like software). Disable those suites and see if TFTP works.
I have looked for the packet capture utility and I can't find it anywhere, not in the user guides, not in the site map. All I found in the user guides is that LMS has packet capture utility in the Troubleshooting Tools.
Could you please tell me where I can find it?
It's under Inventory > Tools > Device Center, according to page 372 of http://www.cisco.com/en/US/docs/net_mgmt/ciscoworks_lan_management_solution/4.0/user/guide/monitoring_troubleshooting/UserGuideforMNT.pdf
I found the packet capture. To be honest I did not expect that an icon of a star is actually a menu button. Nevertheless here is the packet capture result.
LMS is basically saying that it cannot save the file. Anti virus software is disabled on the server (3.2 runs with it enabled no problem) and there is no security device in the way between LMS server and the device.
Troubleshooting this may become difficult as you cannot share the full details of your network. At this point, I need to see the dcmaservice.log after you enable ArchiveMgmt Service debugging under Admin > System > Debug Settings > Config and Image Management Debugging settings and re-run the failing job.
I have tried to send you the log three times as a PM, but it always failed with errors saying that I didn't fill in the username, left blank subject and didn't enter anything in the body. I guess this is caused by the length of the log. I don't want to upload it on some external file sharing website. Could you please send me an email I can send it to?
Thank you very much for taking the time Joe.
Message was edited by: Martin Smid
If the logs cannot be posted to the community, you will need to move this into a formal TAC service request. You can use the "Open Service Request" action in the Actions panel. Because this is an eval, it may only be a P4 issue, but you will be able to share the logs with the TAC engineer.
The Command Failed indicates that something prevented the copy of vlan.dat from happening back to the LMS server. This copy will be done using TFTP only. Therefore, make sure UDP port 69 and all ephmeral UDP ports are open between the device and the LMS server. Try doing the copy operation manually to confirm it works.
If this is an LMS 4.0 eval, there is no TAC support. You can get best-effort support here on CSC.
Thanks Joe so far. The reason I asked about TAC is that the forum is pretty much voluntary so I just wanted to take the pressure of.
All these devices are in LMS 3.2, which we are currently using and are backed up with no issues what so ever (startup, running, vlan.dat, via SSH, Telnet, SNMP you name it). Between the LMS server and the switches is just another switch (for our VMware farm) and then the 6509 (which I am trying to back up as well). Bottom line; there is no firewall. I will try to get the packet sniffer in today and get some more info.
What worries me is the fact that 3.2 has no issues with them. They are both VM builds and I think they even sit on the same host and even if they don't, there are no config differences between our VM hosts.
I have copied the vlan.dat via putty just to make 100% sure it works. And it does work if I copy it to a TFTP on my PC, but it does not work if I punch in the IP address of the LMS server. I guess the LMS server opens the connection only when it needs to (because it didn't work manually even from the 3.2, yet it can backup vlan.dat files)
Here is a brief output from "netstat -a" conserning UDP and TFTP. I can send you the whole lot in a PM if you want me to.
UDP LANMAN-1:tftp *:*
UDP LANMAN-1:snmp *:*
UDP LANMAN-1:snmptrap *:*
This has been now classified as a bug ID CSCts98495.
You can access CSCts98495 from below link :
The workaround for the timebeing is to copy crmtftp.exe from previous version (3.2) into 4.0. You will need to obviously stop the TFTP service (not the whole LMS daemon)
Thank you all for your help with this!