UCS 200 have CIMC issues shows address but does not respond to http requests

Unanswered Question
Apr 29th, 2010
User Badges:
  • Cisco Employee,

ok


i have 5  usc machines and doing the bios updates and CIMC updates  on 4 machine no issues . for some reason on the last machine it waned to be a pain .. i did the bios update just like all the other machines that are working and online ..  and then rebooted after that was complete i went into the CIMC console  web page of the system ( . like the other ones . went into the firmware management page . told the system to update the firmware . it took just the same amount of time as the other 4 machines . i watched it and it said complete with no issues .. .. so i told it to activate the new firmware that it downloaded to the system . and now it will not respond over IP anymore . not sure where to go and what to do next . do i need to call tech support and request a new machines ? .. is it dead ...


i have powered off the server . rebooted ....  and then booting up if does show that the CIMC does have an ipaddress on boot up . so i assume that it is working . i try and ping it from both my laptop and desktop . ( one wired and one wireless on different segments and they both respond but to no avail it still does not respond to http access ..




anyone have any ideas how to get it back online




thanks

ryan


  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (4 ratings)
Loading.
ryfoley Thu, 04/29/2010 - 11:38
User Badges:
  • Cisco Employee,

is there any way to push the old firmware back to make it work again ?


command line interface . or something to allow me to do that .

Ad van Esch Mon, 03/30/2015 - 04:12
User Badges:

I have the same problem is with C220 version1,5(4)

is this problem still not resolved,

is a reboot of the server the only possible solution in this?

 

Keny Perez Mon, 03/30/2015 - 05:03
User Badges:
  • Blue, 1500 points or more

Ad, so you can still ping your server but have no access to CIMC either SSH or Web? If so, there is this bug that talks about the issue https://tools.cisco.com/bugsearch/bug/CSCun88303/?reffering_site=dumpcr  and as you can see the fix is in 2.0(3i ) and above (just released on March 13th) 

If this answers your question, please mark it as such for future users to see you found the information you were looking for.

 

-Kenny

jmeier10510 Fri, 04/17/2015 - 03:47
User Badges:

Looks like I've been bit by this bug.  One of my C240 M3 servers with 1.5(4) has increased fan speed, pings good to CIMC IP address, no access to web interface or SSH.  

Step 11 of the process makes me nervous.  I assume there's no reason why I have to update the HDD firmware.  It seems that's a separate and optional step.  The previous steps in the directions appears to update the firmware of everything else, so if there's truly no requirement to update the HDD firmware, I'd just assume avoid doing it.   No need to tempt fate?  ;-)

Step 11:

If you want to update the firmware of the hard disk of a server, click Update HDD Firmware. A window displays a list
of hard disk drives on the server that support new firmware. Hard disk drives that do not support firmware upgrades are
not listed.
Updating the firmware of the hard disk drive could result in data loss. Cisco recommends that you take a
complete system backup prior to updating the firmware.
Important
a) To update the firmware of all the hard disks, click Update All.
With this option, HDDs with the latest firmware installed are not updated.
b) To update a specific HDD, choose the HDD and click Update.

jmeier10510 Fri, 04/17/2015 - 10:38
User Badges:

I saw your reply about the hard drives in my email but I don't see it here on the web page.  Strange.

Anyway, I decided to update the C240 M3 that is the standby (Update All, but not updating the HDD) because it has no phones or trunks registered to it and so far the process has been a nightmare.  First, it took forever to get Java to work for the virtual KVM.   I had to remove the latest version and install a version 7.  I spent most of my time just troubleshooting to see why Java wouldn't load the KVM.  Then, once I got the update going, I lost contact with the KVM as expected (says so in the HUU user guide, but it says I should be able to reconnect and continue the process)....but no matter what I tried, I could never get the remote KVM to load again.  I finally had get my local guy to plug in a monitor and keyboard to the back of the server.  The HUU was still up and the update status of the CIMC was failed.  I had him click Update All again to try it one more time.  Waiting to see what happens now.  This is extremely frustrating.  

jmeier10510 Fri, 04/24/2015 - 02:09
User Badges:

I was finally able to get it to work correctly with Java 7 update 21 on another machine.  After firmware version 2.0(3i) was installed on the CIMC, I can access the KVM with the latest Java version, which is great.  Java 8 update 45 works now.

ryfoley Fri, 04/30/2010 - 11:28
User Badges:
  • Cisco Employee,

please delete question as i figured out my own issue .



thanks

mrlucmorin Tue, 08/31/2010 - 14:18
User Badges:

I'm seeing the same issue here.


We have two UCS C200M1 servers, both of which show a proper setup when going into the CIMC setup (press F8 at boot time), but yet, I can't connect to the management port with a browser.


Any pointers on what to check for ?


Thank you


Luc

Jason Aarons Fri, 06/17/2011 - 19:15
User Badges:
  • Bronze, 100 points or more

Any fix?


I have 6 C210M2, 5 of them I can ssh/https to ip. 


The trouble one I can ssh to, but no response on https. How do you restart the web server on the CIMC?



ucs-c2xx# show version

Firmware Version

--------------------

1.3(1d)

ucs-c2xx#

ucs-c2xx# show http detail

HTTP Settings:

    HTTP Port: 80

    HTTPS Port: 443

    Timeout: 1800

    Max Sessions: 4

    Active Sessions: 0

    Enabled: yes

ucs-c2xx# show http

HTTP Port  HTTPS Port Timeout  Active Sessions Enabled

---------- ---------- -------- --------------- -------

80         443        1800     0               yes

ucs-c2xx#

abbharga Tue, 02/28/2012 - 09:19
User Badges:
  • Silver, 250 points or more

You can do the following via the SSH


01# scope http

01 /http # set enabled no

Warning: setting "enabled" to "no" will disconnect all

existing http connections and will disable login via WebUI.

01 /http *# commit

01 /http # set enabled yes

01 /http *# commit

01 /http #


./Abhinav

joshua.gertig Tue, 02/28/2012 - 09:38
User Badges:

hmm... I can't seem to issue those commands. Is there a command I need to

put before entering: scope http?


Sorry, I'm a rookie when it comes to UCS (we just moved to UCS from

stand-alone, and a vendor set it all up)...


login as: root

[email protected]'s password:

You have activated Tech Support Mode.

The time and date of this activation have been sent to the system logs.


VMware offers supported, powerful system administration tools. Please

see www.vmware.com/go/sysadmintools for details.


Tech Support Mode may be disabled by an administrative user.

Please consult the ESXi Configuration Guide for additional

important information.


~ # scope http

-ash: scope: not found

~ # scope https

-ash: scope: not found

~ #


Josh Gertig

Voice Communications Analyst - Team Lead

National Rural Utilities Cooperative Finance Corporation

(703) 467-1695 (direct)

(571) 455-9359 (cell)

(703) 467-8976 (fax)




abbharga Tue, 02/28/2012 - 09:44
User Badges:
  • Silver, 250 points or more

Hi Joshua,


Looks like you SSHd' into the ESXi IP address and not the CIMC IP address, can you verify the IP.


./Abhinav

joshua.gertig Tue, 02/28/2012 - 09:51
User Badges:

Yes, that's right. I'm SSH'd to the ESXi address. I can't SSH to the CIMC

address, I don't get any error or anything. It seems like it connects but

it just hangs there, I never see any login prompts or anything for that

matter.


Do you think a re-boot is in order?



screen shot of my attempt to SSH to CIMC:

(Embedded image moved to file: pic23281.gif)


Josh Gertig

Voice Communications Analyst - Team Lead

National Rural Utilities Cooperative Finance Corporation

(703) 467-1695 (direct)

(571) 455-9359 (cell)

(703) 467-8976 (fax)




Attachment: 
abbharga Tue, 02/28/2012 - 09:53
User Badges:
  • Silver, 250 points or more

Yes, so that pretty much matches the behaviour of the bug I mentioned below.


A chassis power re-cycle would be required as a workaround to this issue.


./Abhinav

joshua.gertig Tue, 02/28/2012 - 10:05
User Badges:

Will do this weekend, thanks for your help.

Josh Gertig

Voice Communications Analyst - Team Lead

National Rural Utilities Cooperative Finance Corporation

(703) 467-1695 (direct)

(571) 455-9359 (cell)

ElByte_2 Fri, 06/24/2011 - 13:58
User Badges:

Hello


I have the same issue - the CIMC web interface on one UCS200 is accessible via https, but the second is not. I can ping it, but cannot access the web interface.  Any ideas? Maybe the OP could post his solution?


Thanks!

joshua.gertig Tue, 02/28/2012 - 08:07
User Badges:

We are seeing the same issue on a UCS C210 M2 (1.3(2d)). Last week everything was fine, this week I can't SSH or web to either UCS box. No significant changes were performed on either UCS. I can ping and even open a command prompt and telnet to both boxes on 443 and it connects, so I don't think any firewall related issue has been introduced (our network team also assures us there is no fw element between my PC and the UCS).

Is there a way to restart webservice from the ESXi client or from SSH to the ESXi address (I can SSH to the ESXi address, but it's all linux ). Thanks!

abbharga Tue, 02/28/2012 - 09:15
User Badges:
  • Silver, 250 points or more

We do have a known bug around the CIMC not being accessible.:


CSCtn06772    CIMC - does not respond - cannot access via SSH / HTTP / HTTPS


Symptom:
Cannot access CIMC via SSH / HTTP / HTTPS / Serial over LAN

Conditions:
If system can be accessed via CIMC Serial Console , then system is NOT running into this defect.


Workaround:
Power cycle chassis


./Abhinav

joshua.gertig Thu, 03/01/2012 - 11:31
User Badges:

Hi abbharga,

In reference to your note below "If system can be accessed via CIMC Serial

Console , then system is NOT running into this defect"...

I finally found the console cable for our UCS, the one that plugs into the

front with the DB9 Serial, VGA and two USB ports. But when I connect to

that, I just get the ESXi interface, no CIMC. Is there some other way to

console to CIMC?

Thanks, you have been very helpful!

Josh


Josh Gertig

Voice Communications Analyst - Team Lead

National Rural Utilities Cooperative Finance Corporation

(703) 467-1695 (direct)

(571) 455-9359 (cell)

(703) 467-8976 (fax)




robertkruk Tue, 02/28/2012 - 15:55
User Badges:

yep Powercycling is how I have fixed this in the past

abbharga Tue, 02/28/2012 - 20:05
User Badges:
  • Silver, 250 points or more

Hi Robert,


This fix is avaiable in the current 1.4.x firmware version, post upgrade to the firmware you should not be facing this issue again.


Thanks!

./Abhinav

arrrghhh3 Mon, 08/13/2012 - 11:40
User Badges:

Abhinav,


Do you know this for a fact?


Based on another thread and all the information I can find on Cisco's website on the release notes for CIMC... this bug is NOT resolved.  At least not the bug we are experiencing (CSCtn06772)

joshua.gertig Mon, 08/13/2012 - 11:48
User Badges:

Arrrghhh3,

Just curious, what are you experiencing. We upgraded to 1.4(3)c, which was the latest at the time. It didn't resolve the issue but made it better I would say.

Only one of two UCS servers was affected in the first place. That same server continues to have an issue, where the web interface goes down, but the difference is that we do not loose SSH (CLI) access. So, if needed, we can SSH to the affected server, and restart http (set enabled no ; set enabled yes) and we get the webpage for 10 minutes or so, then it goes out again.

At Cisco Live I spoke to some UCS folks who were familuar with the bug, but had never heard of our symptoms post upgrade. So I was curious if you might be having something similar.

arrrghhh3 Mon, 08/13/2012 - 11:58
User Badges:

Joshua,


Thanks for your post.  My issue is a bit different from yours - our CIMC is completely unresponsive (except for ICMP).  So for whatever reason we can ping it, we can NMAP the IP and it shows port 22, 80, & 443 are all open.  One engineer waited for a good 5 minutes, and evidently part of the CIMC web interface loads... but never provides a login prompt.  SSH never provides a login prompt either - but all of these ports I can connect to with telnet - so they are open and there's no network issues.  All CIMC's are running 1.4(2).


It seems like the only option is to power down all the guest OSes, and then power down the ESXi host.  Unplug all power plugs (probably for ~30 sec) then plug it back in and power everything back on.


This is the "workaround" for the bug, but I would prefer an actual fix.  Based on the thread I linked to, it appears there is one in the works.  Only one of our CIMC servers experienced this issue so far, but I don't want to take a chance and have this happen again.  This is supposed to be an out-of-band management interface - really simple, something that never fails.  With these types of failures, I feel like I need some sort of out-of-band management for the out-of-band management interface...

besahin Thu, 01/03/2013 - 10:05
User Badges:

We also faced a CIMC problem when running on fimrware 1.4(1)..

Thanks to "Arrrghhh3" suggestion of unplugging the chassis and keeping it unplugged for about a minute or so, solved our problem. Very strange.


We also had a great amount of noice coming out of the chassis before we did this unplugging the power. After replugging, the noise was much less as well..


We were having issues with logging into CIMC GUI and we were not getting any response when we were hittin F8 during boot. We were seeing CIMC IP configuration error messages during the boot process.

mohsinsalimvs Thu, 08/13/2015 - 20:28
User Badges:

I had a similar issue. Unable to access using http while I am able to ssh into it.

 

The issue was fixed by following the steps below.


UCSMGMT*# show http detail 
HTTP Settings:
    HTTP Port: 80
    HTTPS Port: 443
    Timeout: 1800
    Max Sessions: 4
    Active Sessions: 0
    Enabled: yes
    HTTP Redirected: yes
UCSMGMT# scope http 
UCSMGMT /http # set enabled no 
Warning: setting "enabled" to "no" will disconnect all 
 existing http connections and will disable login via WebUI.
UCSMGMT /http *# commit 
UCSMGMT /http # exit
UCSMGMT# show http detail 
HTTP Settings:
    HTTP Port: 80
    HTTPS Port: 443
    Timeout: 1800
    Max Sessions: 4
    Active Sessions: 0
    Enabled: no
    HTTP Redirected: yes
UCSMGMT# scope http 
UCSMGMT /http # set enabled yes 
UCSMGMT /http *# commit 
UCSMGMT /http # exit
UCSMGMT# show http detail
HTTP Settings:
    HTTP Port: 80
    HTTPS Port: 443
    Timeout: 1800
    Max Sessions: 4
    Active Sessions: 0
    Enabled: yes
    HTTP Redirected: yes
UCSMGMT# 

salman.shaikh Sun, 01/22/2017 - 09:50
User Badges:

Hi Steven, I had the same issue, and I was able to get in using ssh, but question is if I reboot the CIMC, would that affect any production environment or any VMware, ESXi, etc. will affect? What i know when you reboot CIMC using ssh it will reset Management interface and probably no management access available for 1 or 2 min until management reset, but I want to know during productions hrs if I reboot CIMC using ssh it will affect anything, or we are good to reboot anytime ????

Kirk J Sun, 01/22/2017 - 14:21
User Badges:
  • Cisco Employee,

Greetings.

The CIMC/BMC is an out of band controller.  It stays powered up while the server is powered off, and like wise can be reset while the server is running, and the OS will not be aware of it's restart.

Unless you are utilizing some specific feature of the CIMC such as SOL/serial over Lan, the BIOS,storage, cpus, etc will have no knowledge of the CIMC reset.

You can avoid the CIMC memory leak condition by upgrading the CIMC to HUU 1.43P or higher for your UCS M2 rack servers.  For M3/M4 rack servers, version 2.03i or higher resolves a similar mem leak condition.

As has already been discussed in this thread, a hard power cycle (pull power cords for 30 seconds will reset the CIMC), or if the SSH functionality is still there, you can scope to the CIMC and issue reboot via CLI.

Thanks,

Kirk...

Actions

This Discussion