Service-module failed state

ermalgjylameti · ‎03-06-2015

Hello all,

I have been researching this issue for a while now to no avail. I have a 2821 with a NME-16ES-1G-P module. Up to last week it was working just fine and than after a router reload it stopped working or so it looks like. I am able to get prompted for a user and password when i type "service-module gig 1/0 session" but neither tacacs or local credentials work. Card has been reseated several times, router has been reloaded several times. When trying to get to rommon mode by typing "service-module gig1/0 password-reset" it looks like it is going thru but than nothing happens. Below are some of the show commands from the router.

Please let me know if there is anything I might be missing. This card or the router is not under warranties

R3_2821#service-module gig1/0 status
Service Module is Cisco GigabitEthernet1/0
Service Module supports session via TTY line 66
Service Module is failed
Service Module reset on error is disabled
Service Module heartbeat-reset is enabled
Service Module status is not available

R3_2821#

R3_2821#sho diag 1
Slot 1:
        NME-16ES-1G-P: EtherSwitch SM 16 10/100T PoE + 1 GE Port adapter, 1 port
        Port adapter is analyzed
        Port adapter insertion time 01:18:34 ago
        EEPROM contents at hardware discovery:
        Hardware Revision        : 1.0
        Top Assy. Part Number    : 800-25009-01
        Board Revision           : B0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : FOC10163MWD
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Product (FRU) Number     : NME-16ES-1G-P
        Version Identifier       : V01
        Base MAC Address         : 0017.e004.4e80
        MAC Address block size   : 128
        EEPROM format version 4
        EEPROM contents (hex):
          0x00: 04 FF 40 04 A2 41 01 00 C0 46 03 20 00 61 B1 01
          0x10: 42 42 30 80 00 00 00 00 02 03 C1 8B 46 4F 43 31
          0x20: 30 31 36 33 4D 57 44 03 00 81 00 00 00 00 04 00
          0x30: CB 8E 4E 4D 45 2D 31 36 45 53 2D 31 47 2D 50 20
          0x40: 89 56 30 31 20 D9 02 40 C1 CF 06 00 17 E0 04 4E
          0x50: 80 43 00 80 FF FF FF FF FF FF FF FF FF FF FF FF
          0x60: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
          0x70: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF

R3_2821#

R3_2821#service-module gig1/0 shutdown
Shutdown will not work since module is either in bootloader or IOS has not received
heartbeat from module yet. Please check the module status using
"service-module GigabitEthernet1/0 status" CLI

R3_2821#

R3_2821#
R3_2821#service-module gig1/0 password-reset
Do you want to proceed with password reset process?[confirm]
Starting password reset process...
Wait for 50 secs for password reset process to complete
R3_2821#

Password reset process is complete...

R3_2821#

Regards

Ermal G

Charles Hill · ‎03-06-2015

Hello Ermal G,

What IOS is on the 2821?

Do you see any errors or any information that could be related to the NME-16ES-1G-P module at startup?

ermalgjylameti · ‎03-06-2015

Looking at the console during the startup I do not see any errors at all.

flash:c2800nm-adventerprisek9_ivs-mz.151-4.M9.bin

Regards

Ermal

Charles Hill · ‎03-06-2015

Take look at this link. It looks very similar to what you are experiencing.

http://www.netcontractor.pl/blog/?p=181

Same output:

ROUTER#service-module gigabitEthernet 1/0 status Service Module is Cisco GigabitEthernet1/0 Service Module supports session via TTY line 66 Service Module is failed Service Module reset on error is disabled Service Module status is not available

ermalgjylameti · ‎03-07-2015

Cehill, thank you for the link. Unfortunately I have no access at all to the service-module command line. SSH-telnet does not work and also when I try to get to it from the router it does not let me get in it.

Marcelo Morais · ‎03-06-2015

Hi ermalgjylameti,

try the following

line con 66
transport input all

service-module g1/0 session

Hope this helps.

ermalgjylameti · ‎03-06-2015

Is this what you were thinking?

000031: *Mar 6 22:06:37.888 EST: %HA_EM-2-LOG: Configuration-Begin: Admin has entered device configuration
R3_2821(config)#line con 66
^
% Invalid input detected at '^' marker.

R3_2821(config)#line con
R3_2821(config)#line console ?
<0-0> First Line number

Thank you
Ermal

Marcelo Morais · ‎03-06-2015

Hi,

mistyping ... its 'vty' not 'con'

line vty 66
transport input all

service-module g1/0 session

Regards.

ermalgjylameti · ‎03-06-2015

I was able to make the change to the VTY line but still unable to do anything with the service-module.

Ermal

Marcelo Morais · ‎03-06-2015

Ok.

in other words, if you enter the following commands:

service-module g1/0 session clear

service-module g1/0 session

You have 'credential problem' not 'Connection refused by remote host'?

Regards.

ermalgjylameti · ‎03-07-2015

This is not an issue with authentication. As I said before it looks like the module is in a failed state. None of the devices connected to the module ports are reachable. I was able to authenticate with no issue until the last reload. (not power outage but reload).

Leo Laohoo · ‎03-07-2015

Check the interface (Gi 1/0) and look at Layer 1 status. Is it up or is it down?

Check CDP. Do you see the module there?

I am suspecting the switch module had a crash and, maybe, the IOS is corrupt.

ermalgjylameti · ‎03-07-2015

Gig1/0 is in a UP UP state. I have tried to do shut no shut with no change. I do not see anything coming off gig1/0 on CDP neighbors.
I do see the module when doing a "show diag 1"

I also do think that the IOS somehow got corrupted during boot up. When I try to get into rommon mode by typing "service-module gig1/0 password-reset" from the router id does say that it take 50sec for the proccess to go thru and than it acknowledges that it was completed but than when I try to get back in to the module thru the router it still prompts me for a username and password.

Leo Laohoo · ‎03-07-2015

Gig1/0 is in a UP UP state. I have tried to do shut no shut with no change. I do not see anything coming off gig1/0 on CDP neighbors.

Ok, this confirms that the module is in ROMmon.

You may have to go through the process of recovering the module via ROMmon. If things don't go well, you may also have to consider reloading the module several times until you are able to do so. This is the only thing I can recommend short to getting the module RMA-ed.

ermalgjylameti · ‎03-07-2015

I have reloaded the module and the router several times these past few days but I still cant get to the switch. Unfortunately this is for my home router. Bought off ebay back in december so no coverage on it. Below is how I am trying to get to rommon mode while consoled in the router. Pls let me know if there is any step that I am missing.

R3_2821#service-module gig1/0 password-reset
Do you want to proceed with password reset process?[confirm]
Starting password reset process...
Wait for 50 secs for password reset process to complete
R3_2821#

Password reset process is complete...

R3_2821#service-module gig1/0 ses
R3_2821#service-module gig1/0 session
Trying 192.168.80.1, 2066 ... Open
uthorized access only!

User Access Verification

Username: