7606 Supervisor Redundancy

Unanswered Question
Jan 1st, 2009
User Badges:

Good evening everyone. Well after doing an IOS upgrade from s72033-advipservicesk9_wan-mz.122-33.SRA5 to c7600s72033-advipservicesk9-mz.122 SRB4 (Highest level approved by our org) on our 7606. The standby supervisor

is in a "Current Software state = STANDBY COLD" state vs HOT. Here is the log after the standby SUP was reset with new IOS, The primary SUP is already running the new IOS when this SUP was reset. I have included results from "sh redundancy" also.

Please let me you what you think might be the issue...


Thanks All!


Dec 31 16:02:10: %SNMP-5-MODULETRAP: Module 5 [Down] Trap

Dec 31 16:02:09: %OIR-SP-3-PWRCYCLE: Card in module 5, is being power-cycled (Mo

dule reset)

Dec 31 16:02:10: %RED_MODE-3-RED_MODE_MSG_NOT_OK: Red Mode ISSU msg type (100) i

s not ok

Dec 31 16:02:10: %PFREDUN-SP-6-ACTIVE: Standby processor removed or reloaded, ch

anging to Simplex mode

Dec 31 16:04:04: %ISSU-SP-3-INCOMPATIBLE_PEER_UID: Peer image (c7600s72033_sp-AD

VIPSERVICESK9-M), version (12.2(33)SRB4) on peer uid (5) is incompatible

Dec 31 16:05:03: %PFREDUN-SP-4-INCOMPATIBLE: Defaulting to RPR mode (Runtime inc

ompatible)

Dec 31 16:05:08: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fab

ric in slot 5.

Dec 31 16:05:08: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in

slot 5 became standby

Dec 31 16:05:09: %DIAG-SP-6-RUN_MINIMUM: Dec 31 16:05:09: %DIAG-SP-6-DIAG_OK: Module 5: Passed Online Diagnostics

Dec 31 16:05:10: %OIR-SP-6-INSCARD: Card inserted in slot 5, interfaces are now

online

Dec 31 16:07:34: %PFREDUN-SP-6-ACTIVE: Standby initializing for RPR mode

Dec 31 16:07:34: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ens

ure console debugging output.

Dec 31 16:07:35: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup configuration to

the standby Router.

Dec 31 16:07:39: %RF-SP-5-RF_TERMINAL_STATE: Terminal state reached for (RPR)


sh red

Redundant System Information :

------------------------------

Available system uptime = 19 weeks, 4 days, 27 minutes

Switchovers system experienced = 1

Standby failures = 5

Last switchover reason = user initiated


Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = rpr

Maintenance Mode = Disabled

Communications = Up


Current Processor Information :

-------------------------------

Active Location = slot 6

Current Software state = ACTIVE

Uptime in current state = 2 days, 1 hour, 11 minutes

Image Version = Cisco IOS Software, c7600s72033_rp Software (c7

600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRB4, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2008 by Cisco Systems, Inc.

Compiled Thu 24-Jul-08 01:47 by prod_rel_team

BOOT = sup-bootdisk:c7600s72033-advipservicesk9-mz.122

-33.SRB4.bin,1;sup-bootdisk:s72033-advipservicesk9_wan-mz.122-33.SRA5.bin,1;

CONFIG_FILE =

BOOTLDR =

Configuration register = 0x2102


Peer Processor Information :

----------------------------

Standby Location = slot 5

Current Software state = STANDBY COLD

Uptime in current state = 1 day, 3 hours, 4 minutes

Image Version = Cisco IOS Software, c7600s72033_rp Software (c7

600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRB4, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2008 by Cisco Systems, Inc.

Compiled Thu 24-Jul-08 01:47 by prod_rel_team

BOOT = sup-bootdisk:c7600s72033-advipservicesk9-mz.122

-33.SRB4.bin,1;sup-bootdisk:s72033-advipservicesk9_wan-mz.122-33.SRA5.bin,1;

CONFIG_FILE =

BOOTLDR =

Configuration register = 0x2102

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Giuseppe Larosa Fri, 01/02/2009 - 00:33
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Robert,


Configured Redundancy Mode = sso

Operating Redundancy Mode = rpr


for this reason the state is STANDBY COLD


http://www.cisco.com/en/US/docs/routers/7600/ios/12.2SR/configuration/guide/nsfsso.html#wp1119694


What we can see from logs is:


Dec 31 16:02:10: %RED_MODE-3-RED_MODE_MSG_NOT_OK: Red Mode ISSU msg type (100) i

s not ok

Dec 31 16:02:10: %PFREDUN-SP-6-ACTIVE: Standby processor removed or reloaded, ch

anging to Simplex mode


Dec 31 16:04:04: %ISSU-SP-3-INCOMPATIBLE_PEER_UID: Peer image (c7600s72033_sp-AD

VIPSERVICESK9-M), version (12.2(33)SRB4) on peer uid (5) is incompatible

Dec 31 16:05:03: %PFREDUN-SP-4-INCOMPATIBLE: Defaulting to RPR mode (Runtime inc

ompatible)


for some reasons the active supervisor didn't like the peer and so now they are in RPR mode.


I would try to reboot again the standby supervisor.



the error message decoder says for the first message :


1. %RED_MODE-3-RED_MODE_MSG_NOT_OK: Red Mode ISSU msg type ([dec]) is not ok


The redundancy mode ISSU message type has not negotiated correctly.


Recommended Action: Copy the error message exactly as it appears on the console or in the system log. Research and attempt to resolve the error using the Output Interpreter https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl. Issue the show message types command to gather data that may help identify the nature of the error. Also perform a search of the Bug Toolkit http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs. If you still require assistance, open a case with the Technical Assistance Center via the Internet http://www.cisco.com/cgi-bin/front.x/case_tools/caseOpen.pl, or contact your Cisco technical support representative and provide the representative with the gathered information.


Related documents- No specific documents apply to this error message.


How have you resetted the standby supervisor ?



Hope to help

Giuseppe



robert.callahan Fri, 01/02/2009 - 19:27
User Badges:

Thanks for the sugestions! I have reset the standby a couple of times now. I came to he same conclusion as as as why I was in standby cold. The log also mentions incompatible image... Maybe it is because the old image was not deleted and it does not support ISSU? Anyone else have thoughts on this idea?

Giuseppe Larosa Sat, 01/03/2009 - 02:55
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Robert,

the strange aspect is the standby Sup has the right image see from your first post:


Dec 31 16:04:04: %ISSU-SP-3-INCOMPATIBLE_PEER_UID: Peer image (c7600s72033_sp-AD

VIPSERVICESK9-M), version (12.2(33)SRB4) on peer uid (5) is incompatible


So I suspect it is a question of timing.

connect the console to the standby Supervisor and try to see what it says (not sure the console will work on standby you may need to use the master Supervisor and then switch console command)


Hope to help

Giuseppe


jpazahanick Tue, 01/13/2009 - 17:56
User Badges:

Did you find the issue? I have a set of 7609 12.2.33SRC2 doing the same thing...

georgina.cunha Thu, 10/15/2009 - 09:07
User Badges:

Hi,


I have the same issue on the router 7613 with IOS 12.2.33SRD2. Could you please tell me how you solved the problem?


Thanks in the advance,

Georgina

Giuseppe Larosa Thu, 10/15/2009 - 10:27
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Georgina,

have you got a NAM module on your chassis?


I've found a field notice that should apply to 12.2(33)SRC2


see


http://www.cisco.com/en/US/partner/ts/fn/631/fn63179.html


or


http://www.cisco.com/en/US/ts/fn/631/fn63179.html


it may be good if you can provide a sh module


Edit:

there are also some bugs that may apply like


CSCta36814 Bug Details


After final reload of ISSU SRB3 -> SRD1 the standby loads only to RPR mode and

reports MCL configuration mistmatch.



Hope to help

Giuseppe


georgina.cunha Fri, 10/16/2009 - 04:50
User Badges:

Hi,


Thanks for the inputs.


The problem seems to be related with the NAM module.


After removing the NAM card the redundancy started to work perfectly.


My Best Regards,

Georgina Cunha


robert.callahan Thu, 10/15/2009 - 10:45
User Badges:

We felt it was mis-matched configs between the Sups.We can issued the following command at the enable prompt: "redundancy config-sync ignore mismatched-commands". After issuing this command, we saved the configuration to memory and then reloaded the standby. Assuming that any mis-matched commands was the culprit, this process should revert the standby sup back to sso mode. It did for us. Other input told to us to take ssh out of config before first boot as the keys will always be a "mismatched config"

Actions

This Discussion