sh log Syslog logging: enabled (12 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled) Console logging: level debugging, 124 messages logged, xml disabled, filtering disabled Monitor logging: level debugging, 0 messages logged, xml disabled, filtering disabled Buffer logging: level debugging, 124 messages logged, xml disabled, filtering disabled Logging Exception size (8192 bytes) Count and timestamp logging messages: disabled No active filter modules. Trap logging: level debugging, 131 message lines logged Logging to 10.30.1.40(global) (udp port 514, audit disabled, link up), 131 message lines logged, xml disabled, filtering disabled Logging to 10.30.2.50(global) (udp port 514, audit disabled, link up), 131 message lines logged, xml disabled, filtering disabled Log Buffer (10000 bytes): 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000079: Jun 23 13:56:13.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000080: Jun 23 13:56:13.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 10 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000081: Jun 23 13:56:13.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000082: Jun 23 13:56:13.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 10 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000083: Jun 23 13:56:15.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000084: Jun 23 13:56:15.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 9 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000085: Jun 23 13:56:15.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000086: Jun 23 13:56:15.702 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 9 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000087: Jun 23 13:56:17.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000088: Jun 23 13:56:17.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 8 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000089: Jun 23 13:56:17.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000090: Jun 23 13:56:17.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 8 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000091: Jun 23 13:56:19.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000092: Jun 23 13:56:19.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 7 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000093: Jun 23 13:56:19.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000094: Jun 23 13:56:19.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 7 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000095: Jun 23 13:56:21.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000096: Jun 23 13:56:21.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 6 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000097: Jun 23 13:56:21.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000098: Jun 23 13:56:21.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 6 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000099: Jun 23 13:56:23.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000100: Jun 23 13:56:23.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 5 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000101: Jun 23 13:56:23.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000102: Jun 23 13:56:23.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 5 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000103: Jun 23 13:56:25.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000104: Jun 23 13:56:25.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 4 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000105: Jun 23 13:56:25.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000106: Jun 23 13:56:25.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 4 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000107: Jun 23 13:56:27.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000108: Jun 23 13:56:27.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 3 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000109: Jun 23 13:56:27.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000110: Jun 23 13:56:27.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 3 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000111: Jun 23 13:56:29.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000112: Jun 23 13:56:29.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 2 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000113: Jun 23 13:56:29.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000114: Jun 23 13:56:29.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 2 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000115: Jun 23 13:56:31.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000116: Jun 23 13:56:31.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12. Will retry 1 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000117: Jun 23 13:56:31.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000118: Jun 23 13:56:31.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8. Will retry 1 time(s). 1st retry scheduled in approx. 2000 ms. Last IPC error 7:no such port 000119: Jun 23 13:56:33.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 12, port 'Slot 12: DFS.control.IL', error code 7:no such port. 000120: Jun 23 13:56:33.718 UTC: %IPFAST-2-FAILOPENIPCPORT: Could not open the IPC ports. Slot 12. Number of retries 20. Last IPC error 7:no such port 000121: Jun 23 13:56:33.718 UTC: %IPFAST-4-FAILATTEMPTOPENIPCPORT: Attempt to open the IPC ports failed. Slot 8, port 'Slot 8: DFS.control.IL', error code 7:no such port. 000122: Jun 23 13:56:33.718 UTC: %IPFAST-2-FAILOPENIPCPORT: Could not open the IPC ports. Slot 8. Number of retries 20. Last IPC error 7:no such port 000123: Jun 23 13:56:46.309 UTC: %DSC_ENV-1-WARNING: Slot 24, AC Shelf warning, Flags = 0 ; Check shelf status 000124: Jun 23 13:56:46.309 UTC: %DSC_ENV-1-WARNING: Slot 25, AC Shelf warning, Flags = 0 ; Check shelf status 000125: Jun 23 13:58:38.275 UTC: %REDUNDANCY-3-CONFIG_SYNC: Active and Standby bulk configuration out of sync 000126: Jun 23 14:02:13.453 UTC: %FIB-2-FIBDISABLE: Fatal error, slot 12: keepalive failure 000127: Jun 23 14:02:13.569 UTC: %FIB-2-FIBDISABLE: Fatal error, slot 8: keepalive failure 000128: Jun 23 14:02:13.769 UTC: %FIB_HM_MVL-4-FB_BUSYOUT: Busy out feature board 12, initiated by the Health Monitor 000129: Jun 23 14:02:13.797 UTC: %FIB_HM_MVL-4-FB_BUSYOUT: Busy out feature board 8, initiated by the Health Monitor 000130: Jun 23 14:02:14.141 UTC: %DIAL8-2-MSG: Jun 23 12:02:16.769: %FIB-2-FIBDISABLE: Fatal error, slot 8: Linecard timed out waiting for messages from RP 000131: Jun 23 14:02:14.145 UTC: %DIAL12-2-MSG: Jun 23 12:02:16.773: %FIB-2-FIBDISABLE: Fatal error, slot 12: Linecard timed out waiting for messages from RP 000132: Jun 23 14:02:43.768 UTC: %FIB_HM_MVL-4-FB_RESET: Power cycling feature board 12, as requested by the Health Monitor 000133: Jun 23 14:02:43.780 UTC: %DSIPPF-5-COMMS_STATUS: DSIP/IPC communications to slot 12 are DOWN. 000134: Jun 23 14:02:58.780 UTC: %FIB_HM_MVL-4-FB_BUSYOUT: Busy out feature board 12, cancelled by the Health Monitor due to FB reload 000135: Jun 23 14:04:03.125 UTC: %IPC-5-INVALID: IPC Port info Dest Port 0x0 Session 0x0 Source 0x25E03EF4 -Traceback= 0x201AC6A8 0x219022B0 0x21902A98 0x21906E88 0x20426FDC 0x20420FB0 0x202AB37C 0x202AB360 000136: Jun 23 14:04:25.586 UTC: %DSIPPF-5-COMMS_STATUS: DSIP/IPC communications to slot 12 are UP.