ASA with IPS has control channel communications failure

Unanswered Question
Aug 17th, 2009
User Badges:


ASA5520 with AIP-SSM-20 IPS module


IPS software: 7.0(1)E3

ASA software: 8.0(3)


We have two of these ASA's set up in a failover pair.


The problem seems to be at random times that the primary ASA/IPS member will failover to secondary with:


%ASA-3-323001: Module in slot 1 experienced a control channel communication failure.


The 'sh module' shows that the 'failed' ASA thinks that its IPS is not responsive:


Mod SSM Application Name Status SSM Application Version

--- ------------------------------ ---------------- --------------------------

1 IPS Not Applicable 7.0(1)E3


Mod Status Data Plane Status Compatibility

--- ------------------ --------------------- -------------

0 Up Sys Not Applicable

1 Unresponsive Not Applicable



The way to recover this is to reload the failed ASA. Interestingly, when a reload is done on the ASA and after it is back online, if you session into the IPS module the uptime is still several days - and it seems to function normally.


We've logged a tac case for this, and it is ongoing. The tac has replaced the IPS module twice but the problem remains - it certainly seems like the issue is with the ASA and not the IPS module.


We are currently running a 'debug cplane 255' on the console of the ASA and we are awaiting another failure - which could happen anytime up to 2 weeks from the last failure. Has anyone seen anything like this before and possibly could offer any advice?


Thanks


  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 3.8 (3 ratings)
Loading.
robertson.michael Fri, 08/21/2009 - 10:11
User Badges:
  • Silver, 250 points or more

Hi Cameron,


Unfortunately I don't recall the exact circumstances, but I remember seeing a similar issue. The problem ended up being the backplane connector on the ASA itself. As in your case, the SSM was replaced a couple of times but it turned out that the entire ASA needed to be replaced due to the bad backplane.


Hope that helps.


-Mike

vmoopeung Fri, 08/21/2009 - 10:19
User Badges:
  • Bronze, 100 points or more

1)Console up to the ASA and start capturing all messages and logs etc

2)First try issuing a "hw module 1 reset" which will power cycle on the card

3)If still the same issue, physically reset the module.

4)If the above steps do not work, you need to recover the module.

Step 3 Configure the recovery settings for ASA-SSM:

asa (enable)# hw-module module 1 recover configure

Note If you make an error in the recovery configuration, use the hw-module module 1 recover stop command to stop the system reimaging and then you can correct the configuration. The link below provides additional details.

http://www.cisco.com/en/US/docs/security/ips/5.0/configuration/guide/cli/clissm.html


cstockwe Sun, 08/30/2009 - 14:29
User Badges:

Had to wait a while to get another failure but it happened again on the weekend. I'll take this back to the tac for advice (debug cplane 255 output below).


Thanks for your help guys.


=======


Aug 28 2009 08:33:27: %ASA-3-216001: internal error in es_PostEvent: invalid descriptor


Aug 28 2009 08:33:31: %ASA-3-323001: Module in slot 1 experienced a control channel communication failure.




cp_transport_client_disconnect: closing socket 3


cp_transport_connection_handler: client at socket 3 disconnectedAug 28 2009 08:33:31: %ASA-4-411002: Line protocol on Interface Outside, changed state to down


Aug 28 2009 08:33:31: %ASA-4-411002: Line protocol on Interface Inside, changed state to down


Aug 28 2009 08:33:31: %ASA-4-411001: Line protocol on Interface Outside, changed state to up




Switching to Standby


Aug 28 2009 08:33:31: %ASA-4-411001: Line protocol on Interface Inside, changed state to up


Aug 28 2009 08:33:31: %ASA-1-104002: (Primary) Switching to STNDBY - Other unit wants me Standby. Secondary unit switch reason: Service card in other unit has failed.




Aug 28 2009 08:33:31: %ASA-1-105003: (Primary) Monitoring on interface Outside waiting


Aug 28 2009 08:33:31: %ASA-1-105003: (Primary) Monitoring on interface Inside waiting




Switching to Failed state.


Aug 28 2009 08:33:31: %ASA-1-104003: (Primary) Switching to FAILED.




cp_connect: Connecting to card 1, socket 3, port 7000


cp_transport_connection_handler: listening on socket 3


cp_socket_read_looped: Error during socket read


cp_transport_connection_handler: client at socket 3 disconnected


cp_transport_client_connect: Spawned thread c748a048 for new connection


cp_transport_client_connect: Created new connection with id 0




cp_update_connection: Error updating connection_id 0


cp_connect: Connecting to card 1, socket 3, port 7000


cp_transport_connection_handler: listening on socket 3


cp_transport_client_connect: Spawned thread c748bfe4 for new connection


cp_transport_client_connect: Created new connection with id 0




cp_transport_client_disconnect: closing socket 3


cp_transport_connection_handler: client at socket 3 disconnected


cp_connect: Connecting to card 1, socket 3, port 7000


cp_transport_connection_handler: listening on socket 3


cp_transport_client_connect: Spawned thread c7488288 for new connection


cp_transport_client_connect: Created new connection with id 0


Aug 28 2009 08:33:49: %ASA-3-323001: Module in slot 1 experienced a control channel communication failure.



======


5creedus Wed, 09/30/2009 - 14:28
User Badges:

I've had this happen to ASA models 5510 and 5520 with either ssm-10 or ssm-20 ips modules. I was told by Cisco the solution is to upgrade to version 7.0(1)E3 which I have done and still get that problem. I was then told it may be an issue of the ips module being oversubscribed and to adjust the acl that diverts traffic to the ips module. I've done that and still I have issues with it failing. I think next time I'll demand they send another firewall w/ ips module and let someone else figure out why it happens.

cstockwe Wed, 09/30/2009 - 14:56
User Badges:

Just as an update - we've had our ASA replaced and have not had a problem since (we still have the same IPS modules installed in the new chassis).


Actions

This Discussion