ACE OCM counters increasing

nedian123 · ‎02-07-2010

Hi,

OCM(outbound connection Mgr) Counters are increasing & we have found that ACE is reseting client connection. I have found following bug. Currently we are running A2(1.1a). Anyone there hit by the same bug. Your experiences on this are appreciated.

CSCsr96168

Connection Free List Corruption

Symptom: New TCP connections are sometimes reset. Conditions: ACE module running A2(1.1a) Workaround: Downgrade to A2(1.1) Further Problem Description: This problem is suspected when `show np 2 me-stats -socm` shows the following counter incrementing: OCM Statistics: (Current) -------------- Drop [out of connections]: 14455 16

# show np 2 me-stats -socm
OCM Statistics: (Current)
--------------
Connection create received:                30526367            34
LB dest decision received:                  6724048            15
Drop [LB dest decision fail]:                    51             0
NAT Pool Alloc [addr/port]:                   25586             0
NAT Pool Free [addr/port]:                    25584             0
OCM Packet count (Hi & Lo):                37250415            49
(Context 4 Statistics)
Drop [mac lookup fail]:                           1             0
Connection inserted:                         853174             1
Packet message transmitted:                  853149             1

# show np 2 me-stats -socm
OCM Statistics: (Current)
--------------
Connection create received:                30526681            33
LB dest decision received:                  6724153             8
Drop [LB dest decision fail]:                    51             0
NAT Pool Alloc [addr/port]:                   25586             0
NAT Pool Free [addr/port]:                    25584             0
OCM Packet count (Hi & Lo):                37250834            41
(Context 4 Statistics)
Drop [mac lookup fail]:                           1             0
Connection inserted:                         853195             6
Packet message transmitted:                  853170             6

# show np 2 me-stats -socm
OCM Statistics: (Current)
--------------
Connection create received:                30528663            24
LB dest decision received:                  6724773            11
Drop [LB dest decision fail]:                    51             0
NAT Pool Alloc [addr/port]:                   25586             0
NAT Pool Free [addr/port]:                    25584             0
OCM Packet count (Hi & Lo):                37253436            35
(Context 4 Statistics)
Drop [mac lookup fail]:                           1             0
Connection inserted:                         853328             2
Packet message transmitted:                  853303             2

Regards,

Akhtar

Sean Merrow · ‎02-08-2010

Hello Akhtar,

You've probably noticed that the "Drop [out of connections]" counter referenced in the bug is not shown in the output of your commands. This means that this counter is currently at zero. It will only be shown if it has a non-zero value. Therefore, this implies that this counter is not going up.

Also, from the output you supplied, nothing looks alarming to me that would indicate the cause of the resets. Here is how I would recommend you proceed:

If possible, upgrade to a newer software release. The release you are on is quite dated now and there have been hundreds of software fixes since that release. Further, the A2(1.x) software train will no longer have any maintenance updates. Upgrading to A2(2.3) may be the quickest solution for you if you are hitting a bug on the ACE.
If you cannot upgrade, or you upgrade and the issue persists, then you should gather the following information:

Get a showtech from the context in which the VIP resides
Start a capture of the ACE tengig port(see not below)
Run some test connections to replicate the RST
Stop the capture
Get a second showtech

Now you will be able to look at the connections in Wireshark, and compare your two showtechs to see what, if any, error (or Drop) counters increment. If necessary, please open a SR with Cisco TAC for expert analysis.

NOTE: In order to do the Tengig capture, you would need to configure SPAN on the Supervisor. The source interface would be Te/1. So for example, if your ACE is in slot 3, then the source interface of your monitor session would be Te3/1.

Regards,

Sean