ACE: HowTo troubleshoot SSL connections?

Unanswered Question
Jan 26th, 2010

Hey all,

i need a hint on how to troubleshoot ssl connectivity through the ACE.

Issue:

The client connectivity via the VIP randomly produces strange error messages with the application inside the browsers. A quick sniff of the VIP on the according ACE context sometimes shows following error in the traced packets.

Secure Socket Layer
    SSLv3 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter)
        Content Type: Alert (21)
        Version: SSL 3.0 (0x0300)
        Length: 2
        Alert Message

Symptoms using IE:

User is browsing through the portal application everything works out. Out of nowhere the user randomly receives a "Page cannot be displayed" while clicking a link in the portal. If the user clicks the link again once or sometimes twice the behavior disappears for a while. This error can show up for the whole page or only a single frame within the page.


Symptoms using Firefox:

User is browsing through the portal application everything works out. Out of nowhere the users randomly receives a "secure connection failed" while clicking a link in the portal. The detailed browser message is "SSL_ERROR_DECRYPTION_FAILURE: Bulk data decryption algorithm failed in  selected cipher suite". If the user clicks the link  again once or sometimes twice the behavior disappears for a while.

Question:

I'm not sure if there is a relation between the captured ssl handshake error and the symptoms shown in the browser. To make sure it is an ACE error or not I need a solid approach to troubleshoot. So if anyone has good advice, please let me know.

Thanks for reading

Roble

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Ward Pyper Mon, 02/01/2010 - 12:16

Seen something similar on one of our ACEs. Occasionally, it would fail to complete an ssl handshake.The "fix" was to reboot the ACE, it'd been up 1 year and we're now looking to update the ACE to the latest software release as we're running an early version.

Sounds like you've already identified what the problem is but as to why it's happening, then I'd suggest you raise a TAC case and see what CISCO say. Good luck!

Gilles Dufour Tue, 02/02/2010 - 00:35

Roble,

proper troubleshooting requires capturing a sniffer trace on client AND server side simultanously.

Then qith wireshark and the private key, you can decode the sniffer trace and see what is happening.

Also make sure to run the latest software version. I know we have added recent software fix in the ssl area.

Gilles.

Roble Mumin Tue, 02/02/2010 - 01:21

Hey Gilles, hey Ward,

first of all thanks for the reply.

We recently updated the ACE in all locations to A2(2.3) which has been referenced as "latest version" in the download center. In my opinion the reload issue mentioned by Ward should be ruled out by the fact the we recently updated the blades.

Uptime => DLB001 kernel uptime is 17 days 19 hours 26 minute(s) 22 second(s) (it was about 200+ Days before)

Regarding the SSL issues mentioned by Gilles i am not sure if those have already been fixed in the A2(2.3) release. So this could be another option to look at.

The confusing thing in the whole issue is that in certain contexts running the same application(J2EE based Portal) the error shows up in other contexts e.g "pre production" the error does not occur at all. So the natural techie conclusion => application related.

The symptom preventing me from being sure about the application being the root cause is the observed ssl handshake error. For me to be sure i indeed would need a sniffer trace but in our environment that is a real pita. The application client is running in a loadbalanced Citrix farm with 300+ presentation servers. So in this case i need to sniff the Citrix server and the ACE at the same time. IIRC the ACE only has 5MB capture buffer another hurdle, so i have to be quick with the start stop commands.

I was hoping to get a hint on a hidden "show crypto connection details" option which could have shown me some status information otherwise only being available via a sniff.

As i am seeing different behavior in different context with the same application i also thought about bad certificate. Any experience with the influence on e.g. incorrect generated certificate/key combinations? We are using pkcs12 formatted certificates.

Thanks for reading

Roble

Gilles Dufour Tue, 02/02/2010 - 02:04

Roble,

take a sniff outside the ACE ...

The show stats crypto server command will just show that we sent an SSL alert/error corresponding to what firefox also reports.

But it won´t tell you why we sent that error.

The only way is to see the response from the server unencrypted and the corresponding encrypted packet from ACE so that we can see what happened.

A2(2.3) is your best option currently.

But unforunately it is still possible that there are problems.

Please, try to get the sniffer trace and capture a show tech just before and just after collecting the trace. (so we can verify the counters which are incrementing during the capture).

Thanks,

Gilles.

Actions

This Discussion