06-29-2007 07:37 AM
Hi everyone,
I am facing a strange issue. I have a SSLv3 service running on two servers behind a pair of(active-standby) CSS 11501, wich do advance-balance ssl. Few days ago I noticed that the CSS stopped the LB and start sending he request to only one server( both alive). I think that is hapening due to the SSL-layer 4 fallback. How do I verify it , and restore stickyness with session ID?
Thanks in advance
David
06-30-2007 05:42 PM
Hello David,
What you can do is get simultaneous captures, span the content rules vlan, and the servers vlan, but you need to get the captures on the switch(s) that the CSS is directly connected to. As soon as you get the captures you will see the connection incoming into the CSS and you will see the SSL ID, on the back end side connection you will see the CSS doing the load balancing based on the SSL ID.
Also please add the application SSL command, but the best way to confirm what the CSS is doing is taking the captures.
Hope this help.
- Rodrigo.
07-01-2007 08:08 PM
The issue could be more serious than what you are seeing. When you say you rely on ssl-id for LB, I recall a incident that happened some 6 months ago. Internet Explorer (IE) 5.0/5.5 SSL SID changes approximately every two minutes. if you use ssl id to do load balance you may be in trouble, one reason why we started using alternate methods like arrowpoint cookies to do load balancing ssl sessions. Arrowpoint cookie works fine. See below.
See the Microsoft version of the IE SSL-id issue. MS recommends a registry hack which I never tried as it is hypothetical to change everywhere on all the IE client browsers.
http://support.microsoft.com/default.aspx?scid=KB;en-us;q265369
Note: I have no information how a IE 6.0 or its later versions behaves. You may want to talk to TAC.
07-02-2007 02:49 AM
Hi Skumar,
We are not using IE in this case. There is a custom client application, which does a request and get the answer and then the server closes the connection. The load balance worked well until few days ago, and I still see the SSL ID exchanged between client and server for each new connection, but the SSL does not load balance.
David
07-02-2007 02:41 AM
Hi Rodrigo,
I did some captures on the server side and I can see the SSL ID exchanged with the client, which is different for every new connection as expected. Using the "show sticky-table ssl-sticky sid" I can see the flow details associated. So I think it did not l4-fallback, but I do not understand why it does not load balance. I also use the application SSL command.
Thank you,
David
07-02-2007 05:21 PM
Hello David,
How are you seeing that the CSS is not doing the correct load balancing?
Are you seeing one of the servers being overloaded?
-Rodrigo
07-03-2007 02:25 AM
Hi Rodrigo,
The content is configured for round-robin.
Using the flows command I can see that it sends all the requests to the same server. If I suspend the service pointing to that server then it starts sending to the other and remains on that server even when the other is reactiveted.
That's why I thought it was in SSL-l4-fallback. But I don't have any entries in L4 sticky table and I can see the flows in ssl-sticky table.
The trace on the server side shows that a new Session ID is created for a new connection as expected.
David
07-03-2007 02:44 AM
Do you have Sticky inactive timeout?
"sticky-inact-timeout 240" worked for us.
07-03-2007 03:05 AM
hi s.
No, I don't have it configured. But I don't understand the need for it, in this case. As I wrote before, every new connection ( request) is expected to be a new entry in the sticky-table. The CSS should do a roud-robin load balance for those requests.
David
07-03-2007 04:29 AM
Further observations:
issueing the commnd ssl-l4-fallback disable, restores the load balance. This means, I think, that the l4-fallback was activated. The questions remaiins, why, if we are using ssl v3? why the l4 sticky table has no entries. Here is a ssldump example:
New TCP connection #2: 172.16.20.1(46311) <-> 172.26.80.1(3034)
2 1 0.0011 (0.0011) C>SV3.0(95) Handshake
ClientHello
Version 3.0
random[32]=
46 8a 31 6f 87 94 35 a8 30 16 42 5e e9 6e 31 c0
d6 17 ac eb 92 9c 53 db 85 92 df 30 0e 67 90 90
cipher suites
Unknown value 0x39
Unknown value 0x38
Unknown value 0x35
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
SSL_RSA_WITH_3DES_EDE_CBC_SHA
Unknown value 0x33
Unknown value 0x32
Unknown value 0x2f
SSL_DHE_DSS_WITH_RC4_128_SHA
SSL_RSA_WITH_RC4_128_SHA
SSL_RSA_WITH_RC4_128_MD5
SSL_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA
SSL_RSA_EXPORT1024_WITH_DES_CBC_SHA
SSL_RSA_EXPORT1024_WITH_RC2_CBC_56_MD5
SSL_DHE_RSA_WITH_DES_CBC_SHA
SSL_DHE_DSS_WITH_DES_CBC_SHA
SSL_RSA_WITH_DES_CBC_SHA
SSL_DHE_DSS_WITH_RC2_56_CBC_SHA
SSL_RSA_EXPORT1024_WITH_RC4_56_SHA
SSL_RSA_EXPORT1024_WITH_RC4_56_MD5
SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
SSL_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5
SSL_RSA_EXPORT_WITH_RC4_40_MD5
compression methods
NULL
2 2 0.0025 (0.0014) S>CV3.0(74) Handshake
ServerHello
Version 3.0
random[32]=
46 8a 19 d6 a8 f5 88 8e d4 d5 16 06 df 1e 8e e5
0e 7c 1b f2 f8 bb b9 c8 7d c6 e2 f8 fe e6 7f b0
session_id[32]=
db 23 61 fa 35 5a ef 0e da 3d be b0 c2 85 39 9f
4d b1 75 5d 5b ef c8 46 bc 4d db ab 31 23 d6 d4
cipherSuite SSL_RSA_WITH_3DES_EDE_CBC_SHA
compressionMethod NULL
2 3 0.0025 (0.0000) S>CV3.0(1147) Handshake
Certificate
2 4 0.0025 (0.0000) S>CV3.0(4) Handshake
ServerHelloDone
2 5 0.2346 (0.2320) C>SV3.0(260) Handshake
ClientKeyExchange
EncryptedPreMasterSecret[256]=
1b a5 47 46 7b 9a 8b 84 88 d4 54 ba 23 c7 7a 31
db 8a 74 c2 02 3b 8b 50 75 c3 c8 c0 38 4c 18 e8
0a e8 c8 47 39 ff 9b 3c 6a cc d3 21 78 69 e6 50
88 55 e8 b3 d3 b1 2a 04 b3 ba 66 e4 c8 49 f4 8e
a0 bd 74 60 e9 f2 0c a1 25 47 03 4e 6c ed 96 52
de 2a 9a 60 29 c5 f6 21 c8 3e 58 ef af 3f 12 b2
ee 34 c0 70 12 d0 64 30 28 65 ed fb ff 65 f2 de
d0 bf cd e6 26 79 6f 3c 61 5f df da bf ac 4a cf
4c 0e 0c 66 44 e1 b2 a3 34 f2 27 75 f0 e4 e7 a4
48 06 76 93 73 0d 09 75 35 ea a1 91 d9 c8 ad 58
b9 1f 45 bf c6 09 61 cb 2d 75 4a ba ed 45 15 44
0f fc 9e 5b 90 e7 b2 86 15 1c 43 a5 52 0d c7 1e
d8 81 42 db 77 35 99 4d 0d 5b 20 e6 dd c5 a1 7d
64 9a 13 d2 99 b7 1d 94 a7 fe ce b6 67 7d df b9
25 fb 27 d2 6e 90 49 54 b9 c1 10 32 eb 42 df 43
b6 1c 94 4e ee 0b ca 29 27 8a 3d b8 fe 59 00 f4
2 6 0.2346 (0.0000) C>SV3.0(1) ChangeCipherSpec
2 7 0.2346 (0.0000) C>SV3.0(64) Handshake
2 8 0.2708 (0.0362) S>CV3.0(1) ChangeCipherSpec
2 9 0.2708 (0.0000) S>CV3.0(64) Handshake
2 10 0.3074 (0.0365) C>SV3.0(40) application_data
2 11 7.9036 (7.5961) S>CV3.0(24) application_data
2 12 7.9036 (0.0000) S>CV3.0(104) application_data
2 13 7.9036 (0.0000) S>CV3.0(24) Alert
2 7.9036 (0.0000) S>C TCP FIN
2 14 7.9464 (0.0427) C>SV3.0(24) Alert
2 7.9524 (0.0059) C>S TCP FIN
As you can see session ID is sent correctly.
David
07-03-2007 08:36 AM
Hi guys,
It is solved now.
I guess a few days ago the client or server, starts retransmiting frames activating the SSL-l4-fallback feature.
And , apparently the only way to restore the behavior, is to:
issue ssl-l4-fallback disable
or
suspend/activate the associated content rule
which causes the sticky table to be purged for the flows associated with the content rule.
Still don't figured out why the l4 sticky table was empty.
Thank y'all
you were very helpfull
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: