I am facing a strange issue. I have a SSLv3 service running on two servers behind a pair of(active-standby) CSS 11501, wich do advance-balance ssl. Few days ago I noticed that the CSS stopped the LB and start sending he request to only one server( both alive). I think that is hapening due to the SSL-layer 4 fallback. How do I verify it , and restore stickyness with session ID?
Thanks in advance
What you can do is get simultaneous captures, span the content rules vlan, and the servers vlan, but you need to get the captures on the switch(s) that the CSS is directly connected to. As soon as you get the captures you will see the connection incoming into the CSS and you will see the SSL ID, on the back end side connection you will see the CSS doing the load balancing based on the SSL ID.
Also please add the application SSL command, but the best way to confirm what the CSS is doing is taking the captures.
Hope this help.
The issue could be more serious than what you are seeing. When you say you rely on ssl-id for LB, I recall a incident that happened some 6 months ago. Internet Explorer (IE) 5.0/5.5 SSL SID changes approximately every two minutes. if you use ssl id to do load balance you may be in trouble, one reason why we started using alternate methods like arrowpoint cookies to do load balancing ssl sessions. Arrowpoint cookie works fine. See below.
See the Microsoft version of the IE SSL-id issue. MS recommends a registry hack which I never tried as it is hypothetical to change everywhere on all the IE client browsers.
Note: I have no information how a IE 6.0 or its later versions behaves. You may want to talk to TAC.
We are not using IE in this case. There is a custom client application, which does a request and get the answer and then the server closes the connection. The load balance worked well until few days ago, and I still see the SSL ID exchanged between client and server for each new connection, but the SSL does not load balance.
I did some captures on the server side and I can see the SSL ID exchanged with the client, which is different for every new connection as expected. Using the "show sticky-table ssl-sticky sid" I can see the flow details associated. So I think it did not l4-fallback, but I do not understand why it does not load balance. I also use the application SSL command.
How are you seeing that the CSS is not doing the correct load balancing?
Are you seeing one of the servers being overloaded?
The content is configured for round-robin.
Using the flows command I can see that it sends all the requests to the same server. If I suspend the service pointing to that server then it starts sending to the other and remains on that server even when the other is reactiveted.
That's why I thought it was in SSL-l4-fallback. But I don't have any entries in L4 sticky table and I can see the flows in ssl-sticky table.
The trace on the server side shows that a new Session ID is created for a new connection as expected.
No, I don't have it configured. But I don't understand the need for it, in this case. As I wrote before, every new connection ( request) is expected to be a new entry in the sticky-table. The CSS should do a roud-robin load balance for those requests.
issueing the commnd ssl-l4-fallback disable, restores the load balance. This means, I think, that the l4-fallback was activated. The questions remaiins, why, if we are using ssl v3? why the l4 sticky table has no entries. Here is a ssldump example:
New TCP connection #2: 172.16.20.1(46311) <-> 172.26.80.1(3034)
2 1 0.0011 (0.0011) C>SV3.0(95) Handshake
46 8a 31 6f 87 94 35 a8 30 16 42 5e e9 6e 31 c0
d6 17 ac eb 92 9c 53 db 85 92 df 30 0e 67 90 90
Unknown value 0x39
Unknown value 0x38
Unknown value 0x35
Unknown value 0x33
Unknown value 0x32
Unknown value 0x2f
2 2 0.0025 (0.0014) S>CV3.0(74) Handshake
46 8a 19 d6 a8 f5 88 8e d4 d5 16 06 df 1e 8e e5
0e 7c 1b f2 f8 bb b9 c8 7d c6 e2 f8 fe e6 7f b0
db 23 61 fa 35 5a ef 0e da 3d be b0 c2 85 39 9f
4d b1 75 5d 5b ef c8 46 bc 4d db ab 31 23 d6 d4
2 3 0.0025 (0.0000) S>CV3.0(1147) Handshake
2 4 0.0025 (0.0000) S>CV3.0(4) Handshake
2 5 0.2346 (0.2320) C>SV3.0(260) Handshake
1b a5 47 46 7b 9a 8b 84 88 d4 54 ba 23 c7 7a 31
db 8a 74 c2 02 3b 8b 50 75 c3 c8 c0 38 4c 18 e8
0a e8 c8 47 39 ff 9b 3c 6a cc d3 21 78 69 e6 50
88 55 e8 b3 d3 b1 2a 04 b3 ba 66 e4 c8 49 f4 8e
a0 bd 74 60 e9 f2 0c a1 25 47 03 4e 6c ed 96 52
de 2a 9a 60 29 c5 f6 21 c8 3e 58 ef af 3f 12 b2
ee 34 c0 70 12 d0 64 30 28 65 ed fb ff 65 f2 de
d0 bf cd e6 26 79 6f 3c 61 5f df da bf ac 4a cf
4c 0e 0c 66 44 e1 b2 a3 34 f2 27 75 f0 e4 e7 a4
48 06 76 93 73 0d 09 75 35 ea a1 91 d9 c8 ad 58
b9 1f 45 bf c6 09 61 cb 2d 75 4a ba ed 45 15 44
0f fc 9e 5b 90 e7 b2 86 15 1c 43 a5 52 0d c7 1e
d8 81 42 db 77 35 99 4d 0d 5b 20 e6 dd c5 a1 7d
64 9a 13 d2 99 b7 1d 94 a7 fe ce b6 67 7d df b9
25 fb 27 d2 6e 90 49 54 b9 c1 10 32 eb 42 df 43
b6 1c 94 4e ee 0b ca 29 27 8a 3d b8 fe 59 00 f4
2 6 0.2346 (0.0000) C>SV3.0(1) ChangeCipherSpec
2 7 0.2346 (0.0000) C>SV3.0(64) Handshake
2 8 0.2708 (0.0362) S>CV3.0(1) ChangeCipherSpec
2 9 0.2708 (0.0000) S>CV3.0(64) Handshake
2 10 0.3074 (0.0365) C>SV3.0(40) application_data
2 11 7.9036 (7.5961) S>CV3.0(24) application_data
2 12 7.9036 (0.0000) S>CV3.0(104) application_data
2 13 7.9036 (0.0000) S>CV3.0(24) Alert
2 7.9036 (0.0000) S>C TCP FIN
2 14 7.9464 (0.0427) C>SV3.0(24) Alert
2 7.9524 (0.0059) C>S TCP FIN
As you can see session ID is sent correctly.
It is solved now.
I guess a few days ago the client or server, starts retransmiting frames activating the SSL-l4-fallback feature.
And , apparently the only way to restore the behavior, is to:
issue ssl-l4-fallback disable
suspend/activate the associated content rule
which causes the sticky table to be purged for the flows associated with the content rule.
Still don't figured out why the l4 sticky table was empty.
you were very helpfull