CSS restore sticky ssl

Unanswered Question
Jun 29th, 2007
User Badges:

Hi everyone,

I am facing a strange issue. I have a SSLv3 service running on two servers behind a pair of(active-standby) CSS 11501, wich do advance-balance ssl. Few days ago I noticed that the CSS stopped the LB and start sending he request to only one server( both alive). I think that is hapening due to the SSL-layer 4 fallback. How do I verify it , and restore stickyness with session ID?


Thanks in advance

David

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 1.5 (4 ratings)
Loading.
RODRGUTI Sat, 06/30/2007 - 17:42
User Badges:

Hello David,


What you can do is get simultaneous captures, span the content rules vlan, and the servers vlan, but you need to get the captures on the switch(s) that the CSS is directly connected to. As soon as you get the captures you will see the connection incoming into the CSS and you will see the SSL ID, on the back end side connection you will see the CSS doing the load balancing based on the SSL ID.


Also please add the application SSL command, but the best way to confirm what the CSS is doing is taking the captures.


Hope this help.


- Rodrigo.

skumar1969 Sun, 07/01/2007 - 20:08
User Badges:
  • Bronze, 100 points or more

The issue could be more serious than what you are seeing. When you say you rely on ssl-id for LB, I recall a incident that happened some 6 months ago. Internet Explorer (IE) 5.0/5.5 SSL SID changes approximately every two minutes. if you use ssl id to do load balance you may be in trouble, one reason why we started using alternate methods like arrowpoint cookies to do load balancing ssl sessions. Arrowpoint cookie works fine. See below.


http://www.cisco.com/en/US/products/hw/contnetw/ps789/products_tech_note09186a00801c65b4.shtml#client


See the Microsoft version of the IE SSL-id issue. MS recommends a registry hack which I never tried as it is hypothetical to change everywhere on all the IE client browsers.


http://support.microsoft.com/default.aspx?scid=KB;en-us;q265369



Note: I have no information how a IE 6.0 or its later versions behaves. You may want to talk to TAC.



dalmada Mon, 07/02/2007 - 02:49
User Badges:

Hi Skumar,

We are not using IE in this case. There is a custom client application, which does a request and get the answer and then the server closes the connection. The load balance worked well until few days ago, and I still see the SSL ID exchanged between client and server for each new connection, but the SSL does not load balance.


David

dalmada Mon, 07/02/2007 - 02:41
User Badges:

Hi Rodrigo,

I did some captures on the server side and I can see the SSL ID exchanged with the client, which is different for every new connection as expected. Using the "show sticky-table ssl-sticky sid" I can see the flow details associated. So I think it did not l4-fallback, but I do not understand why it does not load balance. I also use the application SSL command.


Thank you,

David


RODRGUTI Mon, 07/02/2007 - 17:21
User Badges:

Hello David,


How are you seeing that the CSS is not doing the correct load balancing?


Are you seeing one of the servers being overloaded?


-Rodrigo

dalmada Tue, 07/03/2007 - 02:25
User Badges:

Hi Rodrigo,


The content is configured for round-robin.

Using the flows command I can see that it sends all the requests to the same server. If I suspend the service pointing to that server then it starts sending to the other and remains on that server even when the other is reactiveted.

That's why I thought it was in SSL-l4-fallback. But I don't have any entries in L4 sticky table and I can see the flows in ssl-sticky table.


The trace on the server side shows that a new Session ID is created for a new connection as expected.


David

Pavel Bykov Tue, 07/03/2007 - 02:44
User Badges:
  • Silver, 250 points or more

Do you have Sticky inactive timeout?

"sticky-inact-timeout 240" worked for us.

dalmada Tue, 07/03/2007 - 03:05
User Badges:

hi s.


No, I don't have it configured. But I don't understand the need for it, in this case. As I wrote before, every new connection ( request) is expected to be a new entry in the sticky-table. The CSS should do a roud-robin load balance for those requests.


David

dalmada Tue, 07/03/2007 - 04:29
User Badges:

Further observations:


issueing the commnd ssl-l4-fallback disable, restores the load balance. This means, I think, that the l4-fallback was activated. The questions remaiins, why, if we are using ssl v3? why the l4 sticky table has no entries. Here is a ssldump example:


New TCP connection #2: 172.16.20.1(46311) <-> 172.26.80.1(3034)

2 1 0.0011 (0.0011) C>SV3.0(95) Handshake

ClientHello

Version 3.0

random[32]=

46 8a 31 6f 87 94 35 a8 30 16 42 5e e9 6e 31 c0

d6 17 ac eb 92 9c 53 db 85 92 df 30 0e 67 90 90

cipher suites

Unknown value 0x39

Unknown value 0x38

Unknown value 0x35

SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA

SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA

SSL_RSA_WITH_3DES_EDE_CBC_SHA

Unknown value 0x33

Unknown value 0x32

Unknown value 0x2f

SSL_DHE_DSS_WITH_RC4_128_SHA

SSL_RSA_WITH_RC4_128_SHA

SSL_RSA_WITH_RC4_128_MD5

SSL_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA

SSL_RSA_EXPORT1024_WITH_DES_CBC_SHA

SSL_RSA_EXPORT1024_WITH_RC2_CBC_56_MD5

SSL_DHE_RSA_WITH_DES_CBC_SHA

SSL_DHE_DSS_WITH_DES_CBC_SHA

SSL_RSA_WITH_DES_CBC_SHA

SSL_DHE_DSS_WITH_RC2_56_CBC_SHA

SSL_RSA_EXPORT1024_WITH_RC4_56_SHA

SSL_RSA_EXPORT1024_WITH_RC4_56_MD5

SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA

SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA

SSL_RSA_EXPORT_WITH_DES40_CBC_SHA

SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5

SSL_RSA_EXPORT_WITH_RC4_40_MD5

compression methods

NULL

2 2 0.0025 (0.0014) S>CV3.0(74) Handshake

ServerHello

Version 3.0

random[32]=

46 8a 19 d6 a8 f5 88 8e d4 d5 16 06 df 1e 8e e5

0e 7c 1b f2 f8 bb b9 c8 7d c6 e2 f8 fe e6 7f b0

session_id[32]=

db 23 61 fa 35 5a ef 0e da 3d be b0 c2 85 39 9f

4d b1 75 5d 5b ef c8 46 bc 4d db ab 31 23 d6 d4

cipherSuite SSL_RSA_WITH_3DES_EDE_CBC_SHA

compressionMethod NULL

2 3 0.0025 (0.0000) S>CV3.0(1147) Handshake

Certificate

2 4 0.0025 (0.0000) S>CV3.0(4) Handshake

ServerHelloDone

2 5 0.2346 (0.2320) C>SV3.0(260) Handshake

ClientKeyExchange

EncryptedPreMasterSecret[256]=

1b a5 47 46 7b 9a 8b 84 88 d4 54 ba 23 c7 7a 31

db 8a 74 c2 02 3b 8b 50 75 c3 c8 c0 38 4c 18 e8

0a e8 c8 47 39 ff 9b 3c 6a cc d3 21 78 69 e6 50

88 55 e8 b3 d3 b1 2a 04 b3 ba 66 e4 c8 49 f4 8e

a0 bd 74 60 e9 f2 0c a1 25 47 03 4e 6c ed 96 52

de 2a 9a 60 29 c5 f6 21 c8 3e 58 ef af 3f 12 b2

ee 34 c0 70 12 d0 64 30 28 65 ed fb ff 65 f2 de

d0 bf cd e6 26 79 6f 3c 61 5f df da bf ac 4a cf

4c 0e 0c 66 44 e1 b2 a3 34 f2 27 75 f0 e4 e7 a4

48 06 76 93 73 0d 09 75 35 ea a1 91 d9 c8 ad 58

b9 1f 45 bf c6 09 61 cb 2d 75 4a ba ed 45 15 44

0f fc 9e 5b 90 e7 b2 86 15 1c 43 a5 52 0d c7 1e

d8 81 42 db 77 35 99 4d 0d 5b 20 e6 dd c5 a1 7d

64 9a 13 d2 99 b7 1d 94 a7 fe ce b6 67 7d df b9

25 fb 27 d2 6e 90 49 54 b9 c1 10 32 eb 42 df 43

b6 1c 94 4e ee 0b ca 29 27 8a 3d b8 fe 59 00 f4

2 6 0.2346 (0.0000) C>SV3.0(1) ChangeCipherSpec

2 7 0.2346 (0.0000) C>SV3.0(64) Handshake

2 8 0.2708 (0.0362) S>CV3.0(1) ChangeCipherSpec

2 9 0.2708 (0.0000) S>CV3.0(64) Handshake

2 10 0.3074 (0.0365) C>SV3.0(40) application_data

2 11 7.9036 (7.5961) S>CV3.0(24) application_data

2 12 7.9036 (0.0000) S>CV3.0(104) application_data

2 13 7.9036 (0.0000) S>CV3.0(24) Alert

2 7.9036 (0.0000) S>C TCP FIN

2 14 7.9464 (0.0427) C>SV3.0(24) Alert

2 7.9524 (0.0059) C>S TCP FIN


As you can see session ID is sent correctly.


David

dalmada Tue, 07/03/2007 - 08:36
User Badges:

Hi guys,

It is solved now.

I guess a few days ago the client or server, starts retransmiting frames activating the SSL-l4-fallback feature.

And , apparently the only way to restore the behavior, is to:

issue ssl-l4-fallback disable


or

suspend/activate the associated content rule

which causes the sticky table to be purged for the flows associated with the content rule.


Still don't figured out why the l4 sticky table was empty.


Thank y'all

you were very helpfull

Actions

This Discussion