advice on ssl termination setup

cmarva · ‎06-01-2010

I have a VIP that terminates ssl connections, and

the real servers are http on port 8080. There are three separate apps

involved, example is site/inspector, site/reporter, and site/admin.

I do not have logins for the separate apps, but all three are reachable

and apparently acted as they should, and all was going well.

Today I was informed that the reporter app does not work as it should.

Users can log into the app, and can generate reports. The problem is

that if they choose the export option (the app will allow one to export

the reports via csv file), the export fails. I did a client side sniff just

to see if the app was doing something on other than port 8080, and it

does not appear to. Obviously with the encryption you can't see the payload,

but once again I was just looking for connections on other than 8080,

as that would not have been subject to the url rewrite. I do plan on doing

a server side sniff (behind the VIP) tomorrow.

The symptom when the export fails is that when you hit the ok button,

the "save file" box opens, but is immediately followed by a stop box that

says something like "cannot find the file specified". I also had a user

do a direct connect to one of the reals and do the same thing, and everything

worked fine. The sniffer capture once again did not show anything other than

http on port 8080. But the "save file" box popped up, the user selected the

path and was able to save the report.

This is set up on ACE modules in a 6509 chassis. I can certainly provide

more detail, but was kind of hoping that someone has seen this symptom

and might be able to point me in a direction. Other than that, everything else

with this VIP/these apps seems to be working fine.

Thanks for any advice - chris

Sean Merrow · ‎06-02-2010

Hi Chris,

These types of situations can be a challenge to troubleshoot. However, there are ways to see the client side of the connection decrypted. Here are some of them:

One way is to capture the whole session from the client using Wireshark. Then you can place the private key that is used on the ACE onto your PC and let Wireshark use the key to decrypt the traffic.
Another way, if you can't get the private key, is to use a browser tool to view the unencrypted HTTP headers. HTTPWatch Pro can be used in IE or Firefox, or a freebie for Firefox is LiveHTTPHeaders.

The best way to go is to SPAN the ACE's tengig port on the chassis so you can get the client side and server side capture in a single capture file. Also, run one of the above browser tools, along with Wireshark, on the client PC. It is also a good idea to get the output of show tech-support from the context your using before and after your test so that you can see if any specific error counters increment during a failed connection.

With this data, you should be able to see if the client is launching a new connection while trying to export the data. Perhaps the button is sending the client to a different port, or maybe there is an HREF within the HTML that the browser is following. The ACE can only rewrite the Location header of a redirect, not an embedded HREF.

Hope this helps,

Sean

cmarva · ‎06-07-2010

sorry for not responding sooner. I can't quite

seem to get the certificate imported into wireshark at this point.

I do still need to run the capture to see the server side traffic (behind the

ace). I also appreciate the heads up about http watch.

This situation is not a showstopper at this point. I did verify with the users

that this is the only situation that they have seen a problem with. All other

apps and functions within the apps are working fine.

thanks again, I appreciate it. If I get this figured out I will respond with what is

happening, might be useful for others.

chris

Peter Koltl · ‎06-05-2010

Please upload the config and show conn output.

Do you use sticky?

cmarva · ‎06-07-2010

Peter, yes the service is sticky.

Everything else within the three apps appears to be working

fine, I did verify with the users. It's only when they try and

download the csv within the one app that they are having problems

with. I want to run a sniffer capture and see if I can nail this one

myself. If I do get to the point where I'm at a dead end, I will post the

configs. But I will learn more if I dig on this myself first.

I do appreciate the response, and if I get this sorted out, I will post

what I find as it might be useful for others.

Thanks again, chris

Sean Merrow · ‎06-07-2010

I want to run a sniffer capture and see if I can nail this one myself. If I do get to the point where I'm at a dead end, I will post the configs. But I will learn more if I dig on this myself first.

Quite commendable! ;- )

Sean

cmarva · ‎06-10-2010

I have taken capture with the span set up to watch the outside and inside (VIP vlan and server vlan). I also have a cleartext capture done while

connecting directly to one of the webservers in the pool. I am not seeing anything in the content that looks out of order. I do see some things related

to the tcp connection like sack being cleared, etc. but nothing really looks much different within the content. In both captures, I can see the GET

of the csv, and also in both captures I can see the 200 OK response. The only difference is that with a direct connection to the server, the csv does

download, and via the VIP I get the stop dialog that says "internet explorer cannot download xyz from sitename. internet explorer was not able to open

the site, it's either unavailable or cannot be found. Please try again later".

So I haven't made much headway with this, although since the proxied capture looks like the direct connect capture, it almost looks like something

with the ssl proxy service. But that's just a guess. Anyway, I have attached both captures as well as a cleaned config of the ace context. The version

is a2(2.2).

Other than the csv download, everything else seems to be working fine. Thanks for any advice - chris

the ace-in-out capture is the span of the ace module

the clear-2 capture is the direct connect

Sean Merrow · ‎06-10-2010

Hello,

From the capture, I don't see the ACE doing anything wrong that would cause this. I have included a screenshot of the capture where the request for the .csv file is requested. You'll notice that for each packet on the encrypted side, it is exactly 21 bytes larger than it is on the decrypted side. This is for the SSL data. If you start from the packet I have highlighted, which is the GET for the .csv file coming in encrypted to the VIP, you'll notice every packet that is sent by the client or the server is forwarded accordingly by the ACE. The only packet that is not 21 bytes larger on the encrypted side is packet 384. This is because the ACE also had to change the HTTP to HTTPS as configured, which added an extra byte making it 22 bytes larger on the encrypted side.

At this point, I think you're going to need to use HTTPWatch on your client browser both when going through the ACE and when going direct to the server to see exactly what is different from the client's perspective. From the ACE's perspective, things appear fine to me.

One thing you could try is removing the SSL termination from the load balancing and see if it works in clear text through the ACE.

Hope this helps in moving this forward.

Sean

cmarva · ‎06-10-2010

Sean,

thanks for the help.

I did grab httpwatch basic, which does capture the headers, but looks like I need the pro edition to actually see the header content. I'll see what I can

do regarding that.

As far as trying a VIP without the ssl termination, I'll see what I can do because the context this vip is in has multiple production vips (no worry there)

but this particular one is rather important to the organization. I think I might be able to just set up another vip to work on port 8080 just for testing. I'll

look into that.

But I do appreciate the help and it eases my mind a little that you believe the ace is working as expected. Thanks again - chris

cmarva · ‎06-10-2010

Sean,

thanks again for the help.

A co-worker suggested fiddler to watch the headers. I did install it and watched the headers. I did see in the session

that the result when selecting the csv and hit the submit button, the 200 ok did reach the browser, but the same

"can't find /download the file" happened.

So just for the heck of it, I tried firefox......voila! it worked.

Now I am truly befuddled, but I think I have "plausible denialbitility" now. I don't think that there is a problem

with the setup. Maybe domain policy or something within IE, but I think the ACE/network are exonerated.

just wanted to post this in case anyone was following this. I don't know what the IE deal is, but I think firefox working

proves the ace is not the problem. - chris

Sean Merrow · ‎06-10-2010

Great. Thanks for the follow-up and let us know if we can be of any further assitance.

Sean

cmarva · ‎06-11-2010

did a little more digging, and it looks like this truly is particular to IE.

I found two microsoft kb articles regarding this:

"http://support.microsoft.com/default.aspx?scid=KB;EN-US;q316431&"

and also

"
http://support.microsoft.com/kb/323308" (although this one deals with registry hacks on client machines).

So I can probably try header deletion as a test and see if that works.

I hope this might be helpful for others who run into this situation. - chris

Sean Merrow · ‎06-11-2010

Excellent work Chris!

While I'm sure you already know the location, I wanted to let everyone know where they can find out how to remove an HTTP header from a server response using the ACE.

I look forward to seeing whether or not this solves your issue.

Sean

cmarva · ‎06-14-2010

well, I wanted to try rewriting the cache-control header to private, but after reading rfc2616, and knowing that

a lot of the apps used in my environment keep us bound to ie6, I decided to just delete the pragma (http1.0)

and cache control (http1.1) headers altogether. If I'm understanding 2616 correctly, the user should not run into

cached content problems courtesy of the expires header. We'll see. I'm not overly worried at this point.

But yes, deleting the headers did the trick and now the csv report does download.

Thanks again for the help, I appreciate it - chris

Sean Merrow · ‎06-15-2010

Thanks for the follow up Chris.

Sean