The ACE module is configured to direct traffic inbound on port 443 to a farm of internal servers on port 8443. The ACE is setup as a proxy for end-to-end SSL communication between the client and the internal server. The SSL key and certificate on the ACE were both generated external to the system (i.e., the key was not locally generated, and no CSR from the ACE was used).
With this configuration, most SSL web services on the internal server are functional from outside the ACE, but a couple of key functions are broken. Particularly, a Java application that downloads a number of files to the client via the Java Web Start function will hang ("Download stalled") during the file download, finally reporting an "unexpected end of file" or "connection reset" error in the Java console.
Viewing the packet data with Wireshark, there appear to be RST signals that are being sent from the server prematurely, about the same time that the download hangs.
I have removed every extraneous setting from the ACE configuration, with no affect on the problem. I have also attempted to modify a number of settings on the VLAN interfaces, such as adjusting fragment options and setting 'ip df' to 'clear'. None of these changes has made a difference.
The only way the Java application will function through the ACE is to de-configure the SSL proxy settings, letting the SSL data pass through as-is. This, however, breaks other needed functions for layer-7 URL-based load balancing.
Pertinent configuration is below:
hostname ACE_MFG
access-list ANY line 10 extended permit ip any any
rserver host HTTPS1
description HTTPS Server 1
ip address 172.30.3.6
inservice
ssl-proxy service SSL_PROXY_SERVER
key ACE_RSA_KEY_4.PEM
cert ACE_CERT_4.PEM
ssl-proxy service SSL_PROXY_CLIENT
serverfarm host HTTPS
description HTTPS Server Farm
failaction purge
retcode 200 500 check count
rserver HTTPS1
inservice
class-map match-any L4_HTTPS_SLB_VIP_CLASS
4 match virtual-address 172.30.255.2 tcp eq https
policy-map type loadbalance first-match L7_HTTPS_SLB_POLICY
class class-default
serverfarm HTTPS
ssl-proxy client SSL_PROXY_CLIENT
policy-map multi-match L4_HTTPS_SLB_POLICY
class L4_HTTPS_SLB_VIP_CLASS
loadbalance vip inservice
loadbalance policy L7_HTTPS_SLB_POLICY
loadbalance vip icmp-reply active
loadbalance vip advertise active
nat dynamic 1 vlan 310
ssl-proxy server SSL_PROXY_SERVER
interface vlan 110
description Client-side Interface
ip address 172.30.255.254 255.255.255.0
access-group input ANY
service-policy input L4_HTTPS_SLB_POLICY
no shutdown
interface vlan 310
description Server-side Interface
ip address 172.30.0.200 255.255.248.0
nat-pool 1 172.30.0.199 172.30.0.199 netmask 255.255.255.255 pat
no shutdown
ip route 0.0.0.0 0.0.0.0 172.30.255.1
