ACE 30 - rserver connection failing to external host

[Al Nelson]

We have a pair of ACE 30's in 6500 Service Chassis connected to Nexus 7000.

The ACE's are running in router mode. One VIP (VLAN 100 - Client side) is set up for SSL termination, and the traffic is forwarded to a pair of Rservers (VLAN 200 - Server side) in a server farm on port 80.

These servers are Apache Proxy servers which proxy the traffic to a host on port 8830.

This works fine most of the time.

But a small percentage of time, we see the Apache Proxy server send three SYN's to the host on port 8830, but never get a response back. Other connections to this host are running during this time on different source ports from the Apache Proxy server.

Our TCOM team has taken network traces stating that they see these three SYN's leaving the Apache Proxy server, but never getting to the host.

Is there a way that we can find out if these SYN's are getting dropped by the ACE?

If so, is there something that we should be doing different in our configuration (i.e. turn off Normalization on the Server side VLAN, setting TCP-SYN cookie, some other parameter map?)

These servers do handle a large number of requests and when the connections fail (looking at network traces) there appears to be a source transport number missing.

[Kanwaljeet Singh]

Hi Al,

So the traffic is loadbalanced to proxy servers on port and then these servers forward the traffic to a host on port 8830. So when these proxy servers forward the traffic, does that traffic need to go through ACE which i assume it need to since you are suspecting that ACE is dropping it. Does this traffic need to be loadbalanced again or it is just supposed to go to a host through ACE, come back through ACE to servers and then servers will reply back to the original client.

Kindly elaborate more on how is the traffic flow?

Do you see that those SYN's came on ACE? You can take a pcap on ACE and see if you see those SYN's on ACE or better would be to take the pcap on switch where ACE and servers are connected and see if you see that packets were forwarded to ACE. Another capture on front side to see if those SYN's engress ACE and get there. If not, then probably ACE is dropping it.

Is it happening for a single client all the time or is it random?

Normalization should not be cause of drop for SYN packets.

Regards,

Kanwal

[Al Nelson]

First, I must thank you for taking time to look at my discussion.

In answer to your questions....

"does the traffic need to go through ACE..." Yes. The ACE is set up in two arm or router mode, with the interface for VLAN 200 being the Gateway for the Apache Proxy servers.

"Does this traffic need to be loadbalanced again or it is just supposed to go to a host through ACE, come back through ACE to servers and then servers will reply to the original client." No, the traffic that is sourced from the Apache Proxy servers (Rservers) does not need to be load balanced. It is using the ACE as the Gateway/Router.

"Do you see that those SYN's came on the ACE?" So far, I have not been able to grab a pcap file on the ACE. The traffic volume is too large (buffer size on the ACE is only 5 MB and we are capturing 100 MB captures every 10 minutes or so). We only less than 10 of this failures a day out - usually out of normal working hours.

"Is it happening for a single client all the time or is it random?" It is random and has various clients.

Is it possible to take PCAP from the Supervisor of the Service Chassis? It has all Layer 2 VLAN's.

Also....read this from configuration guide:

Note: Because IP normalization is always enabled on the ACE, if you have a Layer 2 connected server that sends traffic to a source MAC address that is not the one advertised by the ARP reply to received traffic, the ACE drops this traffic.

This should not be the case as there are other TCP sessions established at this same time on different source ports from the Rserver.

[Kanwaljeet Singh]

Hi Al,

You are most welcome!

You can disable normalization but i doubt it will help since servers default GW is ACE and servers will always send the traffic to it (so destination MAC would be ACE MAC and ace will always answer ARP for it) so no question of ACE dropping it.

You are right about buffer size and that is why i suggested to take pcaps at backend as well as front end. May be if you have a FW in the path you can see in logs if SYN is coming from ACE which was of course originally forwarded by proxy.

I am not sure about pcaps on sup but you can do a 10G capture for ACE which should capture every packet thrown at ACE.

Regards,

Kanwal