CSm loosing connections every 4 hours ?

[m.baker2]

Hi,

We upgraded our CSm to V4.2.2 and for one of the VIP, it looses connections to the 4 real servers every 4 hours ! Anybody has any idea what could cause this to happen ? If I use IOS SLB, I do not have this problem...

[Gilles Dufour]

the 4 hours is the default arp timeout.

So I would say the problem is related to arp.

Are you losing all real servers every 4 hours ?

Are you in bridge mode or routing mode ?

Are the servers using the CSM as default gateway ?

Sniff the arp traffic to see what is going on.

Regards,

Gilles.

[m.baker2]

Gilles,

The Servers's default gateway point to the MSFC. To get the MSFC to route traffic back towards the CSM when the servers reply, we NAT the connections on the CSM with a NAT pool. and have a route for teh NAT pool on the MSFC to send the traffic back to the CSM...

We tryed changing the arp timeout using the CSM set variable command but still same symptoms... We are using router mode for this one.

All connections to the 4 real servers disapears but the CSm does not report the reals as being out of service. I am assuming the problem is on the CSM but it could be the servers as well. It is just odd that the 4 servers would drop all connections at the same time every 4 hours...And by going back to IOS SLB and not doing any NAT, the problem goes away.

We're trying to get some packet trace to help out the troubleshooting...

Cheers

[m.baker2]

We did find that it is the CSM initiating the disconnects when the CSM arp entries time out. We were able to avoid the problem by increasing the arp timeout on the CSM....but we still do not understand why the ARP entry expiring would terminate all connections....that seems like a bug ?

[Gilles Dufour]

not a bug.

The goal of this behavior is to preserve memory and performance by getting rid of all connections that are useless [if no arp entries exist].

The arp should not timeout so.

Are the servers sending arp request ?

Do they get to the CSM ?

Is the arp timeout on the server higher than on the CSM ?

Gilles.

[m.baker2]

The servers ARP timeout on the servers is smaller then on the CSM. The servers default gateway point to the MSFC and then the CSM and servers have to route via the MSFC to get to/from the servers/CSM (via static routes on the MSFC and route commands on the CSM).

Doing a show module contentswitching 9 arp displays the arp entries on the CSM but not the age of the arp entries. The REAL servers shows as the arp entry of the route (next hop) pointing to the MSFC.

Now, it is possible this isn't a bug but the problem only started when we upgraded the CSm to 4.2.2 from 4.1. (we also upgraded the MSFC IOS to 12.2.18 at the same time)... Now, How would the arp entry timeout on the CSM if there is ongoing traffic to the real servers via the CSM ?

Thanks