CSS service IP replying rather than VIP IP

[eisenberg]

I have a very strange issue going on. I have a CSS setup with a two-armed config. The services live on the private interface of the CSS and the VIPs live on the public interface of the CSS. I am balancing the VIP 10.100.31.242 amongst three services (see config below). I am having trouble with services ws07_13 and ws08_13 which I have since suspended. Service ws06_13 is working fine. The behavior I see with ws07_13 and ws08_13 is very odd. When the are active and get sent a request by rule-242 they reply with thier own IP to the upstream firewall (cisco ASA). In turn the connection on the firewall get dropped. According to the cisco ASA logs the service IP (10.100.32.x) is trying to reply to the actual Internet IP. I have provided a screen shot of these logs via the ASDM below. You will see where services ws07_13 (10.100.32.141) and ws08_13 (10.100.32.240) are trying to reply to Internet IP 199.193.10.250 which gets dropped by the ASA (deny TCP no connection). I am not seeing this behavior with service ws06_13 (10.100.32.242)

The arp tables on the CSS , switch and upstream firewall (cisco ASA) look OK. I do not manage the servers that these services live on. I just want to rule out networking (I manage the CSS, switches and firewall) before I have the server owner compare the working service (ws06_13) to the non-working services (ws07_13 & ws08_13).

!

!************************** CIRCUIT **************************
circuit VLAN1
  redundancy

  ip address 10.100.31.241 255.255.255.0

circuit VLAN201
  redundancy

  ip address 10.100.32.254 255.255.255.0

!************************** SERVICE **************************
!
service ws06_13
  protocol tcp
  port 80
  keepalive frequency 30
  keepalive port 80
  keepalive retryperiod 10
  keepalive uri "/cssindex.htm"
  ip address 10.100.32.242
  keepalive type http non-persistent
  active
!
service ws07_13
  protocol tcp
  port 80
  keepalive frequency 30
  keepalive port 80
  keepalive retryperiod 10
  keepalive uri "/cssindex.htm"
  ip address 10.100.32.241
  keepalive type http non-persistent
  active
!
service ws08_13
  protocol tcp
  port 80
  keepalive frequency 30
  keepalive port 80
  keepalive retryperiod 10
  keepalive uri "/cssindex.htm"
  ip address 10.100.32.240
  keepalive type http non-persistent
  active
!

!*************************** OWNER ***************************
owner acme

!
content rule-242
    protocol tcp
    port 80
    vip address 10.100.31.242
    advanced-balance sticky-srcip
    add service ws06_13
    add service ws07_13
    add service ws07_13
    active

[Pablo]

Hi,

Does the connection work if you suspend ws06_13 and bring up only ws07_13?

Do the routes look the same on the 3 servers? Are ws07_13 or ws08_13 dual-homed?

Can you please paste the result of

CSS# show run acl

CSS# show run group

Regards.

__ __

Pablo

[eisenberg]

Does the connection work if you suspend ws06_13 and bring up only ws07_13?

When ws06_13 is removed and ws07_13 or ws08_13 are in place and active the connection does not work

Do the routes look the same on the 3 servers? Are ws07_13 or ws08_13 dual-homed?

I am still waiting for the server owner to send me the output from the

ROUTE PRINT command for all three servers.

I put the following groups in place since my last post to combat the issue. With the following in place the connection works with all three services in place. This configuration is currently active, as mentioned above I am still waiting for the output of the ROUTE PRINT command.

group sg-165
  vip address 10.100.31.165
  add destination service ws07_3
  add destination service ws08
  add destination service ws06_5
  active

group sg-242
  vip address 10.100.31.242
  add destination service ws08_13
  add destination service ws07_13
  add destination service ws06_13
  active

Here is the output you requested.

CSS11501# sh run acl

CSS11501#

CSS11501#

CSS11501# sh run group

!*************************** GROUP ***************************
group sg-165
  vip address 10.100.31.165
  add destination service ws07_3
  add destination service ws08
  add destination service ws06_5
  active

group sg-242
  vip address 10.100.31.242
  add destination service ws08_13
  add destination service ws07_13
  add destination service ws06_13
  active

CSS11501#

If the routes on all three servers look OK I will simply leave the groups in place. I expect to see an issue with the routes and will post the output of the ROUTE PRINT command for all three servers

, thanks

thanks

[Pablo]

Hi,

If the group addition took care of the problem then I would suspect a routing issue, you're running in routed mode so theoretically the group should not be required.

If not a route difference you can have the server guy take a capture to identify the traffic response path.

Cheers!

__ __

Pablo

[eisenberg]

Turned out that the two servers in question had bad routes pointing all traffic to the incorrect default gateway address.
It is nice to know that I can compensate for a bad DG on the upstream CSS with the group option.

thanks for your assistance with this matter