ACE module: can't use VIP of a virtual server from within the same context

Unanswered Question
Oct 6th, 2008

Hello,

(short version...)

In a multi-context ACE module configuration, is there a reason why we can't access a virtual server using it's virtual IP address (VIP) from machines located in the same context? It works fine from other contexts, but I can't get it to work from within the same context.

(longer version)

Here are some general details on our config.

We have an ACE2 module running the latest version of the code, A2(1.2). We have an Admin context and 2 user contexts. In contextA, we have a virtual server that load-balances two real web servers that are located in the same VLAN, let's call it VLAN A1.

In contextB, we have one virtual server that is load-balanced to two real SMTP servers in VLAN B1. On the same VLAN B1, there's one real IMAPS server; we will soon add others and will define a virtual IMAPS server that will load-balance the real IMAPS servers. In another VLAN of the same contextB (let's call it VLAN B2), there's a real Web server. Again, we will add others and define a new virtual Web server to load-balance between these real web servers.

From any network outside of the ACE module, we can use the virtual IP of the SMTP server in contextB. We can also use it from the real servers in contextA (VLAN A1). But from any of the machines in VLANs of contextB (VLAN B1, VLAN B2), we can't use the VIP of the SMTP server in contextB.

I enabled the "VIP Advertise" option (it seemed that it could help) but it didn't.

When I try: "telnet VIP 25" (replacing VIP by the actual virtual IP of the SMTP server in contextB), it works from the real servers in VLAN A1 (ContextA) but from the real IMAPS server in VLAN B1 or from the real Web server in VLAN B2, I get the following error:

$ telnet 10.32.33.2 25

Trying 10.32.33.2...

telnet: connect to address 10.32.33.2: Connection refused

telnet: Unable to connect to remote host: Connection refused

Any help would be greatly appreciated, because as we move new services behind the ACE module, we will need to access virtual servers using their VIP from within the same context.

Thank you very much!

Marc.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.5 (2 ratings)
Loading.
Syed Iftekhar Ahmed Mon, 10/06/2008 - 12:12

Marc You will need to SRC NAT traffic from the Rservers before they hit the VIP to enable Rserver to VIP traffic.

You will also need to have a multi-match policy listening on VIP address on the vlan where traffic from Rservers is expected.

Some thing in line of the following config

Assumptions:

1.Rservers reside in vlan 20 (10.10.10.0/24 address space.

2.Vip is listening on 192.168.1.Z

access-list NAT-ACL line 10 extended permit ip 10.10.10.0 255.255.255.0 host

192.168.1.Z

class-map match-all RSERVERS2VIP

2 match access-list NAT-ACL

class-map match-all SYED-VIP

2 match virtual-address 192.168.1.Z tcp eq www

policy-map type loadbalance first-match PMAP

class class-default

serverfarm SYED-SF

policy-map multi-match LB-VIPS

class RSERVERS2VIP

nat dynamic 80 vlan 20

class SYED-VIP

loadbalance vip inservice

loadbalance policy PMAP

loadbalance vip icmp-reply

interface vlan 20

description SERVER SIDE VLAN

ip address 10.10.10.1 255.255.255.0

access-group input everyone

nat-pool 80 192.168.1.x 192.168.1.x netmask 255.255.255.0 pat <-- natpool using Client side IPs

service-policy input LB-VIPS

no shutdown

HTH

Syed Iftekhar Ahmed

MMazuhelli_2 Tue, 10/07/2008 - 12:46

Hello Syed,

Thank you very much for the quick response! I looked at your sample configuration and it took me some time to understand it, probably because I used ANM to configure the ACE module and contexts, so I am not completely familiar with the CLI. I had to read parts of the documentation and I think I finally understand half of your explanation.

I understand the part where you say: "You will also need to have a multi-match policy listening on VIP address on the vlan where traffic from Rservers is expected". I know that I don't have that right now, and that I will need to add this.

But I still don't understand why I have to SRC NAT traffic from the RServers to the VIPs.

The Catalyst 6500 with the ACE module is not at the periphery of our network, it's in front of our server room. With our network team, we decided not to NAT internal traffic from our server room. Real servers use 10.x.y.z addressing, and the VServers use external (routable) addresses. Being a university, we are lucky enough to have a class B network, let's call it A.B.0.0/16. Our SMTP server has VIP address A.B.6.34 (external routable address) and it's load-balanced to real servers with IP addresses 10.32.33.9 and 10.32.33.10. Other VLANs routed by the ACE module in the same or other contexts have similar arrangements (10.x.y.z addressing for real servers, A.B.c.d for VIPs).

(in my first post, I used an example with "10.32.33.2" as the VIP of our SMTP server; that's a test I tried of a VIP on the same subnet as the rservers, but it didn't work. The VIP that is used normally is A.B.6.34)

Inter-server traffic will be of two kinds: some will have to go directly to a specific server (because we have only one; it's the case with certain database servers). For the cases where we have more than one server for the same service and we configure load balancing with a serverfarm and a VIP, inter-server traffic should use the VIP to reach the real servers.

With this arrangement, it would be much simpler if we could just leave the addresses as they are (10.x.y.z). "Much simpler" here relates to DNS entries and local firewall rules (iptables on Linux boxes) that are easier to maintain if all addresses remain intact. If an outgoing connection from a real server ever has to reach the Internet, it's policy-NATted on a separate ASA 5540 that we have at the periphery of our network.

I don't know if I included enough information for you to understand the reasons why I would prefer not to use NAT at all on the ACE module... but as a conclusion I will ask the following question: if I add a multi-match policy listening on the VIP addresses on the vlan where traffic from the Rservers is expected, can I get away without SRC NATTing the outgoing RServer traffic and still reach the VIPs from RServers within the same context?

Thank you very much,

Marc.

Syed Iftekhar Ahmed Tue, 10/07/2008 - 13:52

You only need to translate "REAL to VIP" traffic, whichyou can easily filter using ACLs on ACE.

I will try to explain why we need to translate for Real-->VIP traffic.

Lets assume that VIP for app1 is 200.200.200.200

and the servers for app1 are

REAL1(IP: 10.10.10.1 & MAC: aa-aa-aa-aa-aa-aa) & REAL2(IP: 10.10.10.2 & MAC: bb-bb-bb-bb-bb-bb).

Now lets assume that REAL1 initiates traffic for the VIP, the packet will look like

Srcip:10.10.10.1 Dstip:200.200.200.200

The ACE will rcv the packet as its listening for VIP 200.200.200.200. ACE will then select the REAL server using the LB algo.Lets assume ACE selects REAL2 for this request.

ACE will change the request packet as follows

Srcip:10.10.10.1 Dstip:10.10.10.2

Now the REAL2 sees that this request came from REAL1 (10.10.10.1) and since 10.10.10.1 belongs to its local network, instead of sending the reply back to ACE it will send the response directly to REAL1 (using its ARP table).

Since REAL1 sent a request to 200.200.200.200, it is expecting a response from the same IP (VIP).

When it rcvs a response from REAL2(10.10.10.2) it will discard it.As it is not expecting any response from REAL2 (there is no connection entry for such reply).

In order to make sure that servers get responses back from the same ip where they sent the initial request its absoulutely needed to translate the src ip.

HTH

Syed Iftekhar Ahmed

MMazuhelli_2 Wed, 10/08/2008 - 13:00

Hello again Syed,

Thank you very much for your explanations, they were very clear. And it made me realize that the problem is not present in the whole context, but just on the same VLAN.

In fact, I can now use a VServer from real servers located in another VLAN of the same context, which I couldn't do before. In ANM, I only had to include the other VLAN in the list of VLANS associated with the virtual server - except that I was disappointed with the fact that in ANM, we can't add a new VLAN to an existing VServer; if I do, I get the message: "Virtual server Vlan set cannot be changed", so I had to delete the VServer and recreate it with the additional VLANS, which means that there's an interruption of service. Bummer! ;-(

I will now experiment with NATting outgoing connections from real servers to VServers that are load-balanced to real servers in the same VLAN as the requesting rserver.

Thanks again!

Marc.

MMazuhelli_2 Tue, 10/21/2008 - 12:10

Hello Syed,

I've been busy on other things but I finally had the time to test source-natting traffic from the real servers to a VIP in the same VLAN. I can't say that the solution is simple and elegant, but it works great!

Thanks again for taking the time to respond to my questions, it was extremely useful!

By the way, how can I mark this thread with the red check-mark that means "includes response(s) that solved Author's issue"? I checked everywhere and couldn't find anything else than the score (1 to 5) that we can assign to individual messages in the thread.

Thanks again,

Marc.

Syed Iftekhar Ahmed Tue, 10/21/2008 - 13:30

I am glad it worked for you.

Though I do get these "red check marks" regularly.I really don't know how to assign these to a post :)

Thanks

Syed Iftekhar Ahmed

Actions

This Discussion