Access to rservers behind ACE

Answered Question
Jul 19th, 2011

I have ACE 20 in 6500 configured in routed mode, similar to configuration described here: http://docwiki.cisco.com/wiki/Basic_Load_Balancing_Using_Routed_Mode_on_the_Cisco_Application_Control_Engine_Configuration_Example

There is a context that has the following VLANs assigned:

Vlan 100 - management interface: 10.0.100.1/24

Vlan 101 - client side interface: 10.0.101.1/24

Vlan 102 - server side interface: 10.0.102.1/24

I have VIP configured (10.0.101.10) and two rservers (10.0.102.11 and 10.0.102.12) in a single server farm.

Servers have 10.0.102.1 as their default gateway.

Load balancing works fine, client connection reaches the rservers via VIP.

My problem is that I cannot connect to real servers behind ACE (10.0.102.11 and 10.0.102.12) directly, without load balancing involved.

The ACE just blocks all the traffic not destined to VIP, although ifaces vlan101 and vlan102 have "permit ip any any" input ACLs.

What else do I need to configure to make this work?

It worked when interfaces 101 and 102 were in bvi, but I need to have multiple contexts with shared vlans.

I have this problem too.
0 votes
Correct Answer by ahmajabe about 2 years 9 months ago

Hi Alexandr,

Answers are inline:

correct me if I am wrong, ACE separates contexts' traffic, right?

--> Correct.

In other words, I won't be able to route this traffic (10.0.102.0/24) via vlan 101 (client-side vlan) interface in Admin context.

Every server has to have its default gateway set to the corresponding vlan102 (server-side vlan) interface of the corresponding context, right?

-->Right, Now I understood your design

The idea behind that design was to control services consumption of ACE resources (via resource classes for different contexts) without implementing vlan groups for every context (what is required in bridged mode).

--> Correct!

My recommendation will be using /32 static routes on the 6K instead of /24 route, for example if you have 10.0.102.2 is reachable through Context-A and 10.0.102.3 through Context-B.

6K configuration:

route 10.0.102.2 255.255.255.255 10.0.101.1

route 10.0.102.3 255.255.255.255 10.0.101.2

Where 10.0.101.1 is Context-A VLAN 101 IP address and 10.0.101.2 is the VLAN 101 IP on Context-B.

Hope that help,

Ahmad

Correct Answer by ahmajabe about 2 years 9 months ago

Hi Alexandr,

The servers can have only one gateway, then all your servers' traffic will be going over one context, so I am not sure how you are planning to have one context per service, please explain your plan to deploy that.

The best practice design, is to have management VLAN to access the servers, by adding another interface to the servers and allow the admins to access them through the 6K, since no need to pass through the ACE and consume its resources for management access.

Best regards,

Ahmad

Correct Answer by ahmajabe about 2 years 9 months ago

Hi Alexandr,

Do you mean how to allow the clients to access the real servers IP addresses through multiple contexts? If yes, please explain why you need this. I could not think of any situation where you need to allow the clients to access the real IP addresses over multiple contexts.

Best regards,

Ahmad

Correct Answer by ahmajabe about 2 years 9 months ago

Hi Alexandr,

The MSFC will allow these two VLAN to go over the port-channel trunk toward the ACE, but it will not redirect all the traffic on these VLANs to the ACE, the 6K will still perform normal L2 (ARP) and L3 (routing) lookup before sending the traffic.

With the current configuration the clients request will be send to the servers directly by the 6K and it will not go through the ACE for sure even you have added static route, since the server subnet is directly connected and has lower cost, but the server reply will be sent to the ACE and the ACE will drop it due to IP normalization feature.

If you need this communication to work through the ACE, do the following:

1- Remove the VLAN 102 interface from the 6K configuration.

2- Add route 10.0.102.0 255.255.255.0 10.0.101.1 on the 6K

3- Keep the 10.0.102.1 IP address as the servers gateway.

This way the communication should work smoothly, and the capture tool should show you both legs of the communication.

Let me know if that help, and mark this one as correct answer.

Best regards,

Ahmad

Correct Answer by ahmajabe about 2 years 9 months ago

Hi Alexandr,

What is the client default gateway and where is it configured?

Best regards,

Ahmad

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (6 ratings)
ahmajabe Tue, 07/19/2011 - 05:10

Hi Alexandr,

The ACE does not block any traffic as long as the access-list permit it and the ACE knows how to reach it. You just need to make sure the traffic is reaching the ACE's client VLAN distant to server's real addresses and the ACE will pass it over.

Most probably you are missing static route in the upstream device to forward the traffic to the ACE VLAN 101 IP address if it is distant to 10.0.102.0/24 subnet.

Hint: You can use the capture tool on the ACE to make sure the traffic is reaching the ACE or not. Let me know if you need any help using it.

Best regards,

Ahmad

AlexandrKry Tue, 07/19/2011 - 05:26

Hi Ahmad,

Since I can reach the servers via VIP (10.0.101.10), I assume that the traffic reaches ACE's client interface.

However, when I try to reach the server directly (10.0.102.11), the acl hitcount doesn't increase (sh access-list Clients_In detail, where Clients_In is the "permit ip any any" ACL on Vlan101).

I didn't quite understand you about the static route. Do I need to setup it on ACE?

The client is one hop behind ACE, directly connected to 6500 in a different Vlan.

I didn't know about the capture tool. Thanks for pointing out!

ahmajabe Tue, 07/19/2011 - 05:57

Hi Alexandr,

The VIP IP address is located in different subnet for that reason if your client needs to reach the server's real IP address they need to go through L3 device (routing should be applied).

Ex:

Client (192.168.1.2/24) ----- 6K (VLAN 10) 192.168.1.1/24 ---- (VLAN 101) 10.0.101.2 ---- ACE (VLAN 101) 10.0.101.1 ----- Server 10.0.102.2

The clients will send their traffic to 6K VLAN 10 IP in this example to reach 10.0.102.0/24 subnet, the 6K should has static ip route to forward the traffic distant to 10.0.102.0/24 to 10.0.101.1 as next hop to the ACE; this is what I mean by "Most probably you are missing static route in the upstream device to forward the traffic to the ACE VLAN 101 IP address if it is distant to 10.0.102.0/24 subnet.".

If you have this kind of setup, then you will need to add static route on the 6K to forward the traffic distant to 10.0.102.0/24 subnet to 10.0.101.1 IP address.

If all your clients are located within 10.0.101.0/24 subnet, and their default gateway is 10.0.101.1, then most probably you have miss-configuration on the ACE, so you need to go further step and capture the traffic on the ACE using these commands:

capture cap interface vlan 101 access-list everyones

capture cap start

Try to access the server's real IP address.

capture cap stop

If the traffic is reaching the ACE that will generate logging messages on your telnet screan showing the source and destination IP addresses of the packets.

If that did not help, please share your topology, configuration capture command you used and capture output you have got if you dont mind.

Best regards,

Ahmad

AlexandrKry Tue, 07/19/2011 - 05:54

The capture tool doesn't get any traffic. It's obvious, because packets don't hit the Clients_In ACL..

If I cange the rserver's (10.0.102.12) default gateway to Vlan102 ip address of 6500, I can access it.

But when the default gateway is set to Vlan102 interface of ACE (10.0.102.1), I cannot.

To me that's eliminate any possible routing issues outside the ACE.

Correct Answer
ahmajabe Tue, 07/19/2011 - 06:08

Hi Alexandr,

What is the client default gateway and where is it configured?

Best regards,

Ahmad

AlexandrKry Tue, 07/19/2011 - 06:26

Argh, I marked you reply as the correct answer accidentally.

The default gateway on the client is the SVI interface of 6k.

Here is the topology:

Client (10.0.160.2/12) ------- 6k (vlan 20) 10.0.160.1 ----- vlan (102) 10.0.102.1------- ACE (vlan 102?) 10.0.102.1? ---- server (vlan 102) 10.0.102.11

The 6k has both of these networks (vlans 20,102) as directly connected.

The server is directly connected to 6k via access port (vlan102), I can ping it from there.

Am I missing something in how 6k and ACE route traffic in vlans, assigned to ACE? I thought that MSFC routes all the traffic in those vlans to ACE.

Here is how I use the capture tool:

# capture cap1 all access-list everyone

# capture cap1 start

access-list everyone extended permit ip any any



Correct Answer
ahmajabe Tue, 07/19/2011 - 06:39

Hi Alexandr,

The MSFC will allow these two VLAN to go over the port-channel trunk toward the ACE, but it will not redirect all the traffic on these VLANs to the ACE, the 6K will still perform normal L2 (ARP) and L3 (routing) lookup before sending the traffic.

With the current configuration the clients request will be send to the servers directly by the 6K and it will not go through the ACE for sure even you have added static route, since the server subnet is directly connected and has lower cost, but the server reply will be sent to the ACE and the ACE will drop it due to IP normalization feature.

If you need this communication to work through the ACE, do the following:

1- Remove the VLAN 102 interface from the 6K configuration.

2- Add route 10.0.102.0 255.255.255.0 10.0.101.1 on the 6K

3- Keep the 10.0.102.1 IP address as the servers gateway.

This way the communication should work smoothly, and the capture tool should show you both legs of the communication.

Let me know if that help, and mark this one as correct answer.

Best regards,

Ahmad

AlexandrKry Tue, 07/19/2011 - 06:44

Now I see.

But in that case I won't be able to share the servers' vlan (102) between multiple contexts (the servers' traffic will go to the single context via this static route).

Correct Answer
ahmajabe Tue, 07/19/2011 - 06:50

Hi Alexandr,

Do you mean how to allow the clients to access the real servers IP addresses through multiple contexts? If yes, please explain why you need this. I could not think of any situation where you need to allow the clients to access the real IP addresses over multiple contexts.

Best regards,

Ahmad

AlexandrKry Tue, 07/19/2011 - 06:59

The clients are not exactly "clients", but system adminsitrators, who need to get direct access to the servers.

Each server has only one IP interface in the ACE Server vlan.

Here is the design explanation:

10.0.101.0/24 Vlan 101 - the "clients" vlan.

Clients will access the services via VIP addresses over multiple contexts from corporate network. These VIP addresses are all from Vlan101.

Each service will be isolated into separate context having its own VIP.

10.0.102.0/24 Vlan 102 - the servers' vlan.

All servers from all contexts will reside in the Vlan 102 for ease of management.

Administrators need to access servers directly.

If you have any suggestions on the design, I'd like to hear them.

Correct Answer
ahmajabe Tue, 07/19/2011 - 07:20

Hi Alexandr,

The servers can have only one gateway, then all your servers' traffic will be going over one context, so I am not sure how you are planning to have one context per service, please explain your plan to deploy that.

The best practice design, is to have management VLAN to access the servers, by adding another interface to the servers and allow the admins to access them through the 6K, since no need to pass through the ACE and consume its resources for management access.

Best regards,

Ahmad

AlexandrKry Tue, 07/19/2011 - 07:51

Hi Ahmad,

correct me if I am wrong, ACE separates contexts' traffic, right?

In other words, I won't be able to route this traffic (10.0.102.0/24) via vlan 101 (client-side vlan) interface in Admin context.

Every server has to have its default gateway set to the corresponding vlan102 (server-side vlan) interface of the corresponding context, right?

The idea behind that design was to control services consumption of ACE resources (via resource classes for different contexts) without implementing vlan groups for every context (what is required in bridged mode).

Thanks for your help.

Correct Answer
ahmajabe Tue, 07/19/2011 - 18:04

Hi Alexandr,

Answers are inline:

correct me if I am wrong, ACE separates contexts' traffic, right?

--> Correct.

In other words, I won't be able to route this traffic (10.0.102.0/24) via vlan 101 (client-side vlan) interface in Admin context.

Every server has to have its default gateway set to the corresponding vlan102 (server-side vlan) interface of the corresponding context, right?

-->Right, Now I understood your design

The idea behind that design was to control services consumption of ACE resources (via resource classes for different contexts) without implementing vlan groups for every context (what is required in bridged mode).

--> Correct!

My recommendation will be using /32 static routes on the 6K instead of /24 route, for example if you have 10.0.102.2 is reachable through Context-A and 10.0.102.3 through Context-B.

6K configuration:

route 10.0.102.2 255.255.255.255 10.0.101.1

route 10.0.102.3 255.255.255.255 10.0.101.2

Where 10.0.101.1 is Context-A VLAN 101 IP address and 10.0.101.2 is the VLAN 101 IP on Context-B.

Hope that help,

Ahmad

AlexandrKry Tue, 07/19/2011 - 22:10

Hi Ahmad,

Thanks for your comments. /32 routes never crossed my mind. That will do the trick, although it will require to update routing table every time we add a server.

Dedicated management interface is a better solution anyway. Thanks to you help I now have better understanding of how ACE works and what were my mistakes in the original design.

AlexandrKry Tue, 07/19/2011 - 22:54

There is one thing I missed out again.

/32 routes on 6k are still necessary if you want to use a single Vlan across multiple contexts whether or not you use dedicated interfaces on servers for management.

AlexandrKry Tue, 07/19/2011 - 23:23

I should clarify this again for someone who get stuck as I did

You can use the same server Vlan for multiple contexts without host routes. To do that you need to have different subnets for each one of them (even though I'm not sure it's a good design).

My solution is to split the server's /24 network into multiple /28 networks and add appropriate routes on 6k.

This way you won't need to add a static route every time you add a server to the context.

Actions

Login or Register to take actions

This Discussion

Posted July 19, 2011 at 4:40 AM
Stats:
Replies:16 Avg. Rating:
Views:1648 Votes:0
Shares:0
Tags: routing, ace20
+

Related Content

Discussions Leaderboard

Rank Username Points
1 1,551
2 369
3 333
4 228
5 212
Rank Username Points
5