Re: Request for logical LAN design assistance

Josh Mahar · ‎09-10-2010

Howdy Cisco Community,

I am looking for some input to help me properly plan and elegantly design my logical network re-configuration. I'm specifcally hoping to hear from people that have done this before, not just theory based on documentation.

To start, here is some background and the environment setup. The infrastructure exists to host internet-facing web/app/db servers for various external clients. The primary focus here is to ensure client environments are isolated from one another. I inhereted the environment several years ago, so this restructuring has been a long time coming.

The equipment is as follows:

(2) Cisco 2950 (external switching)
(2) Cisco ASA 5520 (active/passive failover)
(4) Cisco 3750G-48TS in stacked configuration

External ethernet internet comes in through the two 2950's, then through the ASA's and finally to the core 3750 stack. Some ~100 servers are dual-linked and trunked to diverse switches in the stack.

There is an assortment of VLAN's, but fundamentally each host is trunked to only two:

a front-side where all web/app/db traffic operates, this includes external to internal (via ASA) for things like HTTP as well as server-to-server communication such as web to db
a back-end management for backups and monitoring, all hosts are connected to this with no filtering

Some hosts may have a 3rd VLAN for iSCSI traffic, but this is not done over a physically seperate interface. Aside from external to internal traffic through the ASA's there is no other filtering. There is no VLAN filtering of any kind (yet)... All hosts point to the ASA for default route. Overall, host traffic is rather light (5-15% utilization) with the exception of backups.

My goals of the reconfiguration are:

Establish several new VLANs for servers based on client/role: web/app, DB or other. I need to logically seperate the web and DB
Each new server VLAN's will have filtering between them (ie. Client A web server can reach Client A DB server, but no Client A server can connect to Client B server). Some VLANs may have "light" internal filtering.
Reconfigure the management VLAN with internal filtering (vlan filter). Since all hosts are connected to the management VLAN, filtering will be extensive.
The ability to monitor internet traffic utilization by client. Right now this is done via each logical VLAN internface on the ASA. If web to db traffic starts to traverse through the ASA as well, this can pose a problem because that isn't internet bound and will skew the metrics.

My primary questions/concerns are:

Which device should handle core routing given the plan for moderately extensive filtering, the ASA or the core stack?
How can I anticipate the load impact of filtering on both the ASA and switching? I'm specifically worried about this on the switch stack.
If the switch is the default route, should I trunk the ASA into all the web server VLAN's, or create a single "external access" VLAN through which all inbound/outbound traffic flows?

Thanks for reading this far, I tried to keep this as short as possible. I am leaning towards making the 3750's the default route but am very concerned about switch load and filtering in the core.

Please ask for any additional information that may be helpful and thanks in advance for your responses.

Josh

Giuseppe Larosa · ‎09-10-2010

Hello Josh,

I would suggest you to consider VRF lite to segregate traffic of the different customers:

shortly: each customer can be placed in its own VRF Virtual Routing Table giving you built in separation between the different VRFs

each VRF has its own end-to-end topology up to the ASA where a different VLAN subinterface is used.

see

http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/25sg/configuration/guide/vrf.html

This can greatly reduce the amount of filtering configuration and activity.

Hope to help

Giuseppe

Josh Mahar · ‎09-10-2010

Giuseppe,

Thanks for the quick response. VRF-lite looks very promising, I'm researching it further but forgot to mention that the 3750's are IP base... We may need a feature set upgrade but am checking with our vendor.

Jon Marshall · ‎09-10-2010

joshmahar wrote:

Howdy Cisco Community,

I am looking for some input to help me properly plan and elegantly design my logical network re-configuration. I'm specifcally hoping to hear from people that have done this before, not just theory based on documentation.

To start, here is some background and the environment setup. The infrastructure exists to host internet-facing web/app/db servers for various external clients. The primary focus here is to ensure client environments are isolated from one another. I inhereted the environment several years ago, so this restructuring has been a long time coming.

The equipment is as follows:

(2) Cisco 2950 (external switching)
(2) Cisco ASA 5520 (active/passive failover)
(4) Cisco 3750G-48TS in stacked configuration

External ethernet internet comes in through the two 2950's, then through the ASA's and finally to the core 3750 stack. Some ~100 servers are dual-linked and trunked to diverse switches in the stack.

There is an assortment of VLAN's, but fundamentally each host is trunked to only two:

a front-side where all web/app/db traffic operates, this includes external to internal (via ASA) for things like HTTP as well as server-to-server communication such as web to db
a back-end management for backups and monitoring, all hosts are connected to this with no filtering

My primary questions/concerns are:

Which device should handle core routing given the plan for moderately extensive filtering, the ASA or the core stack?
How can I anticipate the load impact of filtering on both the ASA and switching? I'm specifically worried about this on the switch stack.
If the switch is the default route, should I trunk the ASA into all the web server VLAN's, or create a single "external access" VLAN through which all inbound/outbound traffic flows?

Thanks for reading this far, I tried to keep this as short as possible. I am leaning towards making the 3750's the default route but am very concerned about switch load and filtering in the core.

Please ask for any additional information that may be helpful and thanks in advance for your responses.

Josh

Some points to think about -

1) if these are separate customers then you absolutely shouldn't use the 3750 to route between vlans. ACLs are in no way comparable to a stateful firewall so each customer should be on it's own vlan.

2) Having said the above a better fit would be to run your ASAs in active/active (assuming you are not terminating VPNs on the ASAs) so each customer has it's own virtual firewall. This is what contexts were primarily designed for ie. hosting multiple customers and keeping their traffic separate. Then a misconfiguration on the firewall only affects the customer whose context you are working on. However you may need to purchase context licenses for this setup and that may not be possible ie. cost.

3) if you went with 2) a logical next step is to use vrf-lite on the 3750 to maintain separate routing tables for each customer.

The above ie. 2 + 3 is the best way to keep customer traffic entirely separate. If you can't do that then absolutely use the firewall as the default-gateway and do not route off the 3750 switch.

There are some other things as well -

1) are the web/app servers different servers from the db servers ?

2) how are the customers connecting ie. on private lines or internet ?

Generally speaking web/app servers are separate servers from the db servers. In addition it is common practice to firewall between the web/app server and the db server. I'm assuming though that each customer vlan contains the web/app and the database server ie. you do not have a database server vlan for all customer database servers ? - if you do you need to change that.

So is the traffic flow -

customer -> web/app -> db server

where the web/app is routed off the ASA and the db server is also routed off the ASA ? Or is the web/app server and the db server per customer in the same vlan so that the traffic from the web/app server can go straight to the db server without going via the ASA ? This can be dangerous as the web/app server is open to external access and if it is compromised you can then go straight to the db server and this is the server that contains the company data. That's why it's important as to whether it is internet or private line/MPLS access.

As for the management vlan, you have separate NICs in each server by the sounds of it which is good. You can filter on this vlan but personally i would concentrate on the customer facing side before worrying too much about this aspect. Obviously make sure that ip forwarding has been disabled on the servers though so you cannot go via one NIC to the other.

Jon

Josh Mahar · ‎09-10-2010

Jon:

Thanks for the detailed response. Allow me to respond with some commentary:

I won't disagree that the filtering capabilities of the stack ACL's no way compare to the ASA's, however, I'm curious as to what extent I may be introducing security issues by doing it at the stack to begin with. If the ACL's have a default drop rule, only allowing the necessary traffic, what am I loosing? I know the ASA goes into packet inspection and such, but can you highlight some of the risks of letting the stack route and filter internally?
I didn't go with active/active on the ASA's because it would make traffic monitoring a little difficult. This isn't paramount however to a solid configuration though. However, there are several VPN's terminating at the ASA, but I could move these off to our old 515e's if necessary. I'm familar with the ASA contexts but again was a bit leary of load on the ASA's. We're also still on 7.2 as I've been avoiding 8 due to reports of issues from colleagues. Will take another look though.
I hear you loud and clear, will plan to use the ASA as the default route. The 150 VLAN limit may be a problem in the near future, but not initially.
1. The web/app and DB servers are different hosts. We need to isolate and filter between them, even if they serve the same client.
2. These all host public-facing sites. The "customers" or clients accessing the web servers come from the general internet population. All private lines/VPNs are for management of the servers and internal data transfer necessary for the web/apps to function.
You hit right on the goal of this reconfig. We need to isolate client servers form one another and also isolate web/app and DB servers for each client. This means each client will have at least 2 VLANs dedicated to them, one web/app and one DB.
Right now the Client A web/app and DB servers are in the same VLAN, but Client A has their own VLAN. Since the web/app servers must be exposed to the internet, we need to isolate them from both the DB server as well as all other client servers in case one is compromised.
I still plan to address the management VLAN with a filter at the stack, this is the only option I can think of. In my mind, this network represent the same security risk as having web/app and DB servers on the same VLAN since a compromised host can get to any other server via the management network. I suppose it depends on what level of control they get over the server.

Jon Marshall · ‎09-10-2010

1) There are a few main issue to do with using acls rather than a stateful firewall -

i) it can get quite difficult to allow return traffic because if for example you have an inbound acl on a L3 SVI to filter traffic from clients on that vlan initiating traffic to a remote vlan that acl also affects return traffic from those clients to connections initiated from a remote vlan to those clients. Reflexive acls can alleviate some of this but i don't believe the 3750 supports reflexive acls.

ii) because an acl considers each packet in isolation ie. not as part of an existing connection all sorts of packets can be sent with different TCP flags set etc. These packets would not get through a stateful firewall. Packets like these can have undesirable effects on the server TCP/IP stack

iii) there is no capability on a 3750 using acls to look beyond the TCP header. On Cisco stateful firewalls there are application inspection pieces of code known as inspect and these can check further into the packet ie. look at some of the actual data. However having said that they are not true application proxies and they don't have full understanding of the application. A lot of SQL hacks can pass right through a stateful firewall but would not necessarily get through an application proxy. IDS/IPS also plays a part here.

So i'm reluctant to say it's okay to use acls for external customers. ACLs can be fine for internal purposes within the company but the risk is too great in my opinion for external access.

2) If you use vrf-lite on the 3750 then you could actually not worry too much about active/active and context licenses are generally not cheap. The big advantage you would still get, even with vrf-lite, is that each customers firewall can be configured independantly of the others. As i mentioned, a mistake in one firewall does not affect any other customers.

3) Thanks for clarification.

4) Glad to hear it. This does mean you will be going in and out of the same pair of firewalls for web/app to db server communication so you need to factor that in. I have been involved with a number of these type of setups and often a 2 tier firewall approach was used where the web/app servers were behind one set of firewalls and then the db server was being another set. Again this may be overkill for your situation but worth bearing in mind if the load on the ASAs gets too much.

5) Agreed.

6) You are right in what you say and this does need addressing, i just wanted to focus on the front-end first. Ideally you only want to initiate connectons to the servers from the management vlan and only from specific addresses so that in the worst case scenario if a server was compromised then they couldn't just jump to other servers because they would not be coming from one of the allowed IP addresses. Any connections initiated from the server (syslog, SNMP traps etc.) should only be allowed to the syslog/SNMP servers etc.

Access to the servers can be locked down even more by authenticating users who are authorised to manage the servers. This has the added benefit of not being IP address dependant which can be a pain if you are remotely logged in via VPN and need to access a server to troubleshoot. The only problem with this is you would need something to intercept the authentication - ASAs can do this but i'm not clear on whether you are planning to run the management vlan off the ASAs or straight off the 3750 switches.

Jon

Josh Mahar · ‎09-10-2010

Thanks again, the plan is starting to get a little clearer. I'm going to read up this weekend and may have some more questions next week.

2) I'm not quite sure how VRF-lite at the switch would work in conjuction with the ASA as you describe without getting a context ASA license. How does VRF-lite alone give us firewall mistake protection, or did I misunderstand something? VRF only applies at the switching but the VLAN's are still trunked to the ASA, correct?

6) The return traffic/reflexive ACLs was something I forgot to ask about... not a problem with the ASA. The 3750 appear to support reflexive ACLs, but I can easily see how this can get very messy which is another reason to route/filter everything through the ASA. With that said, the only switch-based filtering I'll plan for is interior to the management VLAN. Yes, the idea is only to allow connections in the management VLAN from our management servers (backup, monitoring, etc...) and a little of the opposite for traps. From what I can tell in lab testing with VACL's (vlan filter) inside a VLAN, reflection isn't a problem for some reason as it is when traversing between two VLANs??

Looks like we'll need to upgrade the switch feature set, YAMW (yet another maintenance window).

Enjoy the weekend! Josh

Jon Marshall · ‎09-11-2010

Josh

2) You simply allocate the vlan interface on the 3750 into it's own vrf and then that vlan interface is a subinterface on the ASA which you can add access-lists to etc. Not quite as separate as using contexts but pretty good to be honest.

Jon

Josh Mahar · ‎09-13-2010

If implementing VRF on the stack, shouldn't the stack then become the default route for all hosts? I understand that VRF tables can keep my customers' routes seperate, but if I want to filter between the web and DB server VLANs for the same client it would no longer route through the ASA, leaving me with ACL filtering on the stack which is undesireable, correct?

VRF seems to be for much larger environments than what I'm working with here and it would also require a feature set upgrade. Not impossible, but if I spin up a number of VLANs for each customer/host function and route everything through the ASA, this should give me the same level of seperation and filtering I believe. The ASA would be configured to prevent any traffic between different clients.

Am I on track of off my rocker?

Jon Marshall · ‎09-14-2010

joshmahar wrote:
If implementing VRF on the stack, shouldn't the stack then become the default route for all hosts?  I understand that VRF tables can keep my customers' routes seperate, but if I want to filter between the web and DB server VLANs for the same client it would no longer route through the ASA, leaving me with ACL filtering on the stack which is undesireable, correct?
VRF seems to be for much larger environments than what I'm working with here and it would also require a feature set upgrade. Not impossible, but if I spin up a number of VLANs for each customer/host function and route everything through the ASA, this should give me the same level of seperation and filtering I believe.  The ASA would be configured to prevent any traffic between different clients.
Am I on track of off my rocker?

Josh

Yes you are off your rocker, enough said

Seriously though yes if using vrf-lite on the stack you would generally use the vlan interfaces on the stack as the default-gateways for the clients. It depends on how you want to filter between the same customer vlans ie. if you were happy to have customer A's web/app and dbase server vlans in the same vrf then you could route between them on the 3750 and filter using acls.

This has the benefit of keeping each customers vlans totally separate in terms of routing on the switch stack.

It has the drawback of not being able to statefully firewall between the same customers web/app and dbase vlans.

You could assign each vlan on the stack to a different vrf ie. customer A has 2 vrfs one for web/app and one for dbase. If you did that you may as well simply route off the ASA.

Again the benefit is keeping each customers routes completely separate on the switch stack.

Or as you say you can simply route everything off the ASA and not have any L3 vlan interfaces for the customer vlans on the switch stack at all so the 3750 acts simply as a L2 switch for these vlans.

As Giuseppe said, vrf-lite is great for reducing the amount of acls/filtering needed on L3 devices but if you are going to control everything via the ASA and not route anything (or at least the customer vlans) off the 3750 then you don't need any acls on the 3750 anyway.

Jon

Julio Garcia · ‎09-14-2010

i noticed you have 4 x Cisco 3750G-48TS in stacked configuration,

have you ever considered using private vlans for security? could give the flexibility and security you need between clients and between your own servers and clients and the best part is this will come into play before you even hit the ASA and works at wire speed level.