Peer Review -- Critique Network Design

[nsurginer]

Looking for some feedback on this network design for a small data center (see attached)  Some of the equipment I already have, but most will be purchased.

It will primiarly be colocation space, but we'll also have virtulization cluster and san storage, etc.  Please be honest.  I do have budget constraints, so I can't go all out so be realistic if you have suggestions or things I should upgrade now.

[Jon Marshall]

I guess this is a follow on from your L3 routed access-layer question in the DC. By the way if that question is answered could you mark it as answered as it helps idenitfy content that can be useful for others.

There is nothing that stands out as being wrong but the one main question i have would is bandwidth. You are using 4500 switches with SupIV's in them which gives you maximum of 6Gbps per module. If you fully populate one 48 10/100/1000Mbps linecard for example you have 48Gbps potential throughput with only 6Gbps to the swtch fabric so you have an 8:1 oversubscription rate.

This may or may not be an issue but it could affect both the customer feeds and SAN storage. In addition the ASA 5510s support 300Mbps throughput so again this may affect the SAN storage.

Other comments -

1) there is no sign of any firewalling between your DC and the ISPs - are you going to have some ?

2) How are you keeping customer traffic separate within the DC. Are you simply using vlans or are you looking at vrf-lite for example which can be run on both 4500 and 6500 to segregate the customer traffic into separate routing tables ?

3) How do the customers get to the DC - is it via the peering distribution 4500 with dedicated circuits or via the ISP connections ?

Jon

[nsurginer]

Jon,

Thanks for the reply, I went back and added the correct answer to my last post.

As far as the bandwidth, I'm going to be looking into that a bit more and making some changes depending on how much more fudge room I have.  We may also be doing a separate fiber channel for the SAN, so there's still a few things that are up in the air.

As far as your questions:

1.) I'm just planning on using firewalls at the access layer.  Our customers that colocate will get public hand-offs unless they want a managed firewall service from us.

2.)  I'll be using vlans to keep customer traffic separate at the access layer, although reading more about vrf-lite looks to be very nice alternative.  Thanks for suggesting that so I can do some more reading into it.

3.)  Customers will be using our ISP connections for their hosted equipment, and will also come into the data center via that route.  For any point to point, and local peering connections, I was looking at using the 4507s to terminate their connections into instead of directly into the core/distribution.  I figured that would give me more flexibility to control those connections and their routes.

Again, thanks for the reply Jon, any feed back is much appreciated.

[Jon Marshall]

nsurginer wrote:
Jon,
Thanks for the reply, I went back and added the correct answer to my last post. 
As far as the bandwidth, I'm going to be looking into that a bit more and making some changes depending on how much more fudge room I have.  We may also be doing a seperate fiber channel for the SAN, so there's still a few things that are up in the air. 
As far as your questions:
1.) I'm just planning on using firewalls at the access layer.  Our customers that colocate will get public hand-offs unless they want a managed firewall service from us.
2.)  I'll be using vlans to keep customer traffic seperate at the access layer, although reading more about vrf-lite looks to be very nice alternative.  Thanks for suggesting that so I can do some more reading into it.
3.)  Customers will be using our ISP connections for their hosted equipment, and will also come into the data center via that route.  For any point to point, and local peering connections, I was looking at using the 4507s to terminiate their connections into instead of directly into the core/distribution.  I figured that would give me more flexability to control those connections and their routes.
Again, thanks for the reply Jon, any feed back is much appreciated.

Thanks for marking previous post.

Glad to hear you will be looking at bandwidth. I appreciate it always comes to down cost so there is a limited amount you can do.

I'm still not entirely clear about this. You have 2 ISP connections which the customers come in on. What protects you ie. your DC from the internet. Is it the internet that you connect to through these 2 ISP connections ?

In addition once the customers are on their servers in the colo cabs what is to stop them just routing onto another customers server ? Are you planning to use access-lists on the vlan interfaces on the colo access-layer switches ?

The only firewalling i can see is to the SAN storage, is there nothing else the customers access ? Also what is to stop for example a denial of service attack from the internet or from one of the customers to another customer.

I appreciate i am not seeing the addressing or the configs of the devices but for a DC there seems a worrying lack of firewalling/segregation going on.

Note that with vrf-lite in use this will help segregate customer traffic but wouldn't necessarily help with the internet access.

The underlying design with L3 and access to distribution is fine but perhaps you could clarify how traffic flows from the internet into your DC and betweeen the customer drops and the rest of the DC ?

Jon

[nsurginer]

Jon,

We have two upstream providers we'll be doing BGP peering with.  That's our Internet connectivity, they are both 100 burst to 1Ge circuits.  I'll bring up more circuits with other providers as we grow.  There will be no firewall from the internet to the DC, again, they'll be at the access layer.

Colo customers that want to handle their own network will be given a public handoff.  Others that don't, will be put behind a managed ASA firewall.  I didn't diagram that part properly and need to clarify public handoff or, private managed firewall handoff on the access layer to the colo cabinets.    Since we'll likely have a lot of community clients, they might pass a good bit of traffic that stays in the data center, so I didn't plan on totally separating customer traffic.  I planned to have ACLs on the vlans to prevent access layer communications, and then allow flows at the distribution/core layer to cabinets that have packets for one another (email/http, etc.).  This is also where dedicated circuits from the 4507 will connect in to allow traffic from downstream customers to access other tenants for services as well as upstream for Internet connectivity depending on what they purchase.

So basically our data center infrastructure will be protected by the 5510s, and colo customers will be protected by their own CPE firewall devices, or a managed pair of 5510s likely.

It seems your stressing I'm not looking at security and segregation enough and need to re-evaulate the design better to accomadate that?  I'm a firm believer in KISS, so some of the things that I come up with are products of that.  Just throwing that out there.  =]

Again, thanks for the input!!

[Jon Marshall]

I'm a firm believer in KISS as well but only if you can keep it simple and in general DC's are often not that simple.

What i still would be concerned with, is that according to your diagram the entire switching infrastructure ie. all your 6500s and 4500s are exposed to the internet with no firewall. Perhaps i am not understanding something but can you see what i mean ?

Jon

[nsurginer]

Definitely understand, and you are correct.  Besides ASAs (5550 would be minimum as it will handle over 1Gb of throughput), what are other options to I have to protect the core?  On top of that, what else would I be gaining from having firewalls in front of the 6500s at the ISP level?  As it stands, I could block a few common ports and than allow IP after that, so I'm not sure what I gain there.  If that makes sense?

[Jon Marshall · Accepted solution]

nsurginer wrote:
Definitely understand, and you are correct.  Besides ASAs (5550 would be minimum as it will handle over 1Gb of throughput), what are other options to I have to protect the core?  On top of that, what else would I be gaining from having firewalls in front of the 6500s at the ISP level?  As it stands, I could block a few common ports and than allow IP after that, so I'm not sure what I gain there.  If that makes sense?

Note all the below assumes that what happens is you present public IPs to the ISP and the customers connect to these.

Well an example as i said would be if you were flooded with traffic from the ISP to one or more of your public IPs. This flooded traffic then uses a lot of your DC available bandwidth and there are already questions about available bandwidth. This could literally bring your entire DC down.

What if one of the customer IPs is used to jump onto a customer box and the customer does not have very good security on that server. The hacker is then free to start investigating your DC and acl's on vlans are not enough to stop a good hacker. At the very least if they gain access to the customer servers they could tamper with their data, modify/change public websites that you are hosting for that customer.

It's not clear exactly what IPs (public) are presented to the internet but you can bet they will be investigated and not by the people you want.

It's also not clear whether you are running private addressing in your DC and NATting the private addresses to public on the 6500 switches.

Ideally you would be blocking most ports and the customers would only be using a limited port set to access their servers and you should be able to lock down the source IP addresses as well to each customer. If you were hosting web services etc. yes you would need to allow any to access http on the servers but like i say the vast majority of ports should be blocked.

I know you have budget constraints and you want to keep it simple but if you only do one thing you have to secure your switching infrastructure and the customer servers from anyone on the internet. And i would be surprised to be honest if all customers would be happy without firewalling at the internet boundary as you are hosting their servers so you are responsible for their infrastructure.

This will as you say add cost because you need firewalls that can support high bandwidth but unless i am misunderstanding your setup you really do need them.

Jon

[nsurginer]

Jon,

Thanks for the reply.  I totally understand about the security and realize I have overlooked it to an extent.  I've also made a lot of other changes to the design and incorporated firewalls where they are critical at this point.  As we grow and revenue changes budget constraints, I'll want to look at adding firewalls at all of our edges, as well as better DDoS protection.

Attached is a new version of my layout.  Since I already own the 4507s, I'm going to use them until they max out and upgrade them.  I'll keep in mind they have 6Gbs on the line cards.

I have struggled learning the ins and outs of the 6500s are there seems to be a million caveats with all the different sup engines, and line cards and bus/fabric configurations, etc.

My biggest question is the sup720-3BXL on our edge.  If I put a 65xx line card in it that does NOT have the dfc card (an upgrade perhaps later on), will I loose the 3BXL functionality at all?  I know if you have non XL dfcs on the line cards than you'll loose the XL on the Sup engine supposedly.  Basically what I'm getting at is, can I still use the sup720-3bxl to peer with upstream providers and utilize the larger tcam for routing tables while using 65xx line cards (without a dfc) to connect to our peers via BGP?  In other words, do I have to use the sup engine's ports, or can I use the line cards as well?

Thanks,

Nathan

[Robin Martinez]

In regards to the downstream peers, are those connections to other branch offices or are they going to client/customer sites that access resources in your datacenter? If it's the latter I would think you'd want to consider some firewalls there as well.

Additionally, is there no need for redundancy for that section of the network? I noticed it was a single 4500.

[nsurginer]

Robin,

It's a single 4507 with dual sups.  Those connections will be customers coming in from point to point circuits, as well as local peering clients in the community.  These connections will be primarily public connections.  I figured one unit with dual sups would suffice, as they will be individual circuits (T1s, fiber ethernet, etc.) and not multihomed connections.