Small DC Switch Design

dan.letkeman · ‎05-16-2012

Hello,

I'm working on options for a small DC switch design. This DC has 5

virtual hosts with 10-20 guest vm's each. Each server has two quad

port gig nics with 6 of the 8 gig ports connected (3 for iSCSI and 3

for data or management. It also has two 3 node sans each with 2 gig

ports per node, a host of other small servers including voice servers,

management servers, asa firewall, and a few routers. Total of 50-60

ports as of right now.

Connected to the DC is 7 other buildings each with there own 1 gig

fiber connection serving about 3000 devices in total including

desktops, laptops, ip phones, wireless ap's, building automation,

alarm panels, etc....

Right now in each of the 7 buildings has a 3560G as an aggregation

switch connected back to the DC. The DC also has a few 3560G's and

3750G's for the sans and servers. The system seems to work ok for the

most part aside from micro bursts overwhelming the buffers on these

switches and the etherchannel trunks between them dropping a minor

amount of packets. QOS is configured for the voice network and there

are little to no complaints.

What I would like to know (costs being the biggest factor) is what

would be a better switch design for the current and future traffic in

this network. Some options I was thinking about are as follows:

I would needs at least 96 ports.

So option A is to go with a 4506-E bundle with 2 48 port line cards,

sup 6l-e and a WS-X4712-SFP+E or something of the sorts. And then

upgrade to the enterprise services license and do all of the routing

and switching for the DC on this one switch. Means little redundancy

and no failover.

Option B was to go with the same 4506-E bundle, without the extra

license and without the SFP line card and put in some sort of layer

three aggregation switch, possibly an me3600x.

Option C Is to go with the 4503-E, the SFP line card and the IP

Enterprise services license. And two top of rack switches, either

2360's or 4948's.

I would like to do some PBR on the aggregation switch, but I am unsure if the me3600x is capable of doing that.

I have no experience in this matter so any other thoughts or

suggestions would be appreciated.

Thanks,

Dan.

lgijssel · ‎05-18-2012

Although it is probably the most expensive solution, the 4506E option A is a very good solution.

Alternatively you could look at the 3750-X series.

A stack of three or four of these perhaps provides a little more redundancy but I would doubt if this solution is cheaper.

The alternative with a 3600ME is not the best option, this switch is intended more for the customer edge in provider networks, not as a DC switch. You are likely to encounter the same issues which you are already observing in the existing solution.

If you really want to do PBR, go for option A if you want to minimize the performance penalty it implies. Keep in mind this is your core switch and it is not recommended to perform this function there.

Personally I would try to refrain from it completely or do it elsewhere, for example at the aggregation switches in the various buildings.

regards,

Leo

dan.letkeman · ‎05-18-2012

Leo,

Thanks for the reply.

I had looked at the 3750-X series, but the price is the same or more for what I need. And the buffers are apparently not any larger than on the 3560G's that I have so why other than stacking would I want those?

I would consider using the me3600x as a pure layer 3 aggregation switch and having the pbr done on a different device. The price point seems to be lower than a 3750-x with 24 sfp slots and the 3600x has way bigger buffers designed to handle micro bursts.

My though was to set things up this way:

3560G's from each building (7 of them) ----->----- me3600x (layer 3)----->---- 4506E (server and sans)

PBR would not work on the aggregation switches in the buildings as I need it to be done in and around where the me3600x is. I would just consider hanging another 3560 off the me3600x instead.

Leo Laohoo · ‎05-18-2012

Your supervisor card for the 4500 is the 6L? Why? The price of the 6L and 7L are nearly identical.

Are you going to run MPLS. If the answer is YES (now or in the future) then 4500 won't work.

dan.letkeman · ‎05-18-2012

Really? all of the list pricing I see the 7L's are way more. Do you have a reference somewhere that I could take a look at?

I won'd be needing MPLS in my situation.

Leo Laohoo · ‎05-18-2012

 all of the list pricing I see the 7L's are way more.  Do you have a reference somewhere that I could take a look at?

When we priced for our deployment of 4500R+E we specified for 6E. Cisco recommended us to go to the 7E as the price were the same. So we went 7E.

If you don't expect to run MPLS then the 4500R+E solution would work.

Joseph W. Doherty · ‎05-19-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

If cost is the biggest factor, and the only notable issue now is occasional packet drops due to insufficient buffers, perhaps some buffer tuning would be something to consider.

A really important question to answer is how important redundancy. At lower port densities (3 to 4 stack members), the switch stack is less expensive than a chassis that supports redundancy. At very low port densities (2 to 3 stack members) the stack might be less expensive than even a non-redundant chassis.

BTW, the 3750-E/X offers much more performance than the original 3750 series. If offers wire-speed PPS and fabric per switch, twice the ring bandwidth and uses the ring, for unicast, much "smarter".

Perhaps a dual WS-C3750X-12S or WS-C3750X-24S for your core with dual MEC fiber etherchannl links to your aggregation switches? In the DC itself, you might also use 2960s to provide DC edge ports. For growth, a dual 3750-X stack would support four 10 gig ports which could be used with 10 gig servers or 10 gig SAN or as a 10gig link to other DC switches, such as the 2960S (which support their own stacking technology).

dan.letkeman · ‎05-19-2012

Joseph,

Thank you for the reply.

I have spent hour and hours tuning and researching buffer tuning on the 3560G platform, and I have found over and over again that it is designed to be an access switch and it just cannot handle micro bursts. So I have come to the conclusion that no amount of tuning is going to help.

I agree on the redundancy comment, and it is very important, however for the time being I can do without it, as we would look at purchasing a redundant supervisor in the near future as we expand.

I can see that the 3750X does offer much more performance, but there are numerous posts on it and the 3750E as well about how the buffers just don't hold up to micro bursts. I just seem to think that this would be a pointless investment if we are still faced with the same problem as we have now.

I did take a good look at the 2960s, but many people really suggest not using an access layer switch for iSCSI traffic, which we are using on our SAN's. I have not looked at it yet, but I think the 2360 would be a better choice in this type of design.

As far as the aggregation switch goes, it looks to me like the me3600x is a fair amount cheaper than the 3750x-24s and has way bigger buffers.

Joseph W. Doherty · ‎05-19-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

BTW, have you seen this https://supportforums.cisco.com/docs/DOC-8093? (Good explanation of buffer tuning).

Leo Laohoo · ‎05-19-2012

Hmmmm ... If you are very concerned with micro-bursts then I think you need to look into the Nexus 5K and Nexus 3K.

ALIAOF_ · ‎05-20-2012

There is also 4948E and E-F, check them out too. They have a nice buffer size.