Solved: Switching oversubscription rates

ciscors · ‎08-28-2007

Does Cisco have oversubscription guidelines/standards for all three switching layers? I heard they do but cannot find a link.

For eg: access should not be oversubscribed over 1:125 and so on...

If there's no link, can you guys tell me what do you typically oversubscribe at? This is no fancy environment but they may get VOIP in the future

Thank you

roadhouse1387 · ‎08-28-2007

Hi,

There is a white paper which outlines some of this stuff, its called design guidelines for campus networks or something similar, I dont have the link at the moment but it is there.

I cant recall any of the current Cisco edge switches which are oversubscribed within themselves so you would normally not need to worry about stack backplanes.

The rule of thumb I work to on the edge/distribution is 20:1 at the uplinks.

So if you have a gig uplink from your edge to the core/distribution then you can allow for a stack with 192 100meg ports, which gives 19.2gig of aggragate bandwidth on the uplink. Which is a stack of 4 x 48 10/100 ports (assuming you are not using the spare gig ports for hosts).

Naturally, if you have two up links, and are doing some load balancing over them for example HSRP groups or by using multiple vlans and HSRP. In this scenario, you would probably want any vlans to be configured across all switches in the stack for resilliancy. Then you can use more switches per stack because you effectevley have twice the uplink bandwidth (2 x 20:1), but there are a number of problems with large stacks..

First, it is easy to put too many ports in a subnet and you may run into broadcast issues.

This is the last thing you want in a voip environment.

A further problem can arrise if you are using non-stackwise switches and are limited to using gig ports for stack interconnects. In this case there is a danger that you can oversubscribe the inter-switch links (normally gig links). The aggragate bandwidth arriving at the uplink from all switches in the stack may increase the oversubscription ratio too much. If you are using 3750's with the stackwise then the same issue applies, only you have 32gig (assuming non 'E' variant) in the stack interconect, so you wont have to worry about interstack bandwidth.

You will be ok with 5 in a stack giving 24:1 if you are expecting light/medium traffic types, but I wouldnt go any higher than that. You may even get away with 6,7 even 8 switches in a stack up to a point if using load balancing across uplinks from top and bottom switches, but an uplink failure may lead to insufficent remaining uplink bandwidth being available for all traffic. Also think of the broadcasts over 384 ports ! even if they are spread over a number of vlans, the vlan may not suffer from broadcast issues but the effect will be felt on the uplink, its all forwarded on the trunk, just tagged..along with BPDU's..and we all know what happens if we start loosing a few of them from time to time.

I find Its best to stick to 192 10/100 ports per stack. If you need multiple 1000 host ports then factor that into the 20:1 ratio, then again is the host really capable of forwarding 1000 ? probably not. You could consider reducing the stack to 3 x 48 (144) and using 1000 ports for the remaining 4 or 5 gig, thus maintaing the ratio. If I had to provide a large number of gig ports to hosts, then I would either consider 10gig uplinks, which could prove expensive, or use gigabit etherchannell (flowbased concerns accepted) to maintain the 20:1 ratio. Either way, gig or 10 gig I think the ratios are the same.

As for distribution to core links then I would work on as close to 1:1 as possible. We all know that in some circumstances it is not possible or practical, so work on something in the area of 4:1, 8:1 max but it depends on traffic flows/patterns. I cant recall the cisco advice on that but I think it says 1:1.

I always work on 4:1 for server farms, server to edge connectivity.

Hope this helps.

Cheers

Shaun

View solution in original post

Joseph W. Doherty · ‎08-28-2007

How much oversubscription is acceptable is really dependent on what your hosts are doing.

I've seen many use ratios of 24:1 or 48:1 for user edge, and 4:1 or 8:1 for server edge, but take that just as a starting point.

One technique is start on the higher side of the ratios, but have equipment that can easily decrease the ratio. This can be accomplished by ugrading to higher bandwidth uplinks and/or using a channeling technology, such as Cisco's Etherchannel.

For VoIP, instead of just bandwidth, also consider whether the equipment can manage bandwidth, such as always sending VoIP traffic first.

ciscors · ‎08-28-2007

Just to confirm how to calculate oversubscription:

- 3750 access stack made of 5 switches - hence 250 10/100 ports approximately which can aggregate to 250 X 100 = 25000mbps

- from the stack, i have a 2gb or 2000mbps uplink to distribution

Hence, my oversubscription would be 25000/2000 = 12.5 : 1

However, if the access ports are gigabit ports, then it becomes 125 : 1

Also, I've seen oversubscription rates sometimes as x:y:z. What is the third parameter for?

roadhouse1387 · ‎08-28-2007

Hi,

There is a white paper which outlines some of this stuff, its called design guidelines for campus networks or something similar, I dont have the link at the moment but it is there.

I cant recall any of the current Cisco edge switches which are oversubscribed within themselves so you would normally not need to worry about stack backplanes.

The rule of thumb I work to on the edge/distribution is 20:1 at the uplinks.

So if you have a gig uplink from your edge to the core/distribution then you can allow for a stack with 192 100meg ports, which gives 19.2gig of aggragate bandwidth on the uplink. Which is a stack of 4 x 48 10/100 ports (assuming you are not using the spare gig ports for hosts).

Naturally, if you have two up links, and are doing some load balancing over them for example HSRP groups or by using multiple vlans and HSRP. In this scenario, you would probably want any vlans to be configured across all switches in the stack for resilliancy. Then you can use more switches per stack because you effectevley have twice the uplink bandwidth (2 x 20:1), but there are a number of problems with large stacks..

First, it is easy to put too many ports in a subnet and you may run into broadcast issues.

This is the last thing you want in a voip environment.

A further problem can arrise if you are using non-stackwise switches and are limited to using gig ports for stack interconnects. In this case there is a danger that you can oversubscribe the inter-switch links (normally gig links). The aggragate bandwidth arriving at the uplink from all switches in the stack may increase the oversubscription ratio too much. If you are using 3750's with the stackwise then the same issue applies, only you have 32gig (assuming non 'E' variant) in the stack interconect, so you wont have to worry about interstack bandwidth.

You will be ok with 5 in a stack giving 24:1 if you are expecting light/medium traffic types, but I wouldnt go any higher than that. You may even get away with 6,7 even 8 switches in a stack up to a point if using load balancing across uplinks from top and bottom switches, but an uplink failure may lead to insufficent remaining uplink bandwidth being available for all traffic. Also think of the broadcasts over 384 ports ! even if they are spread over a number of vlans, the vlan may not suffer from broadcast issues but the effect will be felt on the uplink, its all forwarded on the trunk, just tagged..along with BPDU's..and we all know what happens if we start loosing a few of them from time to time.

I find Its best to stick to 192 10/100 ports per stack. If you need multiple 1000 host ports then factor that into the 20:1 ratio, then again is the host really capable of forwarding 1000 ? probably not. You could consider reducing the stack to 3 x 48 (144) and using 1000 ports for the remaining 4 or 5 gig, thus maintaing the ratio. If I had to provide a large number of gig ports to hosts, then I would either consider 10gig uplinks, which could prove expensive, or use gigabit etherchannell (flowbased concerns accepted) to maintain the 20:1 ratio. Either way, gig or 10 gig I think the ratios are the same.

As for distribution to core links then I would work on as close to 1:1 as possible. We all know that in some circumstances it is not possible or practical, so work on something in the area of 4:1, 8:1 max but it depends on traffic flows/patterns. I cant recall the cisco advice on that but I think it says 1:1.

I always work on 4:1 for server farms, server to edge connectivity.

Hope this helps.

Cheers

Shaun

ciscors · ‎08-28-2007

That's pretty detailed Shaun. Thank you so much!

Yes, dist -> core, I'm using 10GB dual-homed hence it's actually 20gb to the core

But I may have 5 stacks consisting of 5 switches each connected to distribution hence that's about 1250 ports

It's still oversubscribed only by 6.25:1 which isn't too bad

ciscors · ‎08-28-2007

I've gone through the white papers and SRND's with campus design but cannot find any talk about oversubscription. If you get a chance and can find that link, I would really appreciate it

Thank you!

roadhouse1387 · ‎08-28-2007

Hi,

Many thanks for the vote, glad to be of help (even if i do ramble on a bit).

I will have a look through the stuff I have at work tomorrow and post it up for you.

I hope i didnt dream it... :-)

Cheers

Shaun

Joseph W. Doherty · ‎08-28-2007

Yes, that's roughly how I too would see the oversubscription. (With two active uplinks, though, you'll unlikely see 2x performance because of many load balancing issues. I usually assume the second link will increase my effective bandwidth by 50%.)

As Shaun mentioned, if you moved to 10 gig uplinks, your would restore the 12.5:1 ratio.

In the real world, moving your clients from 10 to 100 or 1000, usually means they don't move any more data, just that it might take less time to move it. So, moving to gig on the edge, where your ratio got 10x worse, doesn't mean you get 10X worse performance, again assuming the overall traffic hasn't changed. In fact, it's possible, the users might see better performance. (I.e. if they aren't already competing for uplink bandwidth, their data will move quicker at the higher bandwidth.)

I don't recall seeing a three way ratio. Could you provide a link?

ciscors · ‎08-28-2007

I think I may have been reading the period (.) as a :

too stressed out ;)

Thx!