Avoiding oversubscription on WS-X6904-40G-2T.

[jmp780718]

Hi.

I need to guarantee the 10G of bandwidth of couple of interfaces that are members of port channel on Cisco Catalyst 6500. Someone have best practice to do it.

Thank you.

[Jon Marshall]

Leo

Thanks. I rarely use a CCO search anymore i just google it because i think it finds things better than the Cisco search engine.

Jon

[Jon Marshall]

Using performance mode may not be the best solution. If you enable performance mode you lose 8 ports. But you do not need to guarantee the remaining 8 ports run with no oversubscription, you only want that for two. So looking at the link i gave you -

In 10 Gigabit Ethernet performance mode, ports 5, 6, 7, 8 and 13, 14, 15, 16 will be active ie. basically the port groups look like 1 - 8 and then 9 - 16

so you could leave the module in oversubscription mode but only use 2 ports out of the 1 - 8 range for your guaranteed 10Gbps and then use the other 8 ports 9 - 16 for the ones you don't need a guarantee for.

In actual fact you could if you wanted run 4 ports in group 1 - 8 and then have them with no oversubscription and then run the remainig 8 ports in oversubscription mode which means you can use 12 ports in total rather than just 8.

It's up to you and it would mean the second port group was oversubscribed 2:1 but this may not be an issue for you.

Edit - was posting while you sent your last reply. You can't use all 16, the best you can get if you want to guarantee two ports with no oversubscription is 12 as described above (assuming my maths is correct ).

The reason you cannot run all 16 ports is because each port group maps to a 40Gbps connection to the switch fabric. If you only have four ports in the port group you have no oversubscription but as soon as you use another port in the same port group you have oversubscription.

And there is no way to select which ports within the same group should or shouldn't be oversubscribed, so you have to treat the port group as as a whole ie. you cannot select a subset of ports in a port group to run with no oversubscription and then have the remaining ports in the same port group be oversubscribed.

Jon

[jmp780718]

Hi.

And thank you so much for the feedback, the supervisor 2T support VoQ, may be that's the solution.

Thank you.

[jmp780718]

Thank you so much Jon, the divices will be VPC on Nexus 5K. Ok the only easy way out Is the performances mode?.


Thnks

Sent from Cisco Technical Support iPad App

[Jon Marshall]

Thank you so much Jon, the divices will be VPC on Nexus 5K. Ok the only easy way out Is the performances mode?.

You can still use oversubscription mode and use 12 ports as i have outlined. You would still get no oversubscription on those four ports connected to one port group and then you use the full eight ports on the other port group to get a 2:1 oversubscrption ratio.

It's important to bear in mind that oversubscription ratios are limits that you might never hit eg. if you have all eight devices connected to a port group then in theory that is a 2:1 oversubscription ratio because the devices can generate 80Gbps of traffic but they only have a 40Gbps connection to the switch fabric.

But the likelihood of all eight devices sending 10Gbps of traffic each all at the same time is very low. That is why it is useful to have some idea of the average amount of traffic each device generates ie. if you knew that any one time no more than 40Gbps is ever sent from all eight devices then you actually don't have any oversubscription.

But that is only an average and there may be times when they do exceed 40Gbps between them. In those times there will be oversubscription. That is what you have performance mode ie. if you absolutely must guarantee there is never any oversubscription.

But usually performance mode is used when you need to guarantee no oversubscription for the majority or all of the ports. This is not the case for you so that's why i suggested it may not be the best choice.

Balancing devices across port groups was the way we used to do it before Cisco introduced performance mode.

Like i say, i'll have a fulll read of that document to make sure there are no major differences with a dual linecard and if there are i'll post back but i think from what i have read that the same basic rules apply.

Jon

[jmp780718]

Last question Even with rate limit?

Sent from Cisco Technical Support iPad App

[Jon Marshall · Accepted solution]

You can ask as many questions as you like

It has actually made me revisit the 6500 architecture and i probably needed the refresher course. One thing i did say that was misleading was talking about shaping on egress because really in terms of oversubscription we are really concerned with ingress traffic ie. traffic coming from devices connected to the linecard.

The switch fabric connection itself is the limiting factor on egress ie. you cannot get more than 80Gbps into that linecard from the switch fabric anyway but you could certainly get more than 80Gbps into the linecard from devices attached to the linecard. However the WS-6904 only supports shaping on egress so that would be no use here.

In terms of policing i have never used that as a way to try and stop oversubscription although the 6500 does support ingress congestion avoidance. Whether your specific module supports it i can't say without doing some more digging as it doesn't specifically mention it although i may have missed it.

I am not sure how practical it would be or how efficient though ie. if you have 6 ports you need to rate limit to no more than 20Gbps then do you just apply it evenly between the ports which may mean some servers do get dropped packets while there is free capacity (in terms of the 20Gbps) left unused. I think you would either need to accept the inefficiences or have a very good idea of the level to set per device and you could see how this would be quite complicated and difficult to manage.

Your original post asked for best practice and i suspect, although i don't work or speak for Cisco, they may well say if you want to be sure of no oversubscription then enable performance mode but i still think in your situation because you only need two ports to run with no oversubscription that is wasting ports you could be using.

There is a person on these forums who is very good at QOS (and also 6500s) so i will see if he can add to this, especially the part about using rate limiting to address the oversubscription problem but again, as far as i can see if you only use 4 ports on one port group i cannot see how this will be oversubscribed.

Jon

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

There is a person on these forums who is very good at QOS (and also 6500s) so i will see if he can add to this, especially the part about using rate limiting to address the oversubscription problem but again, as far as i can see if you only use 4 ports on one port group i cannot see how this will be oversubscribed.

In another posting, Jon asked me if I had seen this thread, and whether I might comment.

Well, yes, I did see it, but I couldn't see where I could add much to what Jon already described in his postings, well maybe beyond the same technique ("knowing" how ports obtain bandwidth to/from the fabric) can be applied to other line cards and/or platforms or that I actually used it too. 

I also agree with Jon's comments about trying to police ingress to manage over subscription, so again, couldn't see where I could contribute.

[jmp780718]

Thank you Jon and Joseph, i just trying to find the best option to address this situation.

Sent from Cisco Technical Support iPad App

[Jon Marshall]

Joseph

Thanks for taking the time to have a look. Just wanted to make sure i hadn't missed anything.

Jon