Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Avoiding oversubscription on WS-X6904-40G-2T.

Hi.

I need to guarantee the 10G of bandwidth of couple of interfaces that are members of port channel on Cisco Catalyst 6500. Someone have best practice to do it.

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions
Hall of Fame Super Blue

Re: Avoiding oversubscription on WS-X6904-40G-2T.

You can ask as many questions as you like

It has actually made me revisit the 6500 architecture and i probably needed the refresher course. One thing i did say that was misleading was talking about shaping on egress because really in terms of oversubscription we are really concerned with ingress traffic ie. traffic coming from devices connected to the linecard.

The switch fabric connection itself is the limiting factor on egress ie. you cannot get more than 80Gbps into that linecard from the switch fabric anyway but you could certainly get more than 80Gbps into the linecard from devices attached to the linecard. However the WS-6904 only supports shaping on egress so that would be no use here.

In terms of policing i have never used that as a way to try and stop oversubscription although the 6500 does support ingress congestion avoidance. Whether your specific module supports it i can't say without doing some more digging as it doesn't specifically mention it although i may have missed it.

I am not sure how practical it would be or how efficient though ie. if you have 6 ports you need to rate limit to no more than 20Gbps then do you just apply it evenly between the ports which may mean some servers do get dropped packets while there is free capacity (in terms of the 20Gbps) left unused. I think you would either need to accept the inefficiences or have a very good idea of the level to set per device and you could see how this would be quite complicated and difficult to manage.

Your original post asked for best practice and i suspect, although i don't work or speak for Cisco, they may well say if you want to be sure of no oversubscription then enable performance mode but i still think in your situation because you only need two ports to run with no oversubscription that is wasting ports you could be using.

There is a person on these forums who is very good at QOS (and also 6500s) so i will see if he can add to this, especially the part about using rate limiting to address the oversubscription problem but again, as far as i can see if you only use 4 ports on one port group i cannot see how this will be oversubscribed.

Jon

24 REPLIES
Hall of Fame Super Gold

Re: Avoiding oversubscription on WS-X6904-40G-2T.

6904 is a line card for the 6500 with four (4) 10 40 Gbps interface.  Coupled with the fact that you are going to be using Sup2T which, potentially, brings the 6500's backplane speed to 80 Gbps. 

So my question is this:  What oversubscription are you talking about?

Message was edited by: Leo Laohoo Not reading the post properly.  :P

New Member

Avoiding oversubscription on WS-X6904-40G-2T.

That´s correct Leo but the card also can enable with operational mode of TenGigabitEthernet on this mode you have 16

TenGigabit  port with oversubscription of 2:1, but i need in this mode to guarantee  the 10G of bandwidth of couple of interfaces that are members of port  channel, do you have any clue?.
Hall of Fame Super Gold

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Ok, I read your post wrong. 

So you have a four-port card which can support 40 Gbps per port. 

I do believe you can enable over-subscription mode on the card.  Basically when you enable over-subscription mode, two of the ports will be PERMANENTLY disabled.  This ensures you have non-blocking, line rate 80 Gbps transfer rate.

New Member

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Hi Leo.

The card is enable on TenGigabit in this mode i have 16  TenGigabit with oversubscription of 2:1, but i need couple of interfaces 10G bandwidth guarantee.

Hall of Fame Super Gold

Avoiding oversubscription on WS-X6904-40G-2T.

The card is enable on TenGigabit in this mode i have 16  TenGigabit with oversubscription of 2:1, but i need couple of interfaces 10G bandwidth guarantee.

Same thing. 

Enable over-subscription and this will permanently disable port 2 and port 4. 

Stick two FourX adapter into port 1 and port 3 and you are guaranteed 80 Gbps line rate.

Hall of Fame Super Gold

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Go here:  http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/data_sheet_c78-696623.html

You will notice in Figure 4, ports 2 and port 4 will be disabled when you enable the over-subscription feature.

Hall of Fame Super Blue

Avoiding oversubscription on WS-X6904-40G-2T.

Use the following link and scroll down to the Port Performance Modes section where it gives you instructions on what to do -

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11-696669.html

Jon

Hall of Fame Super Gold

Avoiding oversubscription on WS-X6904-40G-2T.

Nice link, Jon! 

New Member

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Thank you Jon and Leo, sorry for the insistence is possible to have TenGigabit and oversubscription mode, and have the 16 port available and also 10G bandwidth to couple of interfaces. The performance mode leaves some ports permanently disable, i need all the ports availables on the card.

Hall of Fame Super Gold

Avoiding oversubscription on WS-X6904-40G-2T.

The performance mode leaves some ports permanently disable, i need all the ports availables on the card.

Exactly how many interfaces do you want guaranteed 10 Gbps? 

I am not sure if you can do traffic shaping with Sup2T. 

New Member

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Hi Leo.

I need at least 2 port on 10g?, there Is a QoS feature that can help me?

Sent from Cisco Technical Support iPad App

Hall of Fame Super Gold

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Ok, if you want 2 running at guaranteed 10 Gbps but you also want fourteen ports at 10 Gbps.

I am no authority on this but if you are able to configure bandwidth SHAPING to the fourteen ports, then this is the solution you should be looking for. QoS is not a good solution because trying to fine-tune QoS values is going to be a b1tch.

Sent from Cisco Technical Support WII App

Hall of Fame Super Blue

Re: Avoiding oversubscription on WS-X6904-40G-2T.

I don't think shaping is going to work here. You have to visualise the switch fabric connections as in and out.

In means traffic coming from devices connected to your WS-X6904 module and going to the switch fabric (assuming the destination port is not on the same module).

Out is traffic coming from other linecards from the switch fabric going to your WS-X6904 and then out to end devices.

Shaping inbound would require you to know what amount of traffic is coming into each port or divide the ports by the total amount of bandwidth and shape per port on that which would mean some ports would be underutilsed whilst others would be shaped and potentially having to drop packets.

Shaping outbound would not work because the limiting factor here is the actual switch fabric connection itself ie. you can only push 80Gbps into the linecard from the switch fabric anyway.

Looking at the document i linked to it looks like the actual groupings are 5 - 12 sharing a 40Gbps connection to the switch fabric and 13 - 20 sharing s 40Gbps connection to the switch fabric. But the principle remains the same as already outlined.

The actual architecture of the linecard is quite complcated because it can support both 10Gbps and 40Gbps port so i;ll have a read today to make sure i haven't steered you down the wrong path but the limiting factor is always going to be the connections to the switch fabric.

Jon

Hall of Fame Super Blue

Re: Avoiding oversubscription on WS-X6904-40G-2T.

If you absolutely must run all 16 ports then the only way to do this and ensure no oversubscription for two of the ports is to ensure that the remaining six ports in the port group do not exceed 20Gbps.

You cannot really do this with configuration on the 6500, you have to know that the six devices connected to the port group will never exceed 20Gbps between them. If you know this is the case then you could use all ports but that is a very difficult thing to be certain of and if you get it wrong then you impact upon the two ports you want to run with no oversubscription.

Jon

Hall of Fame Super Blue

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Leo

Thanks. I rarely use a CCO search anymore i just google it because i think it finds things better than the Cisco search engine.

Jon

Hall of Fame Super Blue

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Using performance mode may not be the best solution. If you enable performance mode you lose 8 ports. But you do not need to guarantee the remaining 8 ports run with no oversubscription, you only want that for two. So looking at the link i gave you -

In 10 Gigabit Ethernet performance mode, ports 5, 6, 7, 8 and 13, 14, 15, 16 will be active ie. basically the port groups look like 1 - 8 and then 9 - 16

so you could leave the module in oversubscription mode but only use 2 ports out of the 1 - 8 range for your guaranteed 10Gbps and then use the other 8 ports 9 - 16 for the ones you don't need a guarantee for.

In actual fact you could if you wanted run 4 ports in group 1 - 8 and then have them with no oversubscription and then run the remainig 8 ports in oversubscription mode which means you can use 12 ports in total rather than just 8.

It's up to you and it would mean the second port group was oversubscribed 2:1 but this may not be an issue for you.

Edit - was posting while you sent your last reply. You can't use all 16, the best you can get if you want to guarantee two ports with no oversubscription is 12 as described above (assuming my maths is correct ).

The reason you cannot run all 16 ports is because each port group maps to a 40Gbps connection to the switch fabric. If you only have four ports in the port group you have no oversubscription but as soon as you use another port in the same port group you have oversubscription.

And there is no way to select which ports within the same group should or shouldn't be oversubscribed, so you have to treat the port group as as a whole ie. you cannot select a subset of ports in a port group to run with no oversubscription and then have the remaining ports in the same port group be oversubscribed.

Jon

New Member

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Hi.

And thank you so much for the feedback, the supervisor 2T support VoQ, may be that's the solution.

Thank you.

New Member

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Thank you so much Jon, the divices will be VPC on Nexus 5K. Ok the only easy way out Is the performances mode?.


Thnks

Sent from Cisco Technical Support iPad App

Hall of Fame Super Blue

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Thank you so much Jon, the divices will be VPC on Nexus 5K. Ok the only easy way out Is the performances mode?.

You can still use oversubscription mode and use 12 ports as i have outlined. You would still get no oversubscription on those four ports connected to one port group and then you use the full eight ports on the other port group to get a 2:1 oversubscrption ratio.

It's important to bear in mind that oversubscription ratios are limits that you might never hit eg. if you have all eight devices connected to a port group then in theory that is a 2:1 oversubscription ratio because the devices can generate 80Gbps of traffic but they only have a 40Gbps connection to the switch fabric.

But the likelihood of all eight devices sending 10Gbps of traffic each all at the same time is very low. That is why it is useful to have some idea of the average amount of traffic each device generates ie. if you knew that any one time no more than 40Gbps is ever sent from all eight devices then you actually don't have any oversubscription.

But that is only an average and there may be times when they do exceed 40Gbps between them. In those times there will be oversubscription. That is what you have performance mode ie. if you absolutely must guarantee there is never any oversubscription.

But usually performance mode is used when you need to guarantee no oversubscription for the majority or all of the ports. This is not the case for you so that's why i suggested it may not be the best choice.

Balancing devices across port groups was the way we used to do it before Cisco introduced performance mode.

Like i say, i'll have a fulll read of that document to make sure there are no major differences with a dual linecard and if there are i'll post back but i think from what i have read that the same basic rules apply.

Jon

New Member

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Last question Even with rate limit?

Sent from Cisco Technical Support iPad App

Hall of Fame Super Blue

Re: Avoiding oversubscription on WS-X6904-40G-2T.

You can ask as many questions as you like

It has actually made me revisit the 6500 architecture and i probably needed the refresher course. One thing i did say that was misleading was talking about shaping on egress because really in terms of oversubscription we are really concerned with ingress traffic ie. traffic coming from devices connected to the linecard.

The switch fabric connection itself is the limiting factor on egress ie. you cannot get more than 80Gbps into that linecard from the switch fabric anyway but you could certainly get more than 80Gbps into the linecard from devices attached to the linecard. However the WS-6904 only supports shaping on egress so that would be no use here.

In terms of policing i have never used that as a way to try and stop oversubscription although the 6500 does support ingress congestion avoidance. Whether your specific module supports it i can't say without doing some more digging as it doesn't specifically mention it although i may have missed it.

I am not sure how practical it would be or how efficient though ie. if you have 6 ports you need to rate limit to no more than 20Gbps then do you just apply it evenly between the ports which may mean some servers do get dropped packets while there is free capacity (in terms of the 20Gbps) left unused. I think you would either need to accept the inefficiences or have a very good idea of the level to set per device and you could see how this would be quite complicated and difficult to manage.

Your original post asked for best practice and i suspect, although i don't work or speak for Cisco, they may well say if you want to be sure of no oversubscription then enable performance mode but i still think in your situation because you only need two ports to run with no oversubscription that is wasting ports you could be using.

There is a person on these forums who is very good at QOS (and also 6500s) so i will see if he can add to this, especially the part about using rate limiting to address the oversubscription problem but again, as far as i can see if you only use 4 ports on one port group i cannot see how this will be oversubscribed.

Jon

Super Bronze

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

There is a person on these forums who is very good at QOS (and also 6500s) so i will see if he can add to this, especially the part about using rate limiting to address the oversubscription problem but again, as far as i can see if you only use 4 ports on one port group i cannot see how this will be oversubscribed.

In another posting, Jon asked me if I had seen this thread, and whether I might comment.

Well, yes, I did see it, but I couldn't see where I could add much to what Jon already described in his postings, well maybe beyond the same technique ("knowing" how ports obtain bandwidth to/from the fabric) can be applied to other line cards and/or platforms or that I actually used it too. 

I also agree with Jon's comments about trying to police ingress to manage over subscription, so again, couldn't see where I could contribute.

New Member

Re: Avoiding oversubscription on WS-X6904-40G-2T.

Thank you Jon and Joseph, i just trying to find the best option to address this situation.

Sent from Cisco Technical Support iPad App

Hall of Fame Super Blue

Avoiding oversubscription on WS-X6904-40G-2T.

Joseph

Thanks for taking the time to have a look. Just wanted to make sure i hadn't missed anything.

Jon

1194
Views
50
Helpful
24
Replies
CreatePlease login to create content