Solved: Fiber and Copper Trunks/Etherchannels

lamav · ‎08-10-2008

Are there any design or architectural considerations to make when mixing fiber and copper gigabitethernet ports in a port channel? Is this practice discouraged and, if so, why?

Moreover, I am wondering what the qualitative difference is between fiber and copper for inter-switch trunks for switches that are sitting 10 feet away from each other....? The gigE standard for Cat6e is about 100 meters, so given a distance of less than that between two switches that need to be trunked up, why use fiber and pay the higher cost of fiber jumpers and modules?

Opinions anyone?

By the way, sorry for posting so many questions lately. I am involved in a network reference architecture project and such isses arise all the time. I really do the ned the expertise and insight of seasoned professionals.

Thanks

andrew.burns · ‎08-13-2008

Hi,

1) Cost. (You're wasting money on a fibre link when it's not needed)

2) Standards. (If you're mixing then you obviously don't care about link quality. What kind of fibre is it? OM1/2/3/OS1? Is the copper CAT5/5e/6/7?)

In a small company you can get away with anything if it works - but try defining a Data Center cabling standard which is distibuted to your cabling contractors which says you can mix and match anything you like for an etherchannel as long as it works...

HTH

Andrew.

View solution in original post

andrew.burns · ‎08-11-2008

Hi,

Mixing fibre and copper is more of an operational support issue than a design or architectural issue. Basically, never do this. From experience, I'd go further and say only use identical ports - because if the channel ports have different capabilities (jumbo, qos, whatever) then sooner or later it will bite you. Keeping the ports identical is one less thing that could cause you a problem.

Regarding the second point - you've pretty much answered your own question. If you don't need fibre then why pay extra to install it? Siemon have some good info on this subject, and all their white papers are worth reading:

http://www.siemon.com/us/white_papers/08-07-10-copper-fiber-options-data-center.asp

That said - you need to look to the future, for example if you're planning to move to 10GE in a year or two then installing fibre now will save you recabling later.

HTH

Andrew.

nourelrayes · ‎08-11-2008

Hi,

basically all ports on the same etherchannel should have the same capailities and if there any differences exist this may take down the channel or make an interface out of the channel

Joseph W. Doherty · ‎08-11-2008

Can't comment on mixing fiber and copper in the same port channel. Although, it does bring to mind, why.

As to using gig copper vs. fiber, especially for port channels, seems to work fine.

Had a client with a large data center where they wanted to upgrade the network equipment. The DC network server core was off to the side, the distance required fiber to reach the far side server rows. All uplinks from the server rows were fiber to support gig uplinks. I pointed out if they compared the cost of copper gig (using ordinary 10/100/1000 ports) to fiber gig, there's a considerable difference, not only with the port cost itself, but the port connectors needed for fiber. So, if they would relocate their network server core to the center of the data center, they could use port channel gig copper. They did and it seems to work fine.

The implementation costs also took into account putting in the new topology copper plant vs. adding to the exiting topology fiber plant. (Major purpose of the upgrade was to provide copper gig to servers, which until that time was still mainly 100 Mbps. With gig to the servers, they wanted to be able to increase uplink bandwidth.)

PS:

On the subject of port channels, copper vs, fiber, a newer issue is when to use 10 gig. Of course, 10 gig offers more bandwidth than 8 gig ports in a channel, and it avoids bandwidth hashing issues too. However, the channels provide incremental bandwidth jumps and the price of copper (again assuing you're using ordinary 10/100/1000 ports) is very attractive vs. 10 gig fiber. What can be a surprise, is although gig fiber ports are often much lest expensive then 10 gig ports, the connector costs can make 10 gig less expensive whey you get beyond more than several fiber port gig channel.

lamav · ‎08-11-2008

Andrew:

I guess that is the essence of my question: What are the operational differences/capabilities between fiber and copper gigE ports? Are there any? If not, then whats wrong with mixing them in the same etherchannel?

This question arises because a client has all their etherchannelized uplinks, trunks and crosslinks terminating on the same 24-port gigE fiber module in slot 1 of their 6500 switch. If that module fails, the whole switch is dead in the water. I think its a good idea to have some module diversity incorporated into the design.

Moreover, they only have one fiber card, the rest are copper. Hence, my question....

Thanks

VL

Joseph W. Doherty · ‎08-11-2008

That's a good answer to "why".

If there is an issue with mixing fiber and copper in the same channel, perhaps another option to consider is to stop using the fiber uplink and switch to channeled copper uplinks to separate line cards in the 6500.

andrew.burns · ‎08-12-2008

Hi,

There are two questions there - the second one about path diversity you've already answered, and for any solution it's always a good idea to do a failure analysis. In the end it usually comes down to cost/benefit, i.e. how much would a failure cost vs. how much to mitigate it. Mitigating a line card failure by spreading the channels across two line cards is easy to do, and won't cost anything if you already have that second line card.

Regarding operational differences/capabilities between fibre and copper GE ports - that's less obvious. If you are mixing fibre and copper then by definition you don't need fibre for the link - so why are you paying the extra for the sfp's/gbic's and cable on one link when it's not needed? If you can make one link copper then you can make both links copper, and save the fibre for where you can actually make use of it.

From an operational point of view whether it's fibre or copper can be down simply to the choice of sfp/gbic but a bigger issue (in my experience) is the port capabilities. Sooner or later you'll want to do QoS on the channel, or enable jumbo frames, or something else, and having different port capabilities will become an issue that you'll need to deal with. If you make a standard, and stick to it, it's one less thing that can cause you a problem later on. It might not sound like much but all these little things can add up and really make a difference to the stability of your environment.

HTH

Andrew.

lamav · ‎08-12-2008

Andrew:

While I appreciate your effort and time, Im not sure you have answered my question.

I asked if it is OK to mix fiber and copper for trunks and etherchannels, and you said no because you want the same operational capabilities.

OK, so my question is simply, are there any differences in operational capabilities between fiber and copper ie, QoS, Jumbo frames, etc?

I do, however, understand and agree with your point that, if possible, one should go with either all fiber or all copper....

Thanks

VL

andrew.burns · ‎08-13-2008

Hi,

The short answer is no. Things like QoS, jumbo, etc, are a function of the line card and interface characteristics and not a function of whether it's fibre or copper.

That's the beauty of SFP's/GBIC's, etc, as they hide the physical media from the switch. So, for example, if you have a channel on the same blade with one port copper and one port fibre (using spf's, or whatever) then as far as the switch is concerned they are operationally identical.

HTH

Andrew.

lamav · ‎08-13-2008

OK, Andrew. So given what you just said, why is it not recommended to mix fiber and copper?

andrew.burns · ‎08-13-2008

Hi,

1) Cost. (You're wasting money on a fibre link when it's not needed)

2) Standards. (If you're mixing then you obviously don't care about link quality. What kind of fibre is it? OM1/2/3/OS1? Is the copper CAT5/5e/6/7?)

In a small company you can get away with anything if it works - but try defining a Data Center cabling standard which is distibuted to your cabling contractors which says you can mix and match anything you like for an etherchannel as long as it works...

HTH

Andrew.

lamav · ‎08-13-2008

OK, fair enough....just want to understand.

Thanks for all your answers.

VL