The new F3 series modules for the Nexus 7000 and Nexus 7700 series switches are amazing components of technology and the number of customers investing in them is very positive re-enforcement of that fact. With all of the capabilities available now and a rich roadmap for exciting new features, it’s no wonder customers are ordering them it like crazy. Whether it is fundamental L2/L3 switching, FabricPath or Overlay Transport Virtualization (OTV) or the other capabilities this module can do, there’s no end to the use cases. One of the less discussed features though is one we’ll focus on now, called the breakout feature.
“What is breakout?” you ask? Glad you asked, otherwise this would be a short article. Breakout is the ability to take a 40G port and configure it as four independent interfaces.
“What sorcery is this?! How can one interface become four? Are these sub-interfaces or some other similar idea?” First, remain calm - it’s not sorcery, it’s engineering and it provides a great migration path from 10G to 40G. Let’s dig into the details and start with the OSI system model and go to layer 1, the physical layer. To appreciate how breakout works let’s look at how 40G works over multi-mode fiber(MMF). The industry standard way to deliver 40G is to use a 12 stranded MMF cable with a multiple push on (MPO) end. Eight of the 12 strands are lit up making four pairs of 10G that are multiplexed together at the optical level to achieve a 40G link. It looks like the image below:
Breakout takes advantage of this physical layer requirement and allows the network administrator to configure a port to be used as 4 discrete, independent 10G interfaces. This allows you to connect 10G devices to your new F3 line card without needing a different line card to provide 10G support. These could be routers, switches, servers or any other device; as long as it’s 10G, it’ll work! So now you can buy a 40G module, run some of the ports in 10G mode and others in 40G mode. You can mix and match the configurations because F3 lets you do breakout on a port-by-port basis! An example is in the image below.
Now the big question, how do I get this feature on my switch? A couple of requirements are in order:
NX-OS 6.2(6) or higher
F3 or M2 module
If using F3, SUP2 or SUP2e are required
After you have that, it’s very easy to configure. Let’s start by taking a look at a switch with F3, but no breakout configured.
Eth5/1 -- eth routed down XCVR not inserted auto(D) --
Eth5/2 -- eth routed down XCVR not inserted auto(D) --
Eth5/3 -- eth routed down XCVR not inserted auto(D) --
Eth5/4 -- eth routed down Administratively down auto(D) --
Eth5/5 -- eth routed down XCVR not inserted auto(D) --
Eth5/6 -- eth routed down XCVR not inserted auto(D) --
Eth5/7/1 -- eth routed down Administratively down auto(D) --
Eth5/7/2 -- eth routed down Administratively down auto(D) --
Eth5/7/3 -- eth routed down Administratively down auto(D) --
Eth5/7/4 -- eth routed down Administratively down auto(D) --
Eth5/8 -- eth routed down XCVR not inserted auto(D) --
Eth5/9 -- eth routed down XCVR not inserted auto(D) --
Eth5/10 -- eth routed down XCVR not inserted auto(D) --
Eth5/11 -- eth routed down XCVR not inserted auto(D) --
Eth5/12 1 eth access down XCVR not inserted auto(D) --
We can see that with a single command we created 4 new interfaces – Ethernet 5/7/1 through 4! It’s also important to note that we didn’t need to reload or reset the module or any other component. NX-OS takes care of all the requirements. These interfaces can be configured as routed ports, switchports, port channels or Fabric Extender (FEX) interfaces. As shown below, we have a few different devices connected to these breakout ports.
N7K-1-F3# show int brief
Eth5/7/1 -- eth routed down Link not connected auto(D) --
Eth5/7/2 1 eth trunk up none 10G(D) --
Eth5/7/3 1 eth fabric up none 10G(D) 101
Eth5/7/4 -- eth routed up none 10G(D) --
N7K-1-F3# show cdp ne
Device-ID Local Intrfce Hldtme Capability Platform Port ID
Eth101/1/1 -- eth routed down Administratively down auto(D) --
Eth101/1/2 -- eth routed down Administratively down auto(D) --
Eth101/1/3 -- eth routed down Administratively down auto(D) --
Can breakout mode solve all problems? Unfortunately, no it cannot. It can help you migrate from 10G to 40G over time at your pace in a very cost effective manner though! Following are answers to some common questions about breakout that I’d like to address, based on conversations with many customers are below.
Can I connect a 1G device to a 10G breakout port? No, the device must be 10G, but a 10G FEX can be connected to a breakout port and provide 1G connectivity.
Can I connect LX/LH/ER/ZR/DWDM oo other optics to a 10G breakout port? No, the optical QSFPs that support breakout (QSFP-40G-SR, QSFP-40G-CSR, FET-40G) require you to use the same type of optic on both ends. For example, FET to FET, CSR to CSR, SR to SR, etc.
Is breakout the same as BiDi optics? No, BiDi solve the challenge by reducing the cabling required for 40G to 40G connections, while breakout focuses on 10G to 40G connectivity. BiDi optics are also supported on the Nexus 7000 and 7700 on both the F3 and M2 series modules.
Can I run breakout through my structured wiring system? Certainly! It might look like the equipment below.
Hopefully you see the benefit of the breakout feature with the capability to run both 10G and 40G connectivity on the same module on a port by port basis with the F3 or M2 module. Routers, firewalls, switches, NAS appliances or servers can be connected to breakout ports without any issues. Breakout is just one of many great capabilities of the F3 module.
Ron Fuller, CCIE No. 5851 (Routing and Switching/Storage Networking), is a technical marketing engineer (TME) on the Nexus 7000 team for Cisco. He has 21 years of experience in the industry and has held certifications from Novell, HP, Microsoft, ISC2, SNIA, and Cisco. Ron's focus is working with customers worldwide to address their challenges with comprehensive end-to-end data center architectures and how they can best use Cisco technology to their advantage.
Ron has had the opportunity to speak at Cisco Live on VDCs, NX-OS Multicast, and general design. He lives in Ohio with his wife and four wonderful children and enjoys travel and auto racing. He can be found on Twitter: @ccie5851.