Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Community Member

Bandwidth distribution in Nexus 7009 Fabric-2 module

Hi Everyone!

I have a customer who is purchasing a Nexus 7009 switch with the following configuration:

Product Code

Description

Qty

N7K-C7009

9   Slot Chassis  No Power Supply  Includes Fans

2

N7K-SUP2

Nexus   7000 - Supervisor 2 Includes External 8GB USB Flash

2

N7K-F248XP-25E

Nexus   7000 F2-Series 48 Port 1/10G (SFP+) Enhanced

2

N7K-F248XP-25E

Nexus   7000 F2-Series 48 Port 1/10G (SFP+) Enhanced

2




N7K-C7009

9   Slot Chassis  No Power Supply  Includes Fans

1

N7K-SUP2

Nexus   7000 - Supervisor 2 Includes External 8GB USB Flash

1

N7K-USB-8GB

Nexus   7K USB Flash Memory - 8GB (Log Flash)

1

N7K-F248XP-25E

Nexus   7000 F2-Series 48 Port 1/10G (SFP+) Enhanced

1

N7K-C7009-FAB-2

Nexus   7000 - 9 Slot Chassis - 110Gbps/Slot Fabric Module

3

The customer wants to know how the fabric-2 module distributes bandwidth to eah slot and line card.  How is the 480Gbps bandwidth made available to each line card with five Fabric-2 modules?  Is there any architecture or design details that I can use to explain?  Is this bandwidth full or half duplex? The data sheet mentions each Fab-2 module capable of 110Gbps and five of them gives 550Gbps, so where's the 70Gbps going to, to the sup engine?

Would the ports on the line cards be on line-rate?

regards,

Nitin

2 ACCEPTED SOLUTIONS

Accepted Solutions
Hall of Fame Super Gold

Bandwidth distribution in Nexus 7009 Fabric-2 module

Each FAB-2 can push 110 Gbps PER SLOT and in a single direction (or 220 Gbps).

So for your F248, let's say you fully populate this with 10 Gbps.  So this means that you can potentially push up to 480 Gbps. 

For you to do so you'll need, as a minimum, two FAB-2.  With 2 FAB-2, you push up to 440 Gbps.  Not 1:1 but it's close enough.

The third card would be nice for redundancy.  But guess what, with the third card, you can fully do 1:1. 

Bandwidth distribution in Nexus 7009 Fabric-2 module

Hi,

I think Leo may have an error in his calculation in that he's taken the full-duplex Fabric capacity, but the half-duplex I/O module capacity.

Leo Laohoo wrote:

So for your F248, let's say you fully populate this with 10 Gbps.  So this means that you can potentially push up to 480 Gbps. For you to do so you'll need, as a minimum, two FAB-2.  With 2 FAB-2, you push up to 440 Gbps.

With two Fabric 2 modules you get 220Gbps of per slot capacity, so your 480Gbps would be slightly higher than 2:1 over-subscribed.

As stated, the Fabric 2 module provides 110Gbps per module, per I/O slot. If you have one Fabric module the maximum per I/O module slot throughput you achieve is 110Gbps, with two it's 220Gbps etc., all way up to 550Gbps with five Fabric 2 modules.

The CiscoLive BRKARC-3470 - Cisco Nexus 7000 Switch Architecture session has details of the Nexus 7000 Fabric architecture which I've taken a couple of slides from to help illustrate.

Each I/O module slot has two traces from its on board fabric ASIC to the chassis Fabric module, and when using Fabric 2 modules, each of these traces will be 55gbps. This is seen in the following slide.


So on the N7K-F248XP-25E I/O module where we have 48 10GE ports to fully utilise the available capacity of the I/O module you need all five Fabric 2 modules installed. This is shown in the next slide from the same presentation.


At the bottom left you can see "Five Fabrics: 480G F2/F2E module has maximum bandwidth".

The kit list you provided has three Fabric 2 modules meaning that any slot in the chassis can get at most 330Gbps fabric capacity. The N7K-F248XP-25E obviously has 48 10GE ports meaning a potential 1.44:1 (480:330) over-subscription.

To add to this, in order to more easily understand the number of Fabric modules required it helps to understand the naming convention of the Nexus I/O modules.

The I/O module name is essentially comprised of a number of fields in the format N7K-ABCCDE-FG, where:

  • A = I/O module family e.g., M or F
  • B = generation of the module e.g., either 1, 2 or 3
  • CC = number of ports on the module e.g., 2, 6, 8, 24, 32 or 48
  • D = port speed e.g., G=10/100/1000, X=10G, F=40G or C=100G
  • E = connector type e.g., T=RJ45, S=SFP, 2=X2, P=SFP+, Q=QSFP or F=CFP
  • F = the Fabric ASIC version of the I/O module e.g., either 1 or 2
  • G = the number of Fabric modules required for full line rate I/O

There's also an E or L suffix on certain I/O modules, but we can ignore that in this discussion. Although as Leo has rightfully pointed out in other posts, don't ignore the E when buying F2 modules. Always make sure there's an E at the end

Using the N7K-F248XP-25E as an example, the -25E at the end indicates that the I/O module is Fabric 2 based, and need five fabric modules for full line rate operation.

Hope that clear things up a little.

Regards

5 REPLIES
Hall of Fame Super Gold

Bandwidth distribution in Nexus 7009 Fabric-2 module

Each FAB-2 can push 110 Gbps PER SLOT and in a single direction (or 220 Gbps).

So for your F248, let's say you fully populate this with 10 Gbps.  So this means that you can potentially push up to 480 Gbps. 

For you to do so you'll need, as a minimum, two FAB-2.  With 2 FAB-2, you push up to 440 Gbps.  Not 1:1 but it's close enough.

The third card would be nice for redundancy.  But guess what, with the third card, you can fully do 1:1. 

Community Member

Bandwidth distribution in Nexus 7009 Fabric-2 module

Hi Leo Laohoo,

Thanks for the mail.  Do you mean to say because the customer currently has onlly 3x48-port 10gig line cards (480Gbps) per slot on 1:1, three fab-2 modules are sufficient? And when the customer adds more line cards in future, we may consider additional fab-2 modules (till 5 Nos) to main the 1:1 wire-speed?

Bandwidth distribution in Nexus 7009 Fabric-2 module

Hi,

I think Leo may have an error in his calculation in that he's taken the full-duplex Fabric capacity, but the half-duplex I/O module capacity.

Leo Laohoo wrote:

So for your F248, let's say you fully populate this with 10 Gbps.  So this means that you can potentially push up to 480 Gbps. For you to do so you'll need, as a minimum, two FAB-2.  With 2 FAB-2, you push up to 440 Gbps.

With two Fabric 2 modules you get 220Gbps of per slot capacity, so your 480Gbps would be slightly higher than 2:1 over-subscribed.

As stated, the Fabric 2 module provides 110Gbps per module, per I/O slot. If you have one Fabric module the maximum per I/O module slot throughput you achieve is 110Gbps, with two it's 220Gbps etc., all way up to 550Gbps with five Fabric 2 modules.

The CiscoLive BRKARC-3470 - Cisco Nexus 7000 Switch Architecture session has details of the Nexus 7000 Fabric architecture which I've taken a couple of slides from to help illustrate.

Each I/O module slot has two traces from its on board fabric ASIC to the chassis Fabric module, and when using Fabric 2 modules, each of these traces will be 55gbps. This is seen in the following slide.


So on the N7K-F248XP-25E I/O module where we have 48 10GE ports to fully utilise the available capacity of the I/O module you need all five Fabric 2 modules installed. This is shown in the next slide from the same presentation.


At the bottom left you can see "Five Fabrics: 480G F2/F2E module has maximum bandwidth".

The kit list you provided has three Fabric 2 modules meaning that any slot in the chassis can get at most 330Gbps fabric capacity. The N7K-F248XP-25E obviously has 48 10GE ports meaning a potential 1.44:1 (480:330) over-subscription.

To add to this, in order to more easily understand the number of Fabric modules required it helps to understand the naming convention of the Nexus I/O modules.

The I/O module name is essentially comprised of a number of fields in the format N7K-ABCCDE-FG, where:

  • A = I/O module family e.g., M or F
  • B = generation of the module e.g., either 1, 2 or 3
  • CC = number of ports on the module e.g., 2, 6, 8, 24, 32 or 48
  • D = port speed e.g., G=10/100/1000, X=10G, F=40G or C=100G
  • E = connector type e.g., T=RJ45, S=SFP, 2=X2, P=SFP+, Q=QSFP or F=CFP
  • F = the Fabric ASIC version of the I/O module e.g., either 1 or 2
  • G = the number of Fabric modules required for full line rate I/O

There's also an E or L suffix on certain I/O modules, but we can ignore that in this discussion. Although as Leo has rightfully pointed out in other posts, don't ignore the E when buying F2 modules. Always make sure there's an E at the end

Using the N7K-F248XP-25E as an example, the -25E at the end indicates that the I/O module is Fabric 2 based, and need five fabric modules for full line rate operation.

Hope that clear things up a little.

Regards

Hall of Fame Super Gold

Bandwidth distribution in Nexus 7009 Fabric-2 module

Thanks for the corrections, Steve. 

Community Member

Bandwidth distribution in Nexus 7009 Fabric-2 module

Thank you so much!  I have my question answered and convinced the custoemr as well!

1520
Views
5
Helpful
5
Replies
CreatePlease to create content