Cisco Nexus 5596UP

[bunce.jake]

Hello,

I have two questions regarding traffic forwarding on the Cisco Nexus 5596UP platform.

1) Does all traffic from the 16 port slots hit the L3 daughter card to be routed, or is the IP routing table copied into CEF on the port ASICs locally?

2) Does the L3 daughter card have a maximum connection of 12 Gbps to the backplane?

If all traffic is sent to the L3 daughter card and the maximum backplane connection is 12 Gbps, this would indicate that the limit of routed traffic is 12 Gbps across the platform. If someone could confirm that would be great.

Thanks,

Jake

[Steve Fuller]

Hi Jake,

1) Does all traffic from the 16 port slots hit the L3 daughter card to be routed, or is the IP routing table copied into CEF on the port ASICs locally?

Not quite sure what you mean here in terms of "traffic from the 16 port slots". Traffic from any of the Nexus 5K switch ports will hit the L3 daughter card if that traffic is being routed, irrespective of the physical port that traffic is sourced from or destined to.

2) Does the L3 daughter card have a maximum connection of 12 Gbps to the backplane?

The L3 daughter card has 16 10GE connections to the switch fabric so a total of 160Gbps in total. If you take a look at the diagram I included in the forum post Nexus 5548 interface hardware configuration, what you'll see are Unified Port Controllers, which each provide fabric connectivity for 8 10GE ports. If you look at the two UPC at the top left you'll see both of these with a box marked "Expansion Module" around them. This is how the L3 expansion module gets 160Gbps (2*8* 10GE).

Regards

Regards

[bunce.jake]

Hi Steve,

Thanks, that's very informative. Am I correct in assuming that the 160 Gbps links to the switching fabric are "internal Etherchannels"? If so, are they suseptable to the same flow based issues that conventional Etherchannels are (http://packetpushers.net/the-scaling-limitations-of-etherchannel-or-why-11-does-not-equal-2/) ?

Regards,

Jake

[Steve Fuller · Accepted solution]

Hi Jake,

Exactly. It's seen as a single 160Gbps port-channel with 16 10GE links. The load balancing of traffic across this is exactly the same as any other port-channel on the switch, with the balancing configured using the port-channel load-balance ethernet command in global context.

Regards

[bunce.jake]

Hi Steve,

Do you know how to identify whether there is a bottleneck on the platform up to the layer 3 daughter card? I suspect there may be an issue in the future due to projected traffic profiles - though I'd like to understand the IOS tools to proactively determine bandwidth/internal Etherchannel restrictions on the bus. I found "sh platform fcfib" and "sh platform hardware fcfib", but neither outputs make much sense (perhaps to TAC they do )

Regards,

Jake

[bunce.jake]

Or if anyone else can help that would be great

Regards,

Jake

[Steve Fuller]

Hi Jake,

I don't know any commands off hand as I've never properly looked at checking for over-subscription of the Layer 3 module, but I've been taking a look around and found a few things that may be of use.

The first point I'd make though is that whilst the port-channel into the Layer 3 module can be a bottleneck, the over-subscription is not that high. When you install the Layer 3 module into the Nexus 5596 then your switch becomes an 80-port switch so the worst over-subscription you'll encounter is 5:1.

How badly you're affected by the over-subscription will depend upon the load balancing of traffic across the port-channel. Given a single flow can never be more than 10Gbps, then it'll only be when you get two or more flows hashed to the same link on the Layer 3 port-channel that you'll encounter over-subscription. I think if you enable source/destination port based load balancing with the port-channel load-balance source-dest-port command in global context then you'll get pretty good distribution of the traffic across all 16 member ports. And if we assume that at any given moment in time that some traffic is switched at Layer 2, this won't hit the Layer 3 module.

So a little in what I've found.

The first thing to know is that the Layer 3 module in the Nexus 5500 series switches uses the Broadcom BCM56840 Trident ASIC. As per the diagram I pointed at in my earlier post the Layer 3 module connects to the switch fabric (Sunnyvale ASIC) via 16 10GE connections through two Universal Port Controller (UPC), also known as Carmel ASIC.

The Broadcom Trident chip is a 64 port switch, so you'll only be using 16 of those 64 ports. You can see which ports are used with the show hardware internal bcm-usd info port-info command.

According to the show port-channel internal info all command the port-channel number is 127 and/or 128; this is determined by the fact that the members are the same as shown with the show hardware internal bcm-usd info port-info command.

I've not been able find a way of looking at the overall utilisation of the port-channel, but you can see the port counters for each of the individual member ports with the show hardware internal forwarding l3 counters command.

I've only looked at this in a lab and as I'm not able to generate sufficient traffic volumes I can't say whether the above commands will give you any indication of over-subscription, and so traffic loss, to the Layer 3 module. If you have these in production with the sorts of traffic volumes you're talking about you may see more.

Hope that helps.

Regards

[bunce.jake]

Steve,

You sir, are a legend. This is exactly what I was looking for!

Thanks a lot.

Jake