Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

ovt Bronze
Bronze

6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Hi!

Which mode (bus or crossbar) is used when Sup720 uplinks send traffic to other modules? The following output shows that BUS MODE is used, but I don't understand why -- Sup720 has 20Gbps channel to the internal crossbar. Also, "global switching mode" was "flow-through" until "fabric switching-mode allow truncated threshold 1" is configured (the default threshold is 2). Can anybody explain why ???

cat6506E#sh fabric switching-mode

Global switching mode is Truncated

dCEF mode is not enforced for system to operate

Fabric module is not required for system to operate

Modules are allowed to operate in bus mode

Truncated mode is allowed, due to presence of CEF720 module

Module Slot Switching Mode

1 Bus

2 Crossbar

5 Bus

-------------------------------------

cat6506E#sh module

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 48 48-port 10/100/1000 RJ45 EtherModule WS-X6148A-GE-TX SAL1117MG0E

2 48 CEF720 48 port 1000mb SFP WS-X6748-SFP SAL1131W4S3

5 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1052CGFR

16 REPLIES
Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Hi

Truncated mode is used for switching data between fabric enabled modules when there is a non-fabric enabled module in your chassis ie. the WS-X6148-GE-TX is a classic line card.

The switch has a threshold ie. number of fabric enabled modules present in the system before it can use truncated mode (which is better than flow-through mode by the way).

You have one WS-X6748-GE-TX and the default threshold is 2 so it would not use truncated until either

1) You added another fabric enabled module

2) You lowered the threshold

It would be interesting to see what mode the sup720 would be in if the classic line card was removed, not suggesting that obviously :), because i can't remember what it shows when there are only fabric enabled modules in the chassis.

HTH

Jon

ovt Bronze
Bronze

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Thx. for the replay.

This system _already_ has 2 fabric-enabled modules - 67* module _and_ the Sup720 itself (with uplink ports)! So, it's unclear why not use the crossbar for traffic between the Sup and the 67* module (note that the Sup is in the _Bus_ mode).

I've changed the threshold as indicated in my post and truncated mode has been enabled automatically. Do I need to reset the entire system to force the Sup720 into the _crossbar_ mode?

Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

I'm not sure you can do this. I think once you have a classic card in your chassis the 720 will show as running in bus mode and i think it has to so it can communciate properly with the classic line card.

It's a good point though. I wonder if the Sup720 is using it's fabric connection to talk to the 6748 which is kind of what truncated is all about.

Jon

ovt Bronze
Bronze

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Yes, Sup720 can use its crossbar channel. Look at this:

show fabric switching-mode

Global switching mode is Compact

dCEF mode is not enforced for system to operate

Fabric module is not required for system to operate

Modules are allowed to operate in bus mode

Truncated mode is allowed, due to presence of DFC, CEF720, Standby supervisor module

Module Slot Switching Mode

1 dCEF <- 6708-10GE

2 dCEF <- 6708-10GE

3 Crossbar <- 6748-GE-TX

4 Crossbar <- 6724-SFP

5 dCEF <- Sup720-3B

6 Crossbar <- Sup720-3B

7 Crossbar <- FWSM

Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

And i think that might prove the point although it would be nice if someone from Cisco could confirm.

Because you have all fabric enabled modules you are running in compact mode.

I would very much like to know if when the Sup720 shows as running in bus mode and there is a fabric enabled card in the chassis that it stll uses the 20Gbps connection to swithc fabric to talk with fabric enabled modules.

Jon

Hall of Fame Super Bronze

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Jon,

Truncated Mode

Truncated Mode is used when CEF256 and/or CEF720 line cards are installed in a chassis with a classic line card, but a crossbar switch fabric is present. In this mode, classic line cards will transmit both the header and the data portion of the packet over the DBus. CEF256 and CEF720 line cards will only transmit headers over the DBus and transmit the data portion of the packet over the crossbar switch fabric.

Truncated mode results in centralized forwarding rates up to 15Mpps. In Truncated Mode, since the CEF256 and CEF720 line cards use the crossbar switch fabric to transmit data, overall aggregate bandwidth can actually be higher than the 32-Gbps shared bus capacity.

Performance of DFC enabled line cards are not affected by truncated mode and performance remains the same regardless of the line-card mix in the chassis.

http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper0900aecd80673385.shtml

Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Edison

Thanks. Can you confirm what mode the supervisor would show in a "sh fabric switching-mode" if the switch is running in truncated mode ie.

are we seeing "bus mode" next to the sup720 because of the presence of a classic line card ?

Jon

Hall of Fame Super Bronze

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Yes, it will be bus and not truncated since it needs to communicate with classic cards.

Fabric-enabled cards would communicate to the classic cards via the Supervisor Module not directly onto the classic card.

For instance, from a x67xx module to a x61xx module, it will go truncated to the Sup720 and from the Sup720 to the x61xx will go completely in bus mode because that's the only way the x61xx will accept the packet.

Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Okay, last set of questions, promise. From cisco doc

=============================================

With a Supervisor Engine 720, traffic is forwarded to and from modules in one of the following modes:

•Compact mode-The switch uses this mode for all traffic when only fabric-enabled modules are installed. In this mode, a compact version of the DBus header is forwarded over the switch fabric channel, which provides the best possible performance.

•Truncated mode-The switch uses this mode for traffic between fabric-enabled modules when there are both fabric-enabled and nonfabric-enabled modules installed. In this mode, the switch sends a truncated version of the traffic (the first 64 bytes of the frame) over the switch fabric channel.

•Bus mode-The switch uses this mode for traffic between nonfabric-enabled modules and for traffic between a nonfabric-enabled module and a fabric-enabled module. In this mode, all traffic passes between the local bus and the supervisor engine bus.

=============================================

So

1) It seems to suggest above that in truncated mode more than just the header is sent - is this your understanding.

2) When we talk about the header, the first 64 bytes in compact and truncated mode respecitvely this is traffic that goes from the module to the Supervisor 720 - yes ?

3) If 2 is correct your doc link suggest that this data is transmitted over the Dbus. Whereas the doc i have quoted from is a little more ambiguous.

I guess where i am confused is this

When the sup720 is showing "bus mode" in the output of a "sh fabric switching-mode" does it still use the crossbar to receive the compact and truncated traffic from the fabric enabled modules.

Jon

Hall of Fame Super Bronze

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

This is one of those things when you have to read between the lines. Usually, those queries are handled by the Cat6k group which deals with inner architecture from this monster :)

The way I understand it is the Supervisor will be in truncated mode when communicating to/from fabric-enabled cards while bus mode when communicating to/from classic cards.

Therefore, to answer your question, it uses the crossbar to receive and send truncated traffic. Compact is out of the question the minute you insert the classic card.

Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Edison

That's the way i understand it as well.

Bit embarrassed to ask after all this time but is there any reason i can't rate your post. I have rated other posts before, it just won't let me rate this one.

Jon

Hall of Fame Super Bronze

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

No need to rate, this conversation made me revisit the Cat6k architecture once again.

It seems while we were posting, NetPro hiccuped and we lost our badges :)

Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

No fair, especially as you have just become green :)

Hall of Fame Super Blue

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Oleg

Apologies for hijacking thread, hope it's been useful :)

ovt Bronze
Bronze

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Well...

Unfortunately the discussion is going in the wrong direction...

It's still absolutely unclear what does "bus" (or "crossbar" or "dCEF") mean _in_front_of_ the Sup module. I.e.:

5 Bus

Your guess is that "it uses the bus to communicate with classic modules and crossbar to communicate with fabric-enabled modules". Yes, this makes sense. The problem, however, is that the "Global switching mode" was flow-through until I set the threshold to 1. This probably means that the Sup720 is _not_ considered fabric-enabled in this particular chassis with the modules installed. This, in turn, means that the Sup uplinks are operating in the bus mode.

So, my guess is, yes the Sup is using both bus and crossbar to pass data _between_ classic and fabric-enabled modules. And this is very well described in the Networkers 2007 presentations. _But_ the Sup uplinks are operating in the _bus_ mode. I.e. the entire packet, received thru the uplink port, is sent

to the DBus. I don't think we see "5 - bus" just because we have classic modules in the chassis and because the Sup needs to pass data between classc and fabric-enabled modules. Of course, I might be wrong.

Silver

Re: 6500: Crossbar mode or Bus mode on Sup720 uplinks ???

Why on earth do you have a 6148A-GE-TX (junk) blade in this switch at all? Can't you hear the switch running slower?

What is connected to that blade and what speeds are those devices using (10/100/1000)?

The bigger problem than the fabric switching-mode is the architecture of the 6148-GE-TX blade (Note that the 6548 has the same architecture on the port side, albeit CEF256):

Not sure you are aware, but that blade is configured as 6 groups of 8 ports. In other words, a 6 Gigabit blade, not 48 Gb (or 20 or 40). This is a wiring closet blade for *very* lightly used PCs connected at a Gig. NOT for a server farm, distribution, or core.

There are several problems with this blade:

1. There are not per-port buffers, only per-group buffers across 8 ports.

This can create buffering issues that *will* effect all 8 ports. The bad thing is, this can happen even if all the ports are running at 100 Mb!!!

2. Drop counters are not exposed to the cli because the 'hidden' pinnacle ASIC has no idea which port the buffer problem effects. This behavior can be modified by using the 'hol-blocking enable' hidden command on the ports, see next item. This effectively carves up the per-group buffer into smaller chunks and managing them per port in software.

3. Head of line blocking (HOL) is a *huge* issue with these blades. Basically a bunch of traffic can be *destined* for a port-group and starve the output buffers for all other ports in the group. This will essentially stop traffic to all other ports in that group until the HOL blocking issue dissipates, which could be never. This is particularly a problem with a heavily utilized server and/or using a port in the port-group as a span destination.

I have clients that have these blades who have been told by Cisco in TAC cases that they purchased the wrong (cheaper) blades and are using them for the wrong purpose and will need to use only 6 ports on the blade (every 8th port) to resolve their performance issue! Cisco documentation clearly states that these blades (6148-GE-TX, 6548-GE-TX) are *only* wiring closet blades.

Just my 2 cents since I see a lot of these conversations on these forums about the fabric switching-mode when the incorrectly used blades are a far bigger issue. Who cares how fast traffic gets switched between the blades when the 6148-GE-TX blade itself is 8 to 1 oversubscribed before the traffic even makes it to/from the backplane!!!

2811
Views
5
Helpful
16
Replies
CreatePlease to create content