This discussion is locked

Ask the Expert: Catalyst 6500 Switch Architecture

Unanswered Question
Jul 17th, 2012

With Akshay Balaganur

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn from Cisco expert Akshay Balaganur about Cisco Catalyst 6500, one of the most widely deployed switches in the world. The "Swiss Army knife of network", can do routing, switching, security, wireless and almost everything that you would want your core switch to do.

Remember to use the rating system to let Akshay know if you have received an adequate response. 

Akshay might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Network Infrastructure sub-community discussion forum shortly after the event.  This event lasts through July 27, 2012. Visit this forum often to view responses to your questions and the questions of other community members.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 3.5 (2 ratings)
sr1482613 Wed, 07/18/2012 - 11:26

Hi !  Akshay !

This is Hyun-Goo Hank Kim.

Nice to meet you.

Actually, This is NOT about Catalyst 6500 Switch Architecture. It's about PFC.

When I was looking through a documentation about understanding of MSFC/PFC/DFC on Catalyst 6500 Switch, the documentation mentioned about EARL7 on PFC.

I just wanna know what EARL 7 is exactly.

Thanks.

Best Regards.

Hank.

abalagan Thu, 07/19/2012 - 13:53

Hi Hank,

EARL stands for Enhanced Address Resolution Logic. It is gives the 6500, the aility to do packet forwarding in hardware. EARL is nothing but a set of ASICs or CHIPs that does forwarding lookups at hardware level. The same job that a CPU or Proccesor does. By by doing forwarding lookups in hardware and not involving the CPU, helps us to achieve the Millions Packet/Sec rarte.

Let me give you some history about it. EARL was originally introduced in Catalyst 5000. Then came Catalyst 5500 , Catalyst 6000 and eventually Catalyst 6500. EARL has evolved in each generation. The SUp720 came woth PFC3, which hosts the EARL version 7. Now the latest SUpervisor SUP2T has PFC4 which is EARL8. 

Cheers,

Akshay

harizmthaha Sun, 07/22/2012 - 05:07

Hi

Akshay,

our 6513 switch doesn't supprts SSH. is this anything with the IOS versions?

Here's the show version result.

can u tell me whats the action should i take to resolve this..??

------------------ show version ------------------

Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICES_WAN-M), Version 12.2(33)SXH2a, RELEASE SOFTWARE (fc2)

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)

System image file is "bootdisk:s72033-ipservices_wan-mz.122-33.SXH2a.bin"

cisco WS-C6513 (R7000) processor (revision 1.1) with 458720K/65536K bytes of memory.

Processor board ID SAL1223ST4U

SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache

Last reset from s/w reset

26 Virtual Ethernet interfaces

90 Gigabit Ethernet interfaces

1917K bytes of non-volatile configuration memory.

8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

regards

Hariz

Leo Laohoo Sun, 07/22/2012 - 15:09
our 6513 switch doesn't supprts SSH.

Your IOS doesn't support SSH because the IOS downloaded doesn't support "crypto".

Download the IOS with "crypto" support and SSH will be configurable.

abalagan Mon, 07/23/2012 - 06:00

Hi Hariz,

Yes your IOS is non-crypto image. You would need a crypto enabled image. The one that ends with "k9".

Example:

s72033-ipservicesk9_wan-mz.122-33.SXH2a.bin

harizmthaha Mon, 07/23/2012 - 11:14

Hi Akshay & leolaohoo

  thanks for the informations.

regards

Hariz

srprasaad_nj Mon, 07/23/2012 - 02:42

Hi Akshay,

I am going through a source about 6509 and have got few doubts:

> Where does the IOS reside on 6509 , is it on SUP-Engine (32 or 720) with integrated MSFC and PFC cards?

> What is DataPlane and Control plane and how SUP engine controls both?

> Where CEF engine and CEF Forwarding table is located ?

What is functional  difference between CFC, CEF720 crad, dCEF256/720 cards ? Ar

Aslo, see below output when i login to module 5, it says SP and shows only a basic config, so does this mean that I am on PFC card of mod 5 .

6509-CORE-1#sh mod
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    8  CEF720 8 port 10GE with DFC            WS-X6708-10GE      SAL1532M7MD
  3   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL09444A1J
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-BASE     SAD072800PT
  7   48  48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX    SAD094900WD

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  44d3.ca96.fcc0 to 44d3.ca96.fcc7   2.3   12.2(18r)S1  12.2(33)SXJ1 Ok
  3  0015.629b.9fa0 to 0015.629b.9fb7   2.3   12.2(14r)S5  12.2(33)SXJ1 Ok
  5  000d.290f.c1d0 to 000d.290f.c1d3   2.1   7.7(1)       12.2(33)SXJ1 Ok
  7  0013.c4c2.c0c0 to 0013.c4c2.c0ef   1.2   8.4(1)       12.2(33)SXJ1 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Distributed Forwarding Card WS-F6700-DFC3C     SAL1532LUDB  1.4    Ok
  3  Centralized Forwarding Card WS-F6700-CFC       SAL09528ZT6  2.0    Ok
  5  Policy Feature Card 3       WS-F6K-PFC3A       SAD072704UL  1.1    Ok
  5  MSFC3 Daughterboard         WS-SUP720          SAD072801SC  1.2    Ok

Mod  Online Diag Status
---- -------------------
  1  Pass
  3  Pass
  5  Pass
  7  Pass

.

6509-CORE-1#remote login module 5
Trying Switch ...
Entering CONSOLE for Switch
Type "^C^C^C" to end this session


6509-CORE-1-sp#

abalagan Mon, 07/23/2012 - 05:56

1. Where does the IOS reside on 6509 , is it on SUP-Engine (32 or 720) with integrated MSFC and PFC cards?

    Yes the IOS is mostly stored in the bootflash located on the MSFC of the Supervisor.

    It can also be stored in an external compact flash.

2. What is DataPlane and Control plane and how SUP engine controls both?

    Lets look at how 6500 is designed. And tehn you will get your answer !

    Like any other router the 6500 builds its Routing and CAM table. But unlike routers, 6500 pushes this information into hardware chips called TCAMs( PFC/DFC). Now the PFC/DFC based on this information have the capability to forward the packets independently without engaging the CPU. NOw this forms the Data Plane.

    CPU is now not bothered about the data forwarding, since it is being taken care by PFC/DFCs.  So CPU can continue handling the management tasks like maintaining the protocols (routing, switching, cdp, vtp etc..), monitoring the Chassis and the heardware. This forms the Control Plane.

Since we have separeated the Control plane and Data Plane, it enables us to have a high availability feature when using the 6500. 

3. Where CEF engine and CEF Forwarding table is located ?

    Let us revisit our previous answer with a little more details. :-)

    The CPU first constructs the routing table and ARP tables. These are compressed again made as FIB and Adjaceny tables in CEF (Software CEF).  Next FIB and Adj tables are downloaded from the processor and programmed as HW FIB and HW adjacency in the HW TCAMs. HW FIB and HW Adj are also known as MLS CEF (Multi Layer Switching CEF).

    So the question is a tricky one. We have CEF in both software and hardware. But you can safley say that CEF is programmed in hardware, when referencing to 6500. Because you use MLS CEF the most in 6500.

4. What is functional  difference between CFC, CEF720 card, dCEF256/720 cards ?

I would suggest you to go through the following White paper which clearly explains the architecture of Different categories of line cards.

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html

If I have to explain in short, it would be like this.

Classic Line cards- Utilize the 32 Gbps shared bus.

Cef256 - Utilize the 8Gbps dedicated fabric channel for data communication.

Cef720 - Utilize the 20Gbps dedicated fabric channel for data commanucation.

dcef256- Cef256 Line card equipped with a DFC daughter card.

dcef720- Cef720 Line card equipped with a DFC daughter card.

  CFC is a daughtercard that provides centralized forwarding for the 67xx linecards(Cef720). The CFC is the base requirement for 67xx linecard operation and is a zero cost option. The daughtercard houses two ASICs that function only as a bus interface. As the name implies, the CFC is only used for centralized forwarding.

abalagan Tue, 07/24/2012 - 06:30

Hi ,

I missed to notice your last question. When you do a remote login module 5, you are actaully logging into the Switch processor. And hence you see "SP" at the prompt. It stands for the Switch Processor.

Cheers,

Akshay

srprasaad_nj Tue, 07/24/2012 - 15:05

Hi Akshay,

Thanks.. for your reply.

Please correct me if i am wrong...does SP and RP reside on MSFC or PFC cards

And is there anyway to login to RP as well ?

What config's we will see in SP and RP ?

abalagan Wed, 07/25/2012 - 06:57

Hi Sitarama,

Yes both the RP and SP reside on the MSFC.

When you are running in Native mode , you login to RP by default. If you wish to go to SP , you can use the "remote switch login" command. In Native mode you do all your configurations on RP only.  You would go to SP only check few show command outputs.

If you are running in hybrid mode ( IOS on RP and CatOS on SP), then it is the otherway round. You by default login to SP. To login to RP you can use a "session 15" command.

In hybrid mode , you will do the layer 2 configurations  like spanning-tree ,  vlans , trunking etc on SP. And you will do all the layer 3 stuff on RP.

It gets little confusing with Native and Hybrid modes. Feel free to post if you have any doubts with this.

Hope it helps. Please rate if you find it usefull.

-Akshay

ajay chauhan Tue, 07/24/2012 - 06:23

Hi,

How do we calculate vlan-port instances on 6500 ? what it has to do with STP instances? Also max limit depends upon what factors.

Thanks

Ajay

abalagan Tue, 07/24/2012 - 06:50

Hi Ajay ,

  I think it's best to start with defining what exactly counts as
"vlan-port"
 
  The sum of all logical interfaces (vlan-ports) = [Number of trunks on
switch x Number of active Vlans on trinks] + Number of non trunking
interfaces on the switch 
 
  Also, VLANs can be counted more than once; if VLAN 1 is carried on ten
interfaces, it will count as ten VLAN ports
 
  Now with that explained, let's look into the number that you are
getting. The number is a software/hardware limitation and is mainly
dependant on the kind of spanning tree that you are running in your
network. Pasted below are the different possibilities, the first number
is the total vlan-ports on the whole chassis, the second one is per
module, which is what you are hitting
 
         PVST+ 13,000 total 1,800/module
         RPVST+ 10,000 total 1,800/module
         MST 50,000 total 6,000/module

Could you please provide more information about your device like the Supervisor you are using and the IOS image being run.

1. show vlan virtual-port  
( This command is what you are looking for ! )

2. show spanning-tree summary totals
( Command displays the number of logical interfaces in the STP Active column.)

Cheers,
Akshay
huangedmc Tue, 07/24/2012 - 19:06

hi Akshay,

When we cable uplinks between switches/routers, we try to fan out the redundant ports, so that they're on different ASIC's. We were able to find the port to ASIC mapping's via Networkers slide decks for the Nexus 7K's, but the ones for Cat 6K's are too generalized.

I already checked out the Cat 6500 Architecture Whitepaper, and it too didn't show the exact ASIC to port layout for the CEF720 cards. (most of our line cards are 6748's or 6724's)

Could you please provide URL to reference that shows the ASIC to port mapping?

thanks,

Kevin

abalagan Wed, 07/25/2012 - 07:13

Hello Kevin,

Unfortunatley the port-asic mapping architecture documents for Line cards of 6500 are only internal.

But I can answer your query about 6748s and 6724s.

In WS-X6748-GE-TX ,  12 ports are mapped to one port Asic. And all tweleve ports are serialy placed eg ports 1-12 map to one asic, port 13-24 map to next asic.

In WS-X6748-SFP ,  again 12 ports are mapped to one port Asic like before. But they are not serially placed. The aricheture is like this.  First two asics have all odd ports. Next two asics have all even ports. So fisrts asic has ports form 1,3,5 so on til 23. Second asic has 25, 27 ,29 so on 47 . Third asic has even ports starting from 2 till 24. Fourth asic starts from 26 and ends at 48. You got it right !

In WS-X6724-SFP , 12 ports are mapped to each asic. Ports are placed in sequence.

Hope it helps. Please rate if you find it useful.

Cheers,

Akshay

huangedmc Tue, 07/24/2012 - 19:41

hi Akshay,

A few more questions:

1. Is there any document that shows what functions are performed in hardware, and what are performed in software on the Cat 6K's?

We'd like to utilize things like NBAR & WCCP, but wanted to make sure we don't over run the CPU, if they're done in software.

I suppose wccp L2 redirect/return is hardware accelerated, but what about wccp GRE method?

Also, when we do "show policy-map interface", many counters remain zero.

Why is that? Does it mean QoS is done in hardware?

These are just some of the examples...we'd love to get a full list of the services performed in hardware vs software.

2. Is there an FAQ page for the WiSM2 blade like this one (for WiSM1):

http://www.cisco.com/en/US/prod/collateral/modules/ps2706/ps6526/prod_qas0900aecd8036434e.html

We'd like to know how many WiSM2 blades we can populate in a 6500 chassis, and if there's the same limitation as WiSM1 blades.

How come there can be only four WiSM's in a 6509E, if there are two other service modules present?

What happens to the remaining two available slots? (9 total - 1 sup - 4 WiSM's - 2 = 2)

abalagan Wed, 07/25/2012 - 07:40

There is no exhaustice list of features performed in software.

You can refer teh data sheet of the supervisor enginer and it lists all the features supported in the hardware for that superviosr. For example here is a datasheet of SUP720.

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/product_data_sheet09186a0080159856.html

    WCCP GRE is supported in hardware.

      Please note that the wccp egress mode is NOT supported in hardware and will cause High CPU.

     Please refer the following document for wccp configurations supported in hardware for Cisco catalyst 6500.

     http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11-629052.html

Some of the reasons for switching traffic in software ( CPU) are following.

•Fragmentation

•Same interface forwarding (to generate ICMP redirects)
•ACL log
•ACL deny – no route packet (to generate ICMP unreachable)
•Forwarding exception (out of TCAM/adj space)
•Feature exception (out of TCAM space / conflict)
•SW-supported feature (crypto, nbar, GRE)

•TTL=1

•IP options
•Multicast path setup
•Multicast RPF drops
•Platform-specific traffic handling
•Forwarding path issues – requires troubleshooting
•Glean (Packets requiring ARP resolution) /

     Receive(Packets falling in the Receive case)

Here are few links to troublehooting high cpu issue on 6500. These documents all list some of the feature that are not supported in hardware and punt it to CPU.

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml

https://supportforums.cisco.com/docs/DOC-22037

2. show policy-map interface

http://www.cisco.com/en/US/tech/tk543/tk760/technologies_tech_note09186a0080108e2d.shtml

3. I have not worked with wireless, so do not have much idea about WISMs . I will get back to you on this.

Cheers,

Akshay

abalagan Fri, 07/27/2012 - 10:36

This is the document I could find for the WISM2.

http://www.cisco.com/en/US/products/hw/modules/ps2706/products_tech_note09186a0080b7c904.shtml

Regarding the number of WISMs you can have the points mentioned in the document are correct.

http://www.cisco.com/en/US/prod/collateral/modules/ps2706/ps6526/prod_qas0900aecd8036434e.html

There is cap on the mixed number of service modules you can have in the 6500. And that cap is 6 (including WISMs).

So anypoint if you are using multiiple service modules , then the max you can go is till 6 .

Cheers,

Akshay

afnet.rma Thu, 07/26/2012 - 05:35

Hi, I require some clarifications regarding deployment of Control plane policy in Cat-6500 switch. In a 6500 switch which is functioning normal in a production environment, are there any way to measure the traffic rates (bps) of different types of packets (BGP, HSRP, PIM etc.,) going to control plane. This will help in determining a holistic traffic rates for control plane policy. Also please explain the right way to calculate conform and excess burst values (be & bc).

abalagan Fri, 07/27/2012 - 10:28

Just to add some extra information, you can configure a Span session for the CPU and monitor it during the peak ours.

Do this for a weeks time. After comparing all the captures , you will get a rough estimate of the different control plane trafffic coming to CPU and their rates. You could also use " show ip traffic " but then its not purely control plane. It will show you all the software switched traffic.

kanherinoc Fri, 07/27/2012 - 11:45

Hi Akshay,

My query is regarding the control plane policing in 6500 switch. I have implemented CoPP in 65500 with different rate limits higher than the actual required for different classes of control plane traffic. I wish to tune this value for a near realistic one.

The show policy-map control-plane detail gives me the peak bps for each class for hardware counters (Hardware CoPP).

Whereas the software counters are displaying only the total number of bytes forwarded till now and last 5 minutes rate for each class of traffic.

Is there anyway to find out the peak bps of the traffic forwarded by software CoPP for each class?

Can we be able to find/calculate the peak bps using the statistics available in the software counter display?

Actions

Login or Register to take actions

This Discussion

Posted July 17, 2012 at 9:26 AM
Stats:
Replies:22 Avg. Rating:3.5
Views:10701 Votes:0
Shares:1

Related Content

Discussions Leaderboard

Rank Username Points
1 15,007
2 8,150
3 7,725
4 7,083
5 6,742
Rank Username Points
165
82
69
65
55