Connecting a Switch to a 2232 FEX

Unanswered Question
Sep 5th, 2011

It is my understanding that the host ports on the FEXs can only have servers connected to them and not switch uplinks - like, say, from a blade switch. The reason, as I understand it, is that the host ports are hard-coded for PortFast and cannot be changed. So, of course you would never connect a dot1q inter-switch link to a port configured for PortFast. You can, but its certainly not recommended for obvious reasons.

But is that the ONLY reason?

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (4 ratings)
Jerry Ye Mon, 09/05/2011 - 18:07

It is not hard-coded for portfast/edge port. It has bpduguard is enabled and cannot be turn off.

One way to do it is configure the switch not to send BPDU, the HIF of the FEX will be able to connect to a switch. However, you can have a protential STP loop in your networ. Hense, this is not recommended.

HTH,

jerry

vbhutta Mon, 09/05/2011 - 19:11

From 5.2.1, by default the host interfaces are Layer 3. Hence you can connect it to switch if you want to keep it just Layer 3.

But if you want to keep the port as layer 2, then as Jeye mentioned, bpduguard is enabled by default.

Now when you connect a switch to FEX, it will send out a BPDU and hence the FEX port will get err-disabled.

If you want to connect a switch to those port then you need to enable bpdu filter on the switch interface which connects to FEX. This will prevent any BPDU going from switch to FEX and hence it will work out for you.

Always care should be taken when you enable bpdu filter since that will not help you if you have spanning-tree loop since you are not passing BPDUs.

For more reference:

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter1.html#con_1046083

Hope this helps.

ex-engineer Mon, 09/05/2011 - 19:52

Thank you , gentlemen!

Let's see if I got this straight....

BPDUGUARD is enabled by default on each host port and it cannot be disabled. By the way, since BPDUGUARD is typically enabled when a port is placed in PortFast mode, I got confused with my thinking....

Anyway, so if I did want to connect a switch to a Host port, I theoretically can achieve this by enabling BPDUFILTER on the switch's uplink port. This way the switch will not send a BPDU and the host port will not be forced into err-disable mode.

Correct?

Why did Cisco take this route in the design? Why did they intend to not have any switches connected to the Host ports? If indeed the FEX modules are supposed to emulate a linecard in a chassis-based switch, why not allow them to be configured as regular access or trunk ports?

Also, if the point is to exclude a port from the spanning-tree convergence process, why not hard-code the host ports for PortFast, too?  When we connect servers (non-bridges in general) to switch ports, we enable PortFast for convenience purposes since the hosts do not pose a bridging loop possibility.

I would love to have these questions answered.

Thank you!

vbhutta Mon, 09/05/2011 - 20:28

FEX host interfaces are edge ports (portfast enabled) as well as BPDUGUARD enabled.

However, the main reason you wont be able to connect a switch to FEX host interfaces is because of BPDUGUARD because that will err-disabled the port.

Hence you need to enable the bpdu filter.

FEX was introduced mainly to be at the access layer.

Architecture Flexibility

• Unified server access architecture: The Cisco Nexus 2000 Series offers a highly cost-effective access-layer architecture for 100 Megabit Ethernet, Gigabit Ethernet, 10 Gigabit Ethernet, mixed Gigabit Ethernet and 10 Gigabit Ethernet servers, Ethernet or unified fabric, physical or virtual server, and rack or blade server environments.

• Flexible physical topologies: The Cisco Nexus 2000 Series architecture allows decoupling of the Layer 1 and 2 topologies, therefore providing flexibility in designing physical architectures, including ToR, middle-of-row (MoR), and EoR deployments, while allowing quick expansion of network capacity and remote line-card portability across multiple parent switches. It is also space optimized for all these architectures.

Some outputs from my lab

SITE2-AGG1# show run int e102/1/1

!Command: show running-config interface Ethernet102/1/1------------------non-default configuration

!Time: Tue Sep  6 07:54:57 2011

version 5.2(1)

interface Ethernet102/1/1

  switchport

  switchport access vlan 100

SITE2-AGG1# show run int e102/1/1 all

!Command: show running-config interface Ethernet102/1/1 all-------------------Default and non-default configuration

!Time: Tue Sep  6 07:55:03 2011

version 5.2(1)

interface Ethernet102/1/1

  no description

  shutdown

  lacp port-priority 32768

  lacp rate normal

  switchport

  switchport mode access

  no switchport dot1q ethertype

  switchport access vlan 100

  spanning-tree port-priority 128

  spanning-tree cost auto

  spanning-tree link-type auto

  spanning-tree port type edge

  spanning-tree bpduguard enable

  no spanning-tree bpdufilter

  speed auto

  duplex auto

  flowcontrol receive off

  flowcontrol send on

  no link debounce

  no beacon

  delay 1

  snmp trap link-status

  logging event port link-status default

  logging event port trunk-status default

  mdix auto

SITE2-AGG1#

Hope this helps

Jerry Ye Mon, 09/05/2011 - 20:37

Yes, if you block BPDU on your switch, you shouldn't have issue to connect it to the N2K (except worry about loop).

I will not comment on the N2K design but I can suggest you to look at the below link about forwarding model of the N2K. N2K has no local switching intellegent where communication of different hosts in the same FEX will need to go through the N5K to forward traffic. I thinking is (not official) since the intellegent is still on the N5K and running STP will require allocation of switch resources, you can connect 12+ FEXs (N50x0 allows 12 FEXs max and number of FEXs allowed in the N55xx is much higher) into the N5K where if we allow all ports to become regular switchport (to listen to STP, etc.), how much memory/resource will required in the N5K? It would be a lot.

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter1.html#con_1197054

To your last point, I believe the default configuration on the N2K's HIF is spanning-tree port type edge (portfast). If you want to connect a dot1q trunk server into the N2K, you can change the HIF configuration to spanning-tree port type edge trunk.

HTH,

jerry

ex-engineer Tue, 09/06/2011 - 03:55

Thank you, folks, for your prompt answers. Really appreciate it.

I cant access the links. I am logged into the site, but it doesnt work. I keep getting "Forbidden File". I log in again and get it again....cycle... Thanks

Surya ARBY Tue, 10/04/2011 - 13:47

If you want to connect a switch with redundancy to some FEX, use flexlink instead of spanning-tree on the downstream switch.

ex-engineer Fri, 01/06/2012 - 04:56

In this thread, we cleared up the notion that you cannot connect a dot1q trunk FROM A SWITCH to a FEX because each FEX Host port is hard-coded for BPDUGUARD. So, any BPDU coming from a dot1q switched downlink will force the FEX Host port into the errdisable state.

HOWEVER, I am hearing that one CAN indeed connect a dot1q trunk from a switch to a FEX now - something has changed, or not (?). I dont know.

Can anyone at Cisco please clarify this? I am working with Cisco at a c.ient site and they have shown me that you can indeed do this. But I dont know if they are engaged in a science experiment or if this is now indeed a supported design.

Thanks

Amit Singh Fri, 01/06/2012 - 05:44

Do not get confused between the dot1q trunk port and the spanning tree running on the switches. You can run a dot1q trunk from a server NIC or CNA back to the FEX host ports. That does not mean that NIC or CNA will run spanning-tree to the ports connected to the FEX. 802.1q tagging is different than enabling the spanning-tree and sending BPDU's on a specific port. Even the some of the host PC/Laptop Nic's have capability to form a dot1q trunk to th switchport. You need 802.1q tagging on the CNA as you will be forwarding both SAN and LAN traffic on the same port and you need to tag the specific vlans for LAN and SAN traffic.

You can connect any device that is not running spanning-tree or sending the BPDU's on the ports connected to the FEX's.As mentioned by the other folks, either use BPDU filtering or Flex-links to connect any switch to the FEX ports.

Hope this helps.

Cheers,

-amit singh

ex-engineer Fri, 01/06/2012 - 06:00

Amit, I am NOT getting confused between a dot1q trunk from a NIC and a dot1q trunk from a SWITCH- that is why I specifically asked about a dot1q trunk from a switch, with the words "FROM A SWITCH" in capital letters. See above.

My question is whether we can now connect a dot1q trunk FROM A SWITCH with STP RUNNING to a Cisco FEX Host Port. The Cisco account team is saying YES and TAC is saying YES.

We need a definitive statement from Cisco.

Jerry Ye Fri, 01/06/2012 - 06:15

It is not recommended since it is not designed for that.

You can like what Amit said, block BPDU from the switch if it is running STP or use Flexlink.

HTH,

jerry

ex-engineer Fri, 01/06/2012 - 06:25

jerry, I do not want to bl.ock BPDUs. I definitely want to keep STP enabled on both ends.....so my question is about connecting a full blown switch with STP running to a FEX Host port.....

So what is Flex Link?

Amit Singh Fri, 01/06/2012 - 06:38

You cannot connect a full blown switch with STP running on a FEX-Port.

Please could you provide us with the TAC case number where TAC engineer has suggested that you can connect a switch running BPDU to a FEX port. I would like to talk to the TAC engineer and get his views on this.

Flex link is a L2 technology where you can use a L2 port as a back up of another one and you do not run STP on the ports configured as Flex-Links on the switch.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750x_3560x/software/release/12.2_53_se/configuration/guide/swflink.html

Hope this helps.

Cheers,

-amit singh

ex-engineer Fri, 01/06/2012 - 06:50

Yes, I remember now what Flex Links are. I cant use them for  2 reasons:

1. The downstream switch that I want to connect to the FEX is a non-Cisco switch.

2. The links will be connecting to a vPC domain, and I dont wan to lose the active/active capability.

As far as the account team for this client and the TAC person we dealt with, let me get back to you on that.

Thanks

a.aleksiev Fri, 08/10/2012 - 06:57

Hi,

Since you are going to use the vPC domain, your non-Cisco switch will use port aggregation (i.e. LACP) for his uplinks to the FEX.

This is loop-free architecture and you don't need STP at all. It can be completely disabled of BPDU filtered (depending from the switch features).

From this perspective the non-Cisco switch will never send BPDU to the FEX and the err-disable protection will never be trigered.

Probably that's the Cisco TAC eng. viewpoint...

Greets,

Alex

ekhoo Mon, 07/08/2013 - 08:59

Hi Amit and Jerry,

Need your input on Router connected to N2k. I understand the N2K is design for end host connection rather then connect to other downstream switch which may cause some issue as Jerry and you mention on above post.

Do you see any issue connecting a router such as Cisco 3900 to N2K? I am getting two conflicting information from below two post.

https://supportforums.cisco.com/thread/2165943

https://supportforums.cisco.com/thread/2133188

Leo Laohoo Fri, 08/10/2012 - 18:39

BPDU Guard is PERMANENTLY enabled by default on a FEX port.

The only way you can connect a switch to a FEX port is to DISABLE STP on the port of switch (used as the uplink to the FEX).

Actions

Login or Register to take actions

This Discussion

Posted September 5, 2011 at 5:25 PM
Stats:
Replies:19 Avg. Rating:5
Views:8210 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard

Rank Username Points
1 15,007
2 8,150
3 7,725
4 7,083
5 6,742
Rank Username Points
165
82
69
65
55