Fabric Interconnect is necessary?

[m1xed0s]

Hi, I am totally newbie in Cisco UCS world and I am reading currently...

The IOM/FEX inside B Series has to connect Fabric Interconnect switch (normally dual), which will then be connecting to North-bound switches. So why FEX inside B series can not connect directly to north-bound switches, like a NEXUS 5K/7K?

I know UCSM runs on Fabric Interconnect but connecting to Fabric Interconnect also introduce point of failure, increase the complexity for troubleshooting and hardware cost...

I am also a little bit confused on Top of The Rack and End of The Row Datacenter design with Nexus switches. Why Cisco can not make Nexus 2K TOR be able to perform local inter-connection forwarding? Why traffic has to be travel all the way to EOR and then come back, if 2 hosts connecting to the same 2K need to talk to each other?

[Walter Dey]

IOM in the UCS chassis is a FEX that can only be connected to a UCS fabric interconnect.

why: if your server needs fibre channel, we use a I/O adaptor called CNA (or VIC), which supports FCoE and IP over Ethernet.

Many customers are actually segregating FC and IP traffic on the Fabric Interconnect; this could be done on a N5k, but not on a N7k, which doesn't support native FC.

ToR/EoR Design does

De-Coupling of the Layer 1 and Layer 2 Topologies
Optimization of both Layer 1 (Cabling) and Layer 2 (Spanning Tree) Designs

Your statement is correct, but be aware, that this design

- gives you consistent latency

- allows central implentation of security (EoR)

[m1xed0s]

Thanks, but still not sure my confusion is gone 

The fex/Iom inside chassis should be able to connect to Ethernet and traditional FC, right? If so, why not connect to n5k directly? Or N7K for Ethernet and MDS for traditional FC? Btw, isn't spanning tree is gone in modem datacenter?

[Walter Dey]

IOM has internal 10 G ethernet connections over the chassis backplane, and external 10 G ethernet linkes

between IOM and Fabric Interconnect. To be exact: it's lossless Ethernet, which allows FCoE and IP running over the same wire.

see

https://supportforums.cisco.com/discussion/12205166/bandwidth-question-vic1240

https://supportforums.cisco.com/discussion/12200821/nics-esxi-show-40000-and-10000

and the attachment I posted there

Fabric Interconnect by default is in Ethernet End Host Mode, therefore no SPT between UCS and Northbound switch. Same for FC: NPV mode !

[m1xed0s]

Update: I think I got it now. Can I say FI is for b series chassis which is equivalent to N2K for c series or other servers?

I am not confused by bandwidth concerns. I just donot understand why fabric interconnect has to exist for b series servers? Except the ucsm is running on babric interconnect, any other reasons for its existence? Hp has chassis and it doesnot require some fabric interconnect.

[Walter Dey · Accepted solution]

Sorry ! there is a misunderstanding what the UCS architecture is all about !

FI is part of the architecture !

UCS chassis has no active network components (FC switches, Ethernet switches): HP (Dell, IBM,....) has !

UCS only has IOM=FEX, which acts as remote line card of the FI, similar to the combination of N5k/N2K. FI is the control plane for every IOM !

Just tell me how long it takes, to add a new HP chassis to a existing installation, and compare this with UCS:

http://www.youtube.com/watch?v=RS231c_awFw

http://www.youtube.com/watch?v=g-_Oq1uEp7M

HP requires FC and Ethernet aggregation switches to connect different chassis !

Hope this helps :-)

You can interconnect Rack Mount servers (c-Series), by attaching them either to a Nexus 2K, which is connected to a FI, or directly connect them to a FI (which is possible since UCS release 2.2.1) directly.

[m1xed0s]

Thanks, I am indeed in learning process of cisco dc technology. 

It is definitely different from other vendors otherwise it won't be cisco :)