Cabling Multiple Port Servers to Switches

bobelliot1321 · ‎12-28-2007

Quick question from a networking newbie about best practices - if I have a number of 1U rack mounted servers, say 24 or 48, each with multiple Ethernet ports, e.g 4 - RJ45 ports for each of primary data, redundant data, console / management, KVM in this example, would it be best to cable each same port type of all the servers to a different edge switch or leaf, eg all primary data go to switch 1, all redundant data go to switch 2, etc or is it acceptable to take same port types from the servers and 'spread' them to different switches?

For example servers # 1 through 6 primary data are connected to switch 1, servers # 7 through 12 to switch 2, etc. This would ultimately mean that a particular switch could contain more than one Ethernet cable from a particular server, for example might be cabled up with primary data, redundant data, KVM in different RJ45 ports, of course.

Thinking through this, I guess it makes tracing cables in case of faults developing much more complicated, but there might be advantages in cabling lay-out depending on how switches and servers are laid out in the cabinet. All switch ports are addressable, I guess, so there's no reason from a data flow viewpoint? There might be slightly better failsafe properties - if switch 3 was used for all console / management went down then all console / management would fail?

A colleague of mine said that there might be advantages in balancing and network traffic performance. I have not heard of this aspect, and couldn't find any info so far referring to it.

I'd appreciate any comments, or links to info on this.

Thank you very much, and have a great New Year

BobE

PAUL TRIVINO · ‎12-28-2007

Bob, a couple thoughts.

First, the KVM/management ports (ILO in HP server terms) - we've used a separate network for these "out of band" ports, on separate (cheaper) switches, usually 2950s. Saves expensive core ports. Yeah, if the 2950 dies you have no ILO/management but it won't die at the same time as the main ports/switch (if it does, go check your UPS!!!)

For primary/backup traffic ports, put one on one core switch and the other on the other, for redundancy. Maybe also say PROD PRI/BKP go to SW1/SW2, but TEST PRI/BKP go to SW2/SW1. If you mix this with HSRP for "PROD" server Vlans active on SW1, and "TEST" server Vlans active on SW2, you can balance load AND have failover.

HTH

Paul

PAUL TRIVINO · ‎12-28-2007

Forgot to add this: when there's issues, you're going to use arp/mac-address tables, CDP neighbors etc. to do tracing, tracing cables is the last resort. (For me anyway, but I'm lazy.)

Paul

xcz504d1114 · ‎12-30-2007

As the previous poster mentioned, the dual NIC on the server should be split between two core switches. IE Server #1 connects to Swtich #1 and Switch #2.

To make tracing out cabling a little easier for troubleshooting and future installs you should create a standard practice.

The way mine is setup is server #1 connects both ports to patch panel A-1 ports 1 and 2 (I always group them). Patch panel A-1 connects to patch panel A-2 (B-1 connects to B-2 etc, patch panel A represents rack A and so forth).

Patch panel A-2 port 1 goes to lets say a 6500 labeled switch #1 port 3/1, and the redundant connection goes to switch #2 port 3/1. So it's an exact mirror, and of course i put descriptions on the 6500's.

If you have the appropriate supervisor modules in a 6500 you can have them load balance between the two switches for you.

Also be careful with setting up the NIC teaming on the servers, you can create a broadcast storm and bring down your core switches pretty fast.

Hope that helps.