Nexus dual-sided vPC design with dual-connected

Unanswered Question
Feb 20th, 2012

Hi,

I would like to make a design with 4 Nexus 5596UP. 2 of them equipped with Layer 3 Expansion Module  so they can serve as core layer and the other 2 Nexus used as Layer 2 for aggregation server layer.

  • The 2 Nexus in the core layer will run HSRP and will peer with ISP via BGP for Internet connection
  • The 2 Nexus in the aggregation layer will be configured as layer 2 device and have FEX and switches connected to them

What I am ensure of is how the vpc and port-channel configuration should look like between the 4 nexus. What I was thinking is to run vpc between the 2 Nexus in the aggregation layer and between the 2 Nexus in the core layer. Than I was thinking of connecting each Nexus in the aggragtion layer to both Nexus in the core layer using port-channel and vice-versa.

I have made a drawing og the design I have described above:

NexusDesignNetPro.gif

Do you have any recommendations/comments or improments to this design?

Regards,

Laurent

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (2 ratings)
mohammedrafiq Wed, 02/29/2012 - 06:15

Hi Laurent,

1- Place all of your links between core and aggregation into  one port-channel (double side vPC).

2-Make sure the Domain id's are unique on core and agg switches.

2- Attach your FEX only to one agg switch, because if you daul homed them to both Agg switches , FCoE will not work ( its sees as exta hop).

4- Daul homed your servers instead to both FEX's.

Regards.

lap@axcess.dk Mon, 03/05/2012 - 10:22

Hi Mohammed,

Thanks a lot for your reply.

Regarding point 1 do you have an example on how the configuration looks like with double side vPC apply to my topology?

Regards,

Laurent

onestero Mon, 03/05/2012 - 15:37

Hi Lap

Configuration is much simple in that case:

For example on n5k1 and 2 create portchannel 10 and assign all links from 1 and 2 to 3 and 4 to it, and configure vpc 10.

On n5k 3 and 4 create portchannel 20 and assign all links from 3 and 4 to 1 and 2 to it and configure vpc 20.

on n5k1 and 2

on n5k1

interface port-channel 10

vpc 10

interface ethernet 1/1

description -=link to n5k-3=-

channel-group 10 mode active

interface ethernet 1/2

description -=link to n5k-4=-

channel-group 10 mode active

on n5k2

interface port-channel 10

vpc 10

interface ethernet 1/1

description -=link to n5k-2=-

channel-group 10 mode active

interface ethernet 1/2

description -=link to n5k-3=-

channel-group 10 mode active

on n5k-3

interface port-channel 20

vpc 20

interface ethernet 1/1

description -=link to n5k-1=-

channel-group 20 mode active

interface ethernet 1/2

description -=link to n5k-2=-

channel-group 20 mode active

on n5k-3

interface port-channel 20

vpc 20

interface ethernet 1/1

description -=link to n5k-1=-

channel-group 20 mode active

interface ethernet 1/2

description -=link to n5k-2=-

channel-group 20 mode active

HTH,

Alex

lap@axcess.dk Mon, 03/12/2012 - 07:36

Hi Alex and Mohammed,

I have made a drawing of my setup:

So I have the following:

- VPC peer link: 2x10G between Nk5-1 and Nk5-2 in vPC domain id 12

- VPC peer link: 2x10G between Nk5-3 and Nk5-4 in vPC domain id 34

- 1x10G between Nk5-1 and Nk5-3 and 1x10G between Nk5-1 and Nk5-4 bundle in port-channel 1234 vPC 1234

- 1x0G between Nk5-2 and Nk5-3 and 1x10G between Nk5-2 and Nk5-4 bundle in port-channel 3412 vPC 3412

1) Is that a correct setup for double side vPC?

2) Would it make sense to add more 10G link between Nk5-1 and Nk5-3 and Nk5-1 and Nk5-4 in port-channel 1234 and

between Nk5-2 and Nk5-3 and  between Nk5-2 and Nk5-4 in port-channel 3412? I am asking that because I have 4 extra SFP-H10GB-CUM cable.

Best regards,

Laurent

mohammedrafiq Mon, 03/12/2012 - 08:01

Hi Laurent,

wrote,

1x10G between Nk5-1 and Nk5-3 and 1x10G between Nk5-1 and Nk5-4 bundle in port-channel 1234 vPC 1234

- 1x0G between Nk5-2 and Nk5-3 and 1x10G between Nk5-2 and Nk5-4 bundle in port-channel 3412 vPC 3412

1-These all links will be in one port -channel not two(1234 and 3412). so, crate po1234 on 5k2 and 5k4 as well and  place all 4 links into po1234.

2-Yes , it will be good to have 20 gig between instead of 10g if you have spare ports.

Regards,

Mo.

pls rate this.

Lucien Avramov Mon, 03/12/2012 - 08:14

additionally for 1), you can call the 'vpc' same number vpc 1234 on both so it's clearer in your mind, when you will add more vpc's later this will help you.

When you look into this type of design, the peer-link capacity must be equal or greater than the aggregate of the uplinks, so if you add more uplinks, remember to add more peer-links. Often we recommend > 20G capacity.

The peer-link is used as essentially backplane extension and when we allow it to be configured using a small BW profile we can have problems in these cases (e.g. throttling multicast traffic to certain ports down from the 40G arriving from the core to 2G available to the servers.) In the right design these issues are minimized but in other cases they will impact deployments and therefore it's important to understand when you use a peer-link < uplink capacity it 'will' result in impaired network performance for some common design cases.

lap@axcess.dk Mon, 03/12/2012 - 08:33

Hi Lucien and Mohammed,

Thanks a lot for your great advices.

Just to make sure I have it all right.

- vPC peer link right now has 20 G capacity if you look at my previous post

1) When I bundle all the ports between N5k in port-channel 1234 vPC 1234 that would represent 40 G (4*10G) between N5k as the 4 links will be in port-channel no?

2) Mohammed, you mean to add 4*10 G between N5k so the port-channel 1234 would represent 80 G (8*10G) is that correct? Then Lucien how many link I should add to the peer-link ?

Best regards,

Laurent

Lucien Avramov Mon, 03/12/2012 - 08:40

Yes your Peer-Link is 20 and you have 40GE uplinks, so you need a bigger

peer-link: 40GE or more to be safe for all possible deployments use

cases (and stick to our best practices), but in any case at least 30GE,

so if you 2x10GE fail on one side they can take the whole peer-link

20GE and you will have some additional control traffic on the peer-link,

hence needing another 10GE.

If you run some multicast you may need another one.

Remember the rule of growing your peer-link when you grow your uplinks

(not downlinks).

lap@axcess.dk Tue, 03/13/2012 - 01:07

Hi Lucien,

As  a picture speaks a thousand words I attach the final picture of the design taking in account your advices:

Please let me know if it´s correct also regarding vPC doamain id.

Thanks.

Best regards,

Laurent

Lucien Avramov Tue, 03/13/2012 - 03:42

I agree, some confusion still on the picture, so trying to upload mine to show you :-) Please note that the size of the peer-link should be 4 ( green links ) - I had drawn 2 there for space constraints.

lap@axcess.dk Tue, 03/13/2012 - 08:18

Hi Lucien,

Thank you very much for taking time to help. Now I got it.

I am planing to run HSRP for different VLANs between N5K-1 and N5K-2. Then N5K-1 and N5K-2 will iBGP peer to the local ISP in order to get MPLS prefixes from the diferent locations. Is there any limitation best practice regarding HSRP and BGP on Nexus 5596UP for this design setup?

Ports used for iBGP peering will be configured as routed port instead of VLAN interface on both Nexus. It is ok?

Regards,
Laurent

Actions

Login or Register to take actions

This Discussion

Posted February 20, 2012 at 4:43 AM
Stats:
Replies:12 Avg. Rating:5
Views:2846 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard