cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
12193
Views
7
Helpful
8
Replies

Default routes in Nexus 5k

aLeffingwell
Level 1
Level 1

Hey all,

So .. as of right now I have a pair of N5K's, down stream from them are from Fabric Interconnects and a UCS chassis.  Upstream is a stack of 3750's then ASA5510's. 

I am trying to backup the config to our TFTP server and I am getting 'no route to host'.. I tried to add a route, and found that N5K uses VRF's for routing?? .. After some looking I see there are two base VRF's 'management' and 'default'.. the management VRF has a default gateway entry and a single interface member (mgmt0).. when I look at the default VRF .. there are no interface members or routing entries.. Ok, I can handle that just add some interfaces and add a default gateway.  Then I get lost:

I'm able to access the UCS manager..... so how the heck is that even possible if there's no gateway defined anywhere (or maybe I'm missing something?).  My theory was: add all other ports but mgmt0 to the default VRF, and have the default gateway point out of the uplinks (a vPC).. but wasn't sure how that would affect anything and mainly just wanted to know how I was able to access the UCS manager in light of the fact that there is no default gateway anywhere that I could see...

Thanks so much in advance ! This NX-OS is really something else...

Kindest Regards,

ALAN

1 Accepted Solution

Accepted Solutions

Alan,

You not being able to ping the tftp server from the default vrf (global) is a correct behavior, because the tftp as you already know is part of vrf management and you can't ping it from default.   Remember, vrf management and vrf default have different routing tables and you can not mix them together.  Think of these 2 vrfs as 2 different devices, but in this case it is not a physical separation rather logical.

It that more clear now?

Reza

View solution in original post

8 Replies 8

Reza Sharifi
Hall of Fame
Hall of Fame

Hi Alan,

The Nexus-OS is pretty cool.  Once you get use to it, you don't want to use IOS anymore.

Ok, so you understand the difference between the management VRF and default.  Default is the equivalent of global routing table and the mgmt0 is in management VRF

Are the 5K layer-2 switches only or also layer-3?

If you are running them as layer-2 then all you need is default routes towards the 3750

If they are layer-3, are you running a routing protocol with the 3750?

Can you post your config?

HTH

Hey Reza!

Absolutely - just from scratching the surface of the documentation I think it's a very intelligent OS/design and am looking forward to implementing it properly/fully.

The 5k's are Layer 2 right now (I'm not 100% on this but a 'sh license usage' shows LAN_BASE_SERVICES_PKG as a 'no') afaik.  Again a bit sketchy on this.. from a logical angle, there are no layer 3 protocols running on the device so far.  Our management VRF does have a default route and shows a few directly connected entries though.

So - that makes sense that default is the equivalent of global routing - my initial thought was: Ok, add the uplink port-channel/vPC to the 3750's to the default VRF (along with all the other ports save mgmt0) and set the uplinks as the default route (interface destination not IP).  What do you think?

My only question in all that was: well then how can we access the UCS manager now - if there are no interfaces in the default VRF and there are no routes in the default VRF?? It just made me pause and think before changing stuff.  Very excited to hear your response!!  I'm posting the running config now:

SGCN5K01# sh run

!Command: show running-config

!Time: Fri May 15 15:11:04 2009

version 5.2(1)N1(1b)

hostname SGCN5K01

no feature telnet

cfs eth distribute

feature lacp

feature vpc

feature lldp

username _____ password 5 $1$Bczq.c99$xbC4VG18Xg8M5knPFn2lG0  role network-admin

banner motd #Nexus 5000 Switch

#

ip domain-lookup

logging event link-status default

ip access-list classify_COS_4

  10 permit ip 10.132.13.0/24 any

  20 permit ip any 10.132.13.0/24

ip access-list classify_COS_5

  10 permit ip 10.132.5.128/25 any

  20 permit ip any 10.132.5.128/25

class-map type qos class-fcoe

class-map type qos match-all Silver_Traffic

  match access-group name classify_COS_4

class-map type qos match-all Platinum_Traffic

  match access-group name classify_COS_5

class-map type queuing class-fcoe

  match qos-group 1

class-map type queuing class-all-flood

  match qos-group 2

class-map type queuing class-ip-multicast

  match qos-group 2

policy-map type qos Global_Classify

  class Platinum_Traffic

    set qos-group 2

  class Silver_Traffic

    set qos-group 4

  class class-default

class-map type network-qos class-fcoe

  match qos-group 1

class-map type network-qos class-all-flood

  match qos-group 2

class-map type network-qos Silver_Traffic_NQ

  match qos-group 4

class-map type network-qos class-ip-multicast

  match qos-group 2

class-map type network-qos Platinum_Traffic_NQ

  match qos-group 2

policy-map type network-qos jumbo

  class type network-qos class-default

    mtu 9216

    multicast-optimize

policy-map type network-qos Setup_QOS

  class type network-qos Platinum_Traffic_NQ

    set cos 5

    mtu 9000

  class type network-qos Silver_Traffic_NQ

    set cos 4

    mtu 9000

  class type network-qos class-default

    multicast-optimize

system qos

  service-policy type qos input Global_Classify

  service-policy type network-qos jumbo

snmp-server contact Planview Infrastructure Dept.

snmp-server location  SunGard Austin

snmp-server user admin network-admin auth md5 0x40a021c27c5d155338065b7a16f8b0b9 priv 0x40a021c27c5d155338065b7a16f8b0b9 localizedkey

snmp-server community pvcorporate group network-operator

vrf context management

  ip route 0.0.0.0/0 10.132.5.1

vlan 1

vlan 13

  name vMotion-VLAN

vlan 16,20

vlan 51

  name MGMT-VLAN

vlan 52

  name NFS-VLAN

vlan 61-67

spanning-tree port type edge bpduguard default

spanning-tree port type edge bpdufilter default

spanning-tree port type network default

vpc domain 1

  role priority 10

  peer-keepalive destination 10.132.5.33

port-profile default max-ports 512

interface port-channel4

  switchport mode trunk

  speed 10000

  vpc 4

interface port-channel10

  description vPC peer-link

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  spanning-tree port type network

  vpc peer-link

interface port-channel11

  description sgcnscnt04

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 52

  spanning-tree port type edge trunk

  vpc 11

interface port-channel12

  description sgcnscnt05

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 52

  spanning-tree port type edge trunk

  vpc 12

interface port-channel13

  description SGCFIC01

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  spanning-tree port type edge trunk

  vpc 13

interface port-channel14

  description SGCFIC02

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  spanning-tree port type edge trunk

  vpc 14

interface Ethernet1/1

  description itype:trunk|dtype:switch|rdevice:SGCN5K02|rport:Eth1/1|notes:cisco_5548

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  channel-group 10 mode active

interface Ethernet1/2

  description itype:trunk|dtype:switch|rdevice:SGCN5K02|rport:Eth1/2|notes:cisco_5548

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  channel-group 10 mode active

interface Ethernet1/3

  description itype:trunk|dtype:switch|rdevice:SGCN5K02|rport:Eth1/3|notes:cisco_5548

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  channel-group 10 mode active

interface Ethernet1/4

  description itype:access|dtype:storage_head|rdevice:SGCNSCNT04|rport:e1A|notes:netapp_3220

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 52

  channel-group 11 mode active

interface Ethernet1/5

  description itype:access|dtype:storage_head|rdevice:SGCNSCNT05|rport:e1A|notes:netapp_3220

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 52

  channel-group 12 mode active

interface Ethernet1/6

  description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC01|rport:Eth1/1|notes:cisco_6248

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  channel-group 13 mode active

interface Ethernet1/7

  description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC01|rport:Eth1/2|notes:cisco_6248

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  channel-group 13 mode active

interface Ethernet1/8

  description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC02|rport:Eth1/1|notes:cisco_6248

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  channel-group 14 mode active

interface Ethernet1/9

  description itype:trunk|dtype:fabric_iconnect|rdevice:SGCFIC02|rport:Eth1/2|notes:cisco_6248

  switchport mode trunk

  switchport trunk native vlan 999

  switchport trunk allowed vlan 13,16,20,51-52,61-67

  channel-group 14 mode active

interface Ethernet1/10

interface Ethernet1/11

interface Ethernet1/12

interface Ethernet1/13

interface Ethernet1/14

  switchport mode trunk

  channel-group 4 mode active

interface Ethernet1/15

  switchport mode trunk

  channel-group 4 mode active

interface Ethernet1/16

interface Ethernet1/17

interface Ethernet1/18

interface Ethernet1/19

interface Ethernet1/20

interface Ethernet1/21

interface Ethernet1/22

interface Ethernet1/23

interface Ethernet1/24

interface Ethernet1/25

interface Ethernet1/26

interface Ethernet1/27

interface Ethernet1/28

interface Ethernet1/29

interface Ethernet1/30

interface Ethernet1/31

interface Ethernet1/32

interface mgmt0

  ip address 10.132.5.37/24

line console

line vty

boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.1b.bin

boot system bootflash:/n5000-uk9.5.2.1.N1.1b.bin

ip route 0.0.0.0/0 10.132.5.1


Kindest Regards,

ALAN

Hi Alan,

Looking at your conifg, since the 5Ks are utilized as layer-2 devices, the only default route you need is the one you already have under vrf context management (mgmt0)  The purpose of this default route is just for you to be able to manage the switches remotely. So, you don't need the global default route (below). Also, if you notice, it is pointing to the same ip as the one for mgmt0. 

boot system bootflash:/n5000-uk9.5.2.1.N1.1b.bin

ip route 0.0.0.0/0 10.132.5.1

Also, I am assuming all the SVIs for all your vlans are on the 3750, it that correct?

HTH

Hey Reza,

Right on, the SVI's are defined in the 3750.  Here's something interesting, if I ping from the management vrf, I can hit my tftp server: 10.132.6.144.  If I ping from the default vrf, with the configuration as you see above, there is no response.  That's why I added that global static route that shows up the same as the default gateway for the management vrf. 

I figured as much about the default route for the mgmt0 interface, good to know that assumption was correct.  Is the UCS manager served from the mgmt0 interface then?? Anywho, understanding is increasing but still I cannot ping the TFTP server..  Looking forward to getting this resolved and pending resolution I will grade and award correct answer here!

Thanks so much,

ALAN

Alan,

You not being able to ping the tftp server from the default vrf (global) is a correct behavior, because the tftp as you already know is part of vrf management and you can't ping it from default.   Remember, vrf management and vrf default have different routing tables and you can not mix them together.  Think of these 2 vrfs as 2 different devices, but in this case it is not a physical separation rather logical.

It that more clear now?

Reza

Hey Reza,

Awesome!  You helped me connect the dots!  When I  copied the config to tftp this time I specific the management vrf and  voila it worked.  Still lots to learn but you've put me on the right  path.  Thanks so much, marking as correct answer and rating other posts!

Kindest Regards,

ALAN

Glad to help Alan.

Thanks for the rating and good luck!

Reza

I am having the same issue with management VRF , the stupidest thing ever added to multilayer switches. My environment was even more complicated by adding every device management port to the same vlan attached to the same vlan (10). I am trying to undo this mess .  Now I need to buy a stupid switch(layer 2 1 gig ports) to physically separate the management network from the default VRF and access through firewall.  I think this is a consequence of not spending the big bucks for 7K for layer 3. Especially if look at design guides and marketing material 5K are not the best routers yet alone layer switch.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: