vNICs not working on B-series Blades

Answered Question
May 23rd, 2012

FI : 6248

Chassis : 5108

Blades : B200 M2 x 2

Uplinks : 5548UP

I'm setting up a pretty standard UCS deployment and have VPC uplinks from the 6248 fab interconnects to Nexus 5548s. The uplinks appear to be working correctly as I can manage UCS without any issues and can KVM to each blade. I've set up the mgmt ext pool and each of the blades gets an address correctly.

I initially setup a Xenserver blade and was able to connect to it on a server vlan that I created. However, since then I have reloaded the server and can not get it to work again no matter what I try. I've also tried load a Windows 2008 x64 server and while I am able to load the OS and configure the NICs within Windows (after installing the B200 drivers), there is no network connectivity .. it's as if the VLAN tagged traffic is not being forwarded upstream.

I have verified that all of our vlans are being passed on the uplinks between the FIs and the Nexus 5548s and I know it worked at one point so this has me puzzled.

I must have since misconfigured something but I can not for the life of me figure out what that was. I have recreated the VLANs, Service Templates, Services Profiles, etc. Each time I reload a blade, whether it be with Xenserver or with standard windows, I am able to see and configure the vNICS but I can pass no traffic. I can't do a ping from within the 6248s either since that cmd is not support, nor can I see any way of troubleshooting the traffic being passed.

Can someone please pass along some troubleshooting tips and/or places I can look to troubleshoot where the traffic is dying? I'm pulling my hair out!!

TIA!

I have this problem too.
0 votes
Correct Answer by roberbur about 1 year 10 months ago

Just to clarify,

Native VLAN and Default VLAN are two different things.  Native simply refers to the VLAN on a trunked 802.1q link that will not be appended a VLAN tag.  Default VLAN is the "automatically assigned" VLAN to all vNIC interfaces.  By default the "default vlan" is vlan 1, but this can be changed.

When creating a Service Profile (or Template for that matter) you can use the Simple, or Expert wizard.  The Simple wizard assumes each vNIC will operate as an "access port" by only being assigned to one vlan.  In expert mode you can select either the port to operate as an Access Port (single VLAN) or Trunk (multiple VLANs).  It's only with the Trunk vNIC configuration where you can select the "Native" VLAN radio button.

So let's say you have a Windows host that needs to be assigned to VLAN 10.

You could create a simple vNIC that acccess VLAN 10 only, or you could create a Trunk Interface, but set VLAN 10 as the native.  Either method will return the same result.

For OSes like ESX & Hyper-V you will indeed want Trunking and therefore allowing multiple VLANs.

Now to top it all off, though we present what looks to be "Access" and "Trunk" ports, there's actually no access ports.  What UCS does when you select a vNIC to be an Access port is just make a Trunk interface with the sole VLAN as the native. 

You can see this from NXOS if you look at "show int trunk". 

Regards,

Robert

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (1 ratings)
mipetrin Wed, 05/23/2012 - 18:34

Hi,

First thing to note, the way you manage the UCS and connect to it is via the management ports on the Fabric Interconnects. This includes all the KVM access (hence the reason why the IP management pool for the KVMs need to be on the same IP subnet as the Fabric interconnect management interfaces). Therefore, the fact that you can access the UCS to manage it is not indicative that the upstream network (N5K to UCS with vPC) is setup correctly.

Some things that you need to verify:

* The links and vPC between N5K and UCS are correct and in working order

* You are trunkning the correct VLANs on the links between the N5K and UCS

* You do not have a native vlan mismatch between UCS and N5K

* You are passing the correct vlans down to the server via the vNICs

* You have correctly defined the right vlan as native for the server vNIC

* If you are trunking all the way to the server, that the server NIC is setup to use the right vlan tagging

As there are numerous things that could be wrong, this will be easier to troubleshoot during a Live WebEx session so you should consider opening a case with the Cisco TAC.

Thanks,

Michael

sdunkin Thu, 05/24/2012 - 04:22

Thanks Michael - that is all useful information and you are correct re: the mgmt traffic not validating the uplinks.

The odd part, however, is that when I initially set everything up one of the blades was working on one of our server vlans, so I know I had the uplinks, vlans on trunked uplinks, vlans to the server chassis and vnic vlans all correct - I was able to route that traffic.

Since then I removed the vlans, setup new vlans, vnic templates, service templates, service profiles and assigned back to the same blade and I can not route to the installed blade(s) anymore. The first blade (currently running xenserver) sees all of the NICs, so I know that's not an issue. I installed Windows 2008 on the second blade and it sees the same behavior. It really seems like an upstream routing issue but nothing has changed upstream since I originally installed UCS and configured the Uplinks to the Nexus 5548s.

At the 5548s I can see that all of the appropriate vlans are trunked on the Port-channel (VPC). In addition, I see all of the vlans within the 6248s and also see that the vethernet interfaces for the vNICs to the chassis (ie. Ve870, Ve871, etc) are all configured as trunks.

Here is an output of the port configs if that helps at all - it appears to all be correct to me (it was all configured from the GUI). Again, any help or suggestions is much appreciated!!

Eth1/1        S: Server, Port-ch connected 1         full    10G     1/10g

Eth1/2        S: Server, Port-ch connected 1         full    10G     1/10g

Eth1/5        U: Uplink          connected trunk     full    10G     1/10g

Eth1/6        U: Uplink          connected trunk     full    10G     1/10g

Po1025        S: Server          connected 1         full    10G     --

Veth870       server 1/1, VNIC X connected trunk     auto    auto    --

Veth871       server 1/2, VNIC S connected trunk     auto    auto    --

Veth872       server 1/1, VNIC X connected trunk     auto    auto    --

Eth1/1/1      --                 connected 1         full    10G     --

Eth1/1/5      --                 connected 1         full    10G     --

Eth1/1/33     --                 connected trunk     full    1000    --

interface Ethernet1/5

  description U: Uplink

  pinning border

  switchport mode trunk

  switchport trunk allowed vlan 1,83,93,998

  no shutdown

interface Ethernet1/1

  description S: Server, Port-channel 1025

  pinning server

  switchport mode fex-fabric

  fex associate 1

  channel-group 1025

  no shutdown

interface port-channel1025

  description S: Server

  switchport mode fex-fabric

  pinning server

  fex associate 1 chassis-serial FOX1606H4VB module-serial FCH161272GG

interface Vethernet870

  description server 1/1, VNIC Xen_Mgmt_eth0_A

  switchport mode trunk

  no pinning server sticky

  pinning server pinning-failure link-down

  no cdp enable

  switchport trunk allowed vlan 998

  bind interface Ethernet1/1/1 channel 870

  service-policy type queuing input default-in-policy

  no shutdown

--------------------------------------------------------------------------------

Group Port-       Type     Protocol  Member Ports

      Channel

--------------------------------------------------------------------------------

1025  Po1025(SU)  Eth      NONE      Eth1/1(P)    Eth1/2(P)

e.nieuwstad Thu, 05/24/2012 - 04:47

wat does show port-channel and show interfaces trunk show on the nexus devices?

I assume you created a vPC between each FI and the nexus 5596's

sdunkin Thu, 05/24/2012 - 05:24

Those both look correct to me (we have several other VPCs running on the 5548s as well). I've included some output from both the 5548 and 6248 below. Something else occurred to me .. as the vethernet interfaces are configured as trunks, the traffic at the blades is actually tagged, yes?

I'm wondering then if the the configuration is correct - I changed one of the vlans to be the "native" vlan and some traffic started passing, which makes me think I'm just not doing the native/default vlans correctly. I wouldn't have thought I'd have to configured the default and/or native vlan as I'm not allowing the default vlan on any of my vNICs, nor is the default changed anywhere else in our network (it's always 1)

5548-A

Port channels to each 6248

17    Po17(SU)    Eth      NONE      Eth1/17(P)

18    Po18(SU)    Eth      NONE      Eth1/18(P)

sh int trunk

Po17          1,81,83,91-97,99,222-223,998-999,1002-1005

Po18          1,81,83,91-97,99,222-223,998-999,1002-1005

vPC status

----------------------------------------------------------------------------

id     Port        Status Consistency Reason                     Active vlans

------ ----------- ------ ----------- -------------------------- -----------

17     Po17        up     success     success                    1,81,83,91-

                                                                 97,99,222-2

                                                                 23,998-999

6248-A

--------------------------------------------------------------------------------

Port          Native  Status        Port

              Vlan                  Channel

--------------------------------------------------------------------------------

Eth1/5        1       trunking      --

Eth1/6        1       trunking      --

Veth868       1       trunk-down    --

Veth871       1       trunking      --

Veth872       1       trunking      --

Veth873       1       trunking      --

Eth1/1/33     4044    trunking      --

--------------------------------------------------------------------------------

Port          Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Eth1/5        1,83,93,998

Eth1/6        1,83,93,998

Veth868       93

Veth871       83

Veth872       83

Veth873       998

Eth1/1/33     4044

interface Vethernet873

  description server 1/1, VNIC Xen_Mgmt_eth0_A

  switchport mode trunk

  no pinning server sticky

  pinning server pinning-failure link-down

  no cdp enable

  switchport trunk allowed vlan 998

  bind interface Ethernet1/1/1 channel 873

  service-policy type queuing input default-in-policy

  no shutdown

When I changed this vnic so that vlan 998 was the native vlan, some traffic started passing (when I say 'some', I mean that I was able to ping the default gateway of that vlan from the blade itself, but I could not ping from the gateway back to the blade, presumably because of the default and/or native vlan mismatches). I don't want to change any of the default or native vlan information and I was under the impression I didn't have to modify that on the UCS in any way.

e.nieuwstad Thu, 05/24/2012 - 05:54

the trunk allowed list on the veth seems to imply that only vlan 998 is allowed I assume you want more. How are the nic profiles/templates configured.

sdunkin Thu, 05/24/2012 - 06:50

That particular veth/vnic only needs vlan 998 on it - I've created the vnic templates to only allow one vlan for the time being.

The nic profiles/templates are as simple as I could make them. Each one on only has a single vlan. Here is a picture of the VLAN config, vnic templates and the service templates and profiles. Notice that I'm not using the default or native vlan anywhere as I want all traffic to travel on the configured vlans (ie. no untagged traffic)

sdunkin Thu, 05/24/2012 - 11:08

Ok, so I came to a few realizations. Each of my vNIC templates only had a single VLAN on it, hence there was a one to one relationship between the vnic template, the vnics in the service profile template and the service profile vnics.

I decided to experiment a little bit with the vlans in the vNIC template and got the traffic to pass when I assigned the single VLAN in the vNIC to be the "native" vlan. This doesn't make a lot of sense to me until the vNIC itself is actually acting as a virtual 'switch', which I assume it is because you can have multiple vlans on each vNIC. After setting each of the single vlans in my vnic templates to be 'native', I can pass all of my vlan traffic now.

So my question then is .. do you always have to define a native vlan in each of the vnics unless your blades are tagging traffic (ie. vlan aware).

To me, the term 'native' vlan is a little bit misleading - I would expect it to be called 'default' vlan because the traffic on that vlan is not being tagged at or by the host.

Anyway, thanks for the help everyone. If someone can just quickly answer my question about the native vlan question that would be great. I expect that if you are using a hypervisor that you wouldn't need to define a native vlan because you're creating vlan aware virtual nics there.

Correct Answer
roberbur Thu, 05/24/2012 - 11:28

Just to clarify,

Native VLAN and Default VLAN are two different things.  Native simply refers to the VLAN on a trunked 802.1q link that will not be appended a VLAN tag.  Default VLAN is the "automatically assigned" VLAN to all vNIC interfaces.  By default the "default vlan" is vlan 1, but this can be changed.

When creating a Service Profile (or Template for that matter) you can use the Simple, or Expert wizard.  The Simple wizard assumes each vNIC will operate as an "access port" by only being assigned to one vlan.  In expert mode you can select either the port to operate as an Access Port (single VLAN) or Trunk (multiple VLANs).  It's only with the Trunk vNIC configuration where you can select the "Native" VLAN radio button.

So let's say you have a Windows host that needs to be assigned to VLAN 10.

You could create a simple vNIC that acccess VLAN 10 only, or you could create a Trunk Interface, but set VLAN 10 as the native.  Either method will return the same result.

For OSes like ESX & Hyper-V you will indeed want Trunking and therefore allowing multiple VLANs.

Now to top it all off, though we present what looks to be "Access" and "Trunk" ports, there's actually no access ports.  What UCS does when you select a vNIC to be an Access port is just make a Trunk interface with the sole VLAN as the native. 

You can see this from NXOS if you look at "show int trunk". 

Regards,

Robert

sdunkin Thu, 05/24/2012 - 12:08

Thanks Robert - that helps to clear things up. When I first set the FIs and loaded a server it was working fine and I see now that it was because I initially selected the simple wizard - even though I assigned a specific vlan (lets say vlan10, as in your example) it was treating the vnic (vethernet interface) like an 'access' port as you say - I just didn't see at the time that it was assigning the native vlan to the sole vlan on the vnic.

The next time I setup the server I was using the expert mode and/or creating the vnics manually and not assigning the native vlan. I was afraid the 'native' term was pervasive throughout UCS and it would affect the whole FI, however, as per your explanation (and what I saw from my testing) it only applies to that particular vNIC, which in essence is a like a switch in and of itself.

Phew, I thought the vlans would be the easy part in this whole setup - the configuration guides don't really explain this in good detail I suppose because a lot of installs will use the simple wizard. The uplinks, VPCs, server ports and template and profile creation turned out to be the easy part!

Thanks again!!

roberbur Thu, 05/24/2012 - 13:17

Happy to Help.

I know our documentaiton can be daunting & limited in many aspects but that's what we're here for.  We can answer your questions with a proper explanation

Let us know if you have any other issues or questions.

Cheers,

Robert

Actions

Login or Register to take actions

This Discussion

Posted May 23, 2012 at 11:31 AM
Stats:
Replies:10 Avg. Rating:5
Views:1092 Votes:0
Shares:0
Categories: General UCS Hardware
+

Related Content

Discussions Leaderboard