VLAN Manager is taking all CPU

Unanswered Question
Jun 9th, 2009

Any idea what this process is? Somehow the CPU util of the switch is now about 50%.


What is causing this?


36 297064492 5020384 59172 22.29% 24.76% 24.79% 0 VLAN Manager

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Alvaro Garcia Tue, 06/09/2009 - 12:02

VLAN manager is the IOS platform shim to the VTP protocol. VLAN manager assumes the platform runs IOS and implements the port manager and IFS. VLAN manager implements the vlan database configuration mode and commands, vtp configuration commands and all the related show commands.


It would be good to check how many VLANs and STP instances your switch has. Get a show vlan, show run, show interfaces trunk.


However, my experience says that a CPU of 50% on a 4500 switch is still under a normal range.


Have you recently implement any configuration/topology change?


Regards,


Al Garcia

ronshuster Wed, 06/10/2009 - 06:18

HI Garcia,


There was a change recently, we had a contractor come in and he did something and now our etherchannels to the Blades have a "B" next to them, here it is:


1 Po1(SU) - Gi5/1(P) Gi5/2(P)

10 Po10(SD) LACP

10 Po10B(SU) LACP Gi7/10(P) Gi7/11(P)

11 Po11(SU) LACP Gi7/12(P) Gi7/13(P)

12 Po12(SD) LACP

12 Po12B(SU) LACP Gi7/14(P) Gi7/15(P)

13 Po13(SD) LACP

13 Po13B(SU) LACP Gi7/16(P) Gi7/17(P)

14 Po14(SD) LACP

14 Po14B(SU) LACP Gi7/18(P) Gi7/19(P)

15 Po15(SD) LACP

15 Po15B(SU) LACP Gi7/20(P) Gi7/21(P)

16 Po16(SD) LACP

16 Po16B(SU) LACP Gi7/22(P) Gi7/23(P)

17 Po17(SD) LACP

17 Po17B(SU) LACP Gi7/24(P) Gi7/25(P)

18 Po18(SU) LACP Gi7/26(P) Gi7/27(P)

19 Po19(SD) LACP

19 Po19A(SU) LACP Gi7/28(P) Gi7/29(P)

100 Po100(SU) LACP Gi6/1(P) Gi6/2(P) Gi6/3(P) Gi6/4(D)


What is the B next to the po?

Ever since this B showed up, accessing the servers on the blade is very slow and latency is very high (like 200ms whereas in the past it was about 2ms).


On the blades themselves there are 4 vlans defined :


Port Mode Encapsulation Status Native vlan

Gi0/16 on 802.1q trunking 1

Po1 on 802.1q trunking 1

Po2 on 802.1q trunking 1


Port Vlans allowed on trunk

Gi0/16 1

Po1 1-4094

Po2 1-4094


Port Vlans allowed and active in management domain

Gi0/16 1

Po1 1,60-61,98,100,222

Po2 1,60-61,98,100,222


Port Vlans in spanning tree forwarding state and not pruned

Gi0/16 1

Po1 1,98,100

Po2 60-61,98,100,222


sh vlan:


VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

1 default active Gi0/15

60 VLAN0060 active Gi0/6

61 VLAN0061 active Gi0/1, Gi0/2, Gi0/3, Gi0/4

Gi0/5, Gi0/7, Gi0/8, Gi0/9

Gi0/10, Gi0/11, Gi0/12, Gi0/14

98 VLAN0098 active

100 ComputerRoom-Servers active

222 VLAN0222 active Gi0/13


So why do we see the B on the po?

Why is the latency so high to the servers?

why is VLAN manager taking up all the CPU?


I tried to bounced one of the po and the B disappeared.


Any idea?


Alvaro Garcia Wed, 06/10/2009 - 12:38

Hi,


About the "B" next to the Po; that is called alpha-numerical lacp portchannel, and it is usually seen when LACP protocol detects some inconsistency between channel member ports.. The inconsistency could be in either side of the channel. You should check logs of BOTH devices, usually this is due to channel misconfiguration between the ports you are bundling, or with their remote peers.


See the following document:


http://www.cisco.com/en/US/tech/tk389/tk213/technologies_configuration_example09186a0080094470.shtml#po1a


This mis configuration may be affecting the CPU as well.


I would suggest to contact the consultant and make him check the configuration.


Or send a show run, show etherchannel summ and show interfaces for both switches and I would give them a quick look.

ronshuster Wed, 06/10/2009 - 15:40

The blades are connected to two cores configured with etherchannel (made of 2 physical ports to each core):


Core 1:


12 Po12(SD) LACP

12 Po12B(SU) LACP Gi7/14(P) Gi7/15(P)


interface Port-channel12

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

end


Core 2:


12 Po12(SD) LACP

12 Po12B(SU) LACP Gi7/14(P) Gi7/15(P)


interface Port-channel12

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

end


Blade switch:


1 Po1(SU) LACP Gi0/17(Pd) Gi0/18(P)

2 Po2(SU) LACP Gi0/19(Pd) Gi0/20(P)


interface Port-channel1

description Uplink to Core1: G6/7 - G7/7

switchport mode trunk

switchport nonegotiate

!

interface Port-channel2

description To Core2: G06/7,G7/7

switchport mode trunk

switchport nonegotiate


If you want more info, may I have your email address?



Alvaro Garcia Wed, 06/10/2009 - 15:48

Hi,


Email is [email protected]


I'm seeing some discrepancy between the blade config and the cores' config. Is the command "switchport trunk encapsulation dot1q" available under the Int Port-channel on the blade switch?


Please send:


From Core1:


show run int po12

show run int gi7/14

show run int gi7/15

show int status


From Core2:


show run int po12

show run int gi7/14

show run int gi7/15

show int status


From blade switch:


show run int po1

show run int po2

show run int gi0/17

show run int gi0/18

show run int gi0/19

show run int gi0/20

show int status


Regards

ronshuster Wed, 06/10/2009 - 16:00

core1:


interface Port-channel12

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

end


interface GigabitEthernet7/14

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

channel-group 12 mode active

end


Current configuration : 200 bytes

!

interface GigabitEthernet7/15

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

channel-group 12 mode active


Core2:


interface Port-channel12

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

end


interface GigabitEthernet7/14

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

channel-group 12 mode active

end


interface GigabitEthernet7/15

description BC2-SW1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

no ip address

channel-group 12 mode active

end


BLADE


interface Port-channel1

description Uplink to ESGBB1: G6/7 - G7/7

switchport mode trunk

switchport nonegotiate

!

interface Port-channel2

description To ESGBB2: G06/7,G7/7

switchport mode trunk

switchport nonegotiate


interface GigabitEthernet0/17

description Uplink to ESGBB1: G6/7 - G7/7

switchport mode trunk

switchport nonegotiate

channel-group 1 mode active

!

interface GigabitEthernet0/18

description Uplink to ESGBB1: G6/7 - G7/7

switchport mode trunk

switchport nonegotiate

channel-group 1 mode active

!

interface GigabitEthernet0/19

description Uplink to ESGBB2: G6/7 - G7/7

switchport mode trunk

switchport nonegotiate

channel-group 2 mode active

!

interface GigabitEthernet0/20

description Uplink to ESGBB2: G6/7 - G7/7

switchport mode trunk

switchport nonegotiate

channel-group 2 mode active

!


To answer your question, the encapsulation option is not available on the switch, ie:


BC2-SW1(config-if)#switchport trunk ?

allowed Set allowed VLAN characteristics when interface is in trunking mode

native Set trunking native characteristics when interface is in trunking

mode

pruning Set pruning VLAN characteristics when interface is in trunking m


One thing worth noting, and that is when i bounced the port channels on the core, the "b" disappeared, but I only did this on one of the legs, I need to do it on both and see if hte CPU goes down.


From what you see so far, do you see any problems with the trunk configuration?


Alvaro Garcia Wed, 06/10/2009 - 16:21

The trunk and channel config looks fine. The mismatch could also be related to physical interface characteristics, like speed/duplex/status, etc.


If you do a show int for each of the physical ports do you noticed any discrepancy?


Bouncing the port channels would force the LACP auto negotiation and may fix the problem temporally. I used to be a LAN switching engineer on Cisco TAC and I worked many times with etherchannels; and I know that this channel negotiation configuration process maybe kind of tricky because you always have to configure each interface in the same way following the correct process otherwise the channel may present some problems.


What I normally do with this kind of alphanumerical channel issues is to send the physical ports to default settings and reconfigure the etherchannels from scratch, shutting down the physical interfaces firsts.


That is my recommendation.


ronshuster Wed, 06/10/2009 - 16:34

I will be bouncing all the port channels in a few hours and let you know if the problem is fixed... thanks for the advise.

Actions

This Discussion