need help tracking down auxiliaryvlan issue on 6500 w/CatOS

Unanswered Question
Dec 15th, 2009

This touches VoIP stuff, but it's really a LAN issue as I see.  I'm using Polycom IP430 and IP600 series phones against Interactive Intelligence EIC 3.0 server.  These phones pull DHCP from a Win2008 DHCP server.  DHCP settings give out Option 066 to tell the phones where to go for their config files.  This is an http connection on port 8088 to the EIC 3.0 server.

I was previously using CAT3560 switches running IOS 12.2(25r)SEC.  The EIC server was on a trunked port with both the voice and native VLAN's teh same (port is trunked for QoS), and all the phones where on trunked ports with a voice vlan specified.  Voice VLAN was differnet than the native vlan on the port.

Fast forward a few months and I've moved locations, now my switch is a Cat6513 w/Sup2 running CatOS 8.5(4) software.  EIC server is again on a trunked port again with both native and auxiliaryvlan the same.  Phones are on trunked ports.

When I define an auxiliaryvlan on the port, the polycom phones will pull DHCP fine.  They get an address and I can ping the phone from a desktop on the native vlan, the switch itself, and the EIC server which is on the same vlan as the phone.   BUT, once the phone has an IP, it will not contact the boot server (EIC server) and pull it's config files.

If I remove the auxiliaryvlan setting, thus putting th ephoens on the same vlan as the native port, the phoens boot fine.

Since the issue did not occur on the 3560's with IOS, it seems to be something that the Cat6513 and it's version of CatOS are interfering with.  Problem for me, I don't have a clue how I go about figuring this out.  Any help would be greatly appreciate.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
sachinraja Tue, 12/15/2009 - 09:43

Hi pfc

Please find the open and solved caveats with respect to 8.x switch Cat OS.. this should tell you if you have any bugs identified with aux vlans on a 6500..

with regards to your issue, you say you get an IP address for the phone after defining aux vlan.. aux vlans obviously work on layer 2, and hence if you get an IP address, your aux vlan stage is crossed, meaning it is more to do with layer 3 then ! is your boot server on the same vlan as aux vlan ? What are the numbers given for aux, data & server vlans ? Let us know more on how your layer 3 is setup...

Just for troubleshooting, why dont you configure the aux vlan number directly on a switchport (set port vlan 20 , considering the aux vlan is 20), and see if it works... are there any other configs on the switchport apart from aux vlans ? security configs, qos configs etc ??


pfc-corporate Tue, 12/15/2009 - 10:08

All routing is done by the MSFC2 on the Sup2. Here's some specific's

VLAN #, interfaceIP/subnet

VLAN 53,

VLAN 210,

VLAN 250,

The DHCP server is on VLAN210

The Phone System/BOOT server is on VLAN53

Workstations are on VLAN250

The setup that works is if a switch port has NO auxiliaryvlan specified.  the native VLAN can be 250 or 53.  In either case, the phone pulls an IP for the defined native VLAN (agian, I've tested it with native VLAn of 250 or 53), contacts the boot server and pulls it's config files and registered with the VoIP server.

When things go wrong is if I add an auxiliaryvlan.  So the setup would ne native vlan 250, auxiiliaryvlan 53.  With the switch port defined this way, the phone pulls an IP of and the desktop connected through the phone pulls an IP of  So, CDP is working appropriately to get the phone on the VLAN it needs to be on, and to pull an IP on that VLAN.  The phone is reachable on the network, but a communication problem is now occuring between the phoen and the boot server.  Both devices are on the same VLAN at this point.

There are no security configs on the ports, but there is QoS defined by using "autoqos trust cos".

I used the same settings (VLAN Numbers, interface IP's, and QoS, native and voice vlan definitions) when I was previously using 3560's with IOS on them and the same exact phones, worked fine.  Likewise, I was using this same 6513 and same version of CatOS a few years ago, with the same vlan numbers, QoS, native/auxiliaryvlan settings, and it worked back then.  The only change is the software on the phone.  Unfortunately, I don't really have any way to invoke Polycom support to assist.

sachinraja Tue, 12/15/2009 - 10:13

Hi pfc

thanks for the info.. it looks a bit strange though.. Just to troubleshoot, can you see by connecting other models of ip phones to this port, instead of plugging a polycom ? Does your call manager show any log or errors when the polycom tries to download the config files ? am sure looking at the boot server errors, we can troubleshoot something here... also did u try to configure a port directly on vlan 53 and try accessing the boot server ?

now since you already get an ip address for the phone, i really donno if we need to look more on layer 2 technologies like cdp, l2 qos, aux vlan etc... can you span this port, and run a sniffer of it to see what exactly happens ?


pfc-corporate Tue, 12/15/2009 - 10:23

Not actually running a CallManager, but another product (Interactive Intellgience EIC).  The boot server is an embedded HTTP server with the EIC server.  There is logs I can look at though.  If I recall correctly, when I troubleshot this originally, the boot server wasn't even seeing a request for files from the phone.

Only phones in use that would even use the boot server are Polycom phones.  A Cisco phone can work with my system, but I have to configure those using a 3rd party TFTP/FTP server, so it's not a useful troubleshooting step.

I did set a port to native vlan of 53 with no auxiliaryvlan.  The phone contacts the boot server fine in this configuration, which makes it seem more like L2 problem.

I can do a new port span dn sniff with wireshark.  I did that this morning but didn't save the info. There was very little information from my phones IP to other IP's.  As I recall, it requested an IP, and then it made an NTP request to get updated time, and then there was absolutely nothing until the phone timedout on it's boot server request and defaulted to the last known config saved in the phone.

sachinraja Tue, 12/15/2009 - 10:28

"when I troubleshot this originally, the boot server wasn't even seeing a request for files from the phone" - are you sure of the dhcp settings on pointing to the correct boot server on the auxillary vlan ?

Is it possible for you to hardcode the bootserver settings for the polycom on the aux vlan , instead of having DHCP give these parameters ?


pfc-corporate Wed, 12/16/2009 - 07:36

Hard coding the boot server doesn't make any difference.

I guess I'm going to upgrade to 8.6(4) this weekend.  That's the last version of CatOS put out for Sup2's.  Maybe there's a bug causing the problem that is only now present due to new firmware on the phones.

sachinraja Wed, 12/16/2009 - 07:42

Ya.. thats a good idea.. but have a look at the release notes of 8.6 to make sure there are any mocre bugs.. as of your ios version, we didnt see much on the release notes right ? 

Upgrade your switch and let us know the results.. all the best..


pfc-corporate Thu, 12/17/2009 - 07:39

something that may or may not be useful in me figuring this out.

I did packet captures with no auxiliaryvlan set and with.  I'm trying to compare entry-by-entry what is differnet.  I noticed that there is unanswered ARP brodcast traffic when the phone is on an auxiliaryvlan.

The request is from the phone and it says "who has  tell". = boot server/phone system = IP of phone

This request repeats over and over again.  I also a request "who has tell", which does get answered by the switch.  The answer from the switch is the mac of the switch, as the is the vlan interface IP for this vlan.

when no auxiliaryvlan is set, there is still some ARP broadcast traffic, bu tit's different.  I see:

"who has  tell" = vlan interface IP on switch = IP of phone

this request is answered by the switch, and the answer is the mac of switch once again.

Does this indicate something useful to troubleshoot with?

sachinraja Thu, 12/17/2009 - 08:40


the sniffer traces looks to show what it is supposed to.. with aux vlan set, it basically points to the same VLAN as the EIC boot server, and hence the ARP request directly gets the IP address of the server ( since the phone and the server are in the same LAN...

with aux vlan not set, the phone takes the data vlan , and the ARP tries going to the next hop (which is the VLAN Layer 3 IP address -

Did you still find the errors that you see on the EIC server when the phone is connected on auxillary vlan ? Is there any requirement for the phone to be on the aux vlan similar to where the boot server is ? Can you try setting the auxillary vlan to some other vlan (say create a new vlan 500) and have it routed across to the boot server? Just to check if it is a problem with aux vlan commands, or the real boot server ?

1) create vlan 500 (say

2) Configure ip helper or commands to enable DHCP for the ip phone..

3) configure layer 3 forwarding on the switch to communicate with the boot server...

4) check connectivity

Let us know..


pfc-corporate Thu, 12/17/2009 - 11:22

The problem occurs ONLY when I define auxiliaryvlan on the switch.  If auxiliaryvlan is NONE, I can put the port on ANY VLAN (including 53, where the server is), and everything is fine.

As you suggested, I created an entirely nerw VLAN on a different network ( and assigned the auxiliaryvlan to this VLAN.  Same results as when I used VLAN 53.

Just as a quick comment  When you refer to layer 3 forwarding..  This is all handled by the MSFC2 on the Sup2, which has ip routing enabled.  I am not using anything like EIGRP or OSPF however (I did have eigrp active earlier when trying to troubleshoot this problem).  All the VLAN's are defined on the MSFC2 and the VLAN's on the Sup2 match.

sachinraja Thu, 12/17/2009 - 12:17

EIGRP or OSPF should not be an issue here.. phones work good on layer 3, if the aux vlan isnt set.. and on layer 3 the msfc will forward the packets directly based on the destination ip address..

I just have one query.. what is the aux vlan settings you have configured on the port ? Can you copy paste the "set port auxilary vlan" configuration ? you can optionally enable cdpverify after defining the aux vlan..

set port auxiliaryvlan mod[/port] {vlan | untagged | dot1p | none} [cdpverify {enable | disable}]

try forcefully enabling the cdpverify, since the phones work on sending the info through CDP..

let us know


pfc-corporate Thu, 12/17/2009 - 12:39

set port auxiliaryvlan 13/3 53

with "cdpverify enable", the switch never sets the auxiliaryvlan to active with the phone plugged in, so I have to set cdpverify disable.   This is the only way the phone ever goes onto the auxiliaryvlan.  Wireshark logs show the DHCP process goes properly as well, and the phone ACK's all of the DHCP parameters it accepts.  From that point forward, there is just no communication to the EIC server.  When the switch port doesn't have an auxiliaryvlan set (thus the phone is on the native vlan), the next thing I see after the DHCP Ack is TCP traffic to the HTTP boot server.

Unrelated, but just so information: The switch sees the device as "cisco+ieee".  The switch doesn't actually think it's a phone.  I know this because I tried using "set port qos 13/2 trust-device ciscoipphone" and the switch never enables QoS on the port for the phone so I use "set port qos 13/2 autoqos trust cos" instead

sachinraja Thu, 12/17/2009 - 13:42

normally the switches see the cdp packet from the phone, before they allow traffic on the voice vlan.. cdp passes on voice vlan info with tagged frames to the switch port.. the switch can block traffic if cdpverify is disabled ... did you try enabling it now and see if it works ? can you also post other configs on the switch port related to cdp, portsecurity (if any) etc ?

with 3560, with voice vlan commands, you can detect cisco phones directly with commands, but not sure if you have equivalent commandsets in 8.4 for this.. the only way the switch recognises that it is a cisco phone is through CDP..


pfc-corporate Thu, 12/17/2009 - 13:56

Using my new vlan 3, I did "set port auxiliaryvlan 13/2 3 cdpverify enable" and rebooted my phone.

The phone "about" page shows VLAN 3, but it shows "resolving" for address and boot server.  The phone appears to ne unable to pull an IP address from th DHCP server (confirmed in DHCP server)

"show port 13/2" shows AuxiliaryVlan = 3 and AuxVlan-Status = none

if I have the port set as "set port auxiliaryvlan 13/2 3 cdpverify disable" the phone shows VLAN 3 in the about page and se pulls an IP from DHCP server fine, but it never talks to the boot server. In this mode, the switch displays for "show port 13/2" that AuxiliaryVlan = 3 and AuxVlan-Status = active

sachinraja Thu, 12/17/2009 - 14:09

can you post other configs on the switchport apart from the aux vlan config ? is there any other specific configuration ? are you forwarding dhcp requests with "Ip helper" from the new vlan 3? when have you planned for IOS upgrade ?


pfc-corporate Thu, 12/17/2009 - 14:16

ip helper is active on VLAN 3.  as I can confirm on the phone, it's being told what VLAN to go on by CDP, but cdpverify seems to not acknowledge that the device is actually a PHONE.  As I read the docs, cdpverify only makes the auxiliaryvlan active if it knows it's a phone.  I think this is the case because of the qos thing I mentioned earlier, where the switch doesn't think the device is a phone and auto-enable qos on the port.  Here's some stuff. not sure what else you would want to see.  There really isn't much outside of default configuration.

I was hoping that with the packet sniffing info, there would be an obvious fix, but that doesn't seem to be the case (thus far).  I don't know what else to look for in the sniffer logs to find more useful info.  Software update may not be able to occur for a few weeks (originally thought this weekend).

sh port cdp

CDP               : enabled
Message Interval  : 60
Hold Time         : 180
Version           : V2
Device Id Format  : Other

Port      CDP Status
--------  ----------
13/2      enabled     

sh port security

* = Configured MAC Address 

Port  Security Violation Shutdown-Time Age-Time Max-Addr Trap     IfIndex
----- -------- --------- ------------- -------- -------- -------- -------
13/2  disabled  shutdown             0        0        1  enabled     383

Port  Flooding on Address Limit Last-Src-Addr     Vlan
----- ------------------------- ----------------- ----
13/2                    Enabled                 -    -

Port  Num-Addr Secure-Src-Addr     Vlan Age-Left Shutdown/Time-Left
----- -------- -----------------   ---- -------- ------------------
13/2         0                
sachinraja Thu, 12/17/2009 - 14:34

I saw a specific bug related to cdpverify enable command on the 8.x IOS:

With cdpverify enabled, the auxiliary VLAN might not come up  on a Cisco IP conference station 7936 IP phone and the phone might not boot.

Workaround: Disable cdpverify on that port using the set  port auxiliaryvlan mod/port cdpverify disable command. This problem is  resolved in software release 8.3(4). (CSCef23681)

Can you check the status of this bug through "bug navigator" ? It says it is resolved in software 8.3(4) but doesnt seem to ! since you have 8.4(5) ! but on someother documents, i see disabling cdpverify will not detect the ip phones automatically !!! one way was to see the errors on the boot server EIC, but you said there are no messages to the boot server ! do you know if your polycom has the updated software ? sometimes software issues can cause such bizare things !


pfc-corporate Mon, 05/10/2010 - 13:02

Bumping this thread up again, still having this issue.  I updated the Sup2's to CatOS 8.6(4), which is latest release available, but no help.


if I define auxiliaryvlan, the IP phone pulls an IP from DHCP server on the auxiliaryvlan.  IP Phone can be pinged on this IP, but the IP phone does not communicate with the BOOTP server to pull config files.

IP phone is polycom IP430 running 3.1.2 software.


This Discussion