(TL;DR at the bottom)
We're in the process of rolling out VoIP across our campus. The network is divided up such that we have a router per building (about 20 buildings on site), off each of which anything from 10 to 40 edge switches hang depending on building size. Each building router is uplinked to each of our two core devices. We've converged our VoIP and data infrastructure, so the VoIP network in each building is merely a different VLAN on the same equipment as our user PCs (all switches can provide PoE to every port). Each building router has, as much as possible, an identical configuration. The only differences from building to building are the actual VLAN IDs and the subnet defined for each VLAN of a specific type (Across all buildings all VLANs of a specific type will share a wider range, so for example all VoIP VLANS might be within 10.10.0.0/16, even though an individual building might only have 10.10.44.0/22. Any firewall config references 10.10.0.0/16 and thus we can have the same firewall config on each router).
DHCP for all networks, including VoIP, is handled by a central isc-dhcpd server which understands and hands out all the extended attributes required by IP Phones, Wireless Access Points, PXE booting machines etc based on the source IP (i.e. the gateway address) the request is being forwarded from.
Now on to the actual issue:
We've completed about 15 buildings of POTS -> VoIP migration so far, no problems with any of them except the latest one. In the latest building, we've had numerous complaints of phones displaying the message "IP Address Released", and requiring a reboot to get them back online. We've yet to manage to catch a packet dump of the moment this occurs, but mirroring the port the phone is on *after* the occurrance simply shows no traffic going from the phone out to the network. It's not attempting DHCP discover/request or anything, although the switch shows the port as up, on the correct VLAN, and sees the phone's MAC as being on that port. There is broadcast traffic for the relevant VLAN being sent out to the phone.
Everything I can find online about the "IP Address Released" message has to do with how to force this state to occur through the settings menu of the phone itself, which has to do with purposefully releasing an IP on one VoIP VLAN for use by another device before moving the device which you cleared the IP on to a different VoIP VLAN where it will obtain a new IP. We've never had to do this, as our DHCP server sets the age of a lease to 48 hours and frees the IP if the phone hasn't re-requested it within that timeframe (All devices should re-request before the max lease time in the original DHCP response). I assume this is to do with if your CUCM is also your VoIP DHCP server.
But we're getting phones which randomly show this message with no user input. They're sitting at their desk, working away fine on their PC, and suddenly notice that the phone is displaying this error message. There has been no noticeable interruption of networking for the PC (generally plugged into a datapoint going to the next port up on the same switch). If they unplug and reconnect the phone's network cable and cause the phone to reboot it'll come online just fine and obtain an IP. I just have no idea what could be causing this message to occur all over this one building nut nowhere else. Do the phones enter this state if they re-request their current IP from the DHCP server and receive no response? Our DHCP server is run by a different team so it's awkward but not impossible to get logs from it to see if a request came in.
TL;DR Phones on a specific VoIP subnet keep displaying "IP Address Released" message and needing a reboot to get back online. Users are not choosing the option in the settings menu that manually releases the IP, so what could be causing this behaviour? All other phones on site, on other VoIP subnets, are fine.
