DHCP Snooping issue after reloading by power on

Oscar Falcon Ortiz · ‎07-08-2013

We are experiencing issues with some devices, PCs and IP phones, after a massive reload of some stacks of switches by an elctrical failure. The root of the problems is DHCP Snooping.

Some of the PCs and phones of some of the stacks, are not getting IP. For example, some IP Phones stuck in configuring IP, not showing any entries in the dhcp snooping binding database. Seems like DHCP snooping is filtering server responses. I’ve tested with a dhcp snooping debug proving that only discovers packets are sent by clients and no responses arrive.

It is a very strange issue, because it only affects to a few devices. The workaround has been to disable ip dhcp snooping trust in the uplink to the DHCP server and re-enable it again. After doing this, DHCP server responses arrive and devices get IP addresses. It seems like a bug. Has anybody experienced

this as well?

Switches are C2960S with IOS version 122-55.SE2.

Thanks.

Peter Paluch · ‎07-08-2013

Hi Oscar,

There is a possibility that these devices do not start the DHCP exchange from the very start using the DHCPDISCOVER message, rather, they remember their last IP address and the last DHCP server they communicated with, and tried to contact it using DHCPREQUEST directly. Your switches running DHCP Snooping may have considered this to be an illegal attempt to talk to a DHCP server and dropped it. This behavior is strongly client-dependent which could explain why this happened only to a handful of clients.

Usually, this happens to Windows clients, in which case it helps to issue the ipconfig /release and ipconfig /renew commands in succession, as this will cause the DHCP client to start from scratch.

Best regards,

Peter

Oscar Falcon Ortiz · ‎07-08-2013

Hi Peter,

I've checked debugging the mac of the client, that it sends DHCPDISCOVERs continuously, without receiving DHCPOFFERs, so it's not a problem of the client.

We used Ipconfig release/renew in our first attempt to troubleshoot without success. Debugging make us thinking the problem is in the switch.

We also have seen the dropped packets count rise in the dhcp snooping statistics, more exactly by "misdirected packet" reason. This matches with the fact of having to re-enable the ip dhcp snooping trust command in the uplink. It seems like the port does not apply the trust state for dhcp packets when it's reloaded, but only affecting some clients...very strange.

Best regards.

Peter Paluch · ‎07-09-2013

Hello Oscar,

Hmm, I see. Strange. This could indeed be a bug but I do not want to jump to conclusions just yet. Do you happen to still have the debugs that say something about misdirected packets?

Best regards,

Peter

Oscar Falcon Ortiz · ‎07-09-2013

Hello Peter,

This is the output of the debugs before re-enabling the trust configuration of the uplink:

Jul 8 11:22:48.120 METDST: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Gi2/0/19, MAC da: ffff.ffff.ffff, MAC sa: 108c.cfe0.ae61, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 108c.cfe0.ae61

Jul 8 11:22:48.120 METDST: DHCP_SNOOPING_SW: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (407)

Jul 8 11:22:52.120 METDST: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Gi2/0/19, MAC da: ffff.ffff.ffff, MAC sa: 108c.cfe0.ae61, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 108c.cfe0.ae61

Jul 8 11:22:52.120 METDST: DHCP_SNOOPING_SW: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (407)trus

Jul 8 11:22:48.120 METDST: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Gi2/0/19, MAC da: ffff.ffff.ffff, MAC sa: 108c.cfe0.ae61, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 108c.cfe0.ae61

Jul 8 11:22:48.120 METDST: DHCP_SNOOPING_SW: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (407)

Jul 8 11:22:52.120 METDST: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Gi2/0/19, MAC da: ffff.ffff.ffff, MAC sa: 108c.cfe0.ae61, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 108c.cfe0.ae61

Jul 8 11:22:52.120 METDST: DHCP_SNOOPING_SW: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (407)trust

Any idea?

Best regards.

Peter Paluch · ‎07-13-2013

Hi Oscar,

I apologize for responding so lately. I do not know if this issue is still relevant...

Thank you for posting the debug output. Sadly, I am afraid this is not going to help me, as it is captured - according to what you wrote - "before re-enabling the trust configuration of the uplink" meaning that the erroneous dropping of DHCP messages on trusted ports can not be seen here. What would probably be helpful would be the debug output when the trusted ports were actually configured and yet they were dropping DHCP messages.

Anyway, has the problem reappeared?

Best regards,

Peter

Oscar Falcon Ortiz · ‎07-16-2013

Hi Peter,

I'm afraid my last post wasn't so clear.

The debug output was taken before I put the uplink in the untrusted stated and thereafter in the trust state again. So the debug is showing how the swtich is filtering the DHCP server responses when it shouldn't (uplink trusted), and as I've mentioned previously this only happened to some devices.

The problem dissapears after doing in the uplink:

no ip dhcp snooping trust

ip dhcp snooping trust

By now, It hasn't reappeared.

Best regards.

Peter Paluch · ‎07-16-2013

Hi Oscar,

Hmmm... There is a bug that seems to be related:

https://tools.cisco.com/bugsearch/bug/CSCtr44296

The ip dhcp snooping command causes specific programming of the switching hardware, so if there is an issue with this process, reprogramming the hardware may help. The bug report states that the bug should be solved in 12.2(55)SE5. Would you mind trying 12.2(55)SE7 that has been considered to be stable?

Best regards,

Peter

Oscar Falcon Ortiz · ‎07-16-2013

Yes, it seems to be related although it is referred to 3750 stacks.

We'll take into account the upgrade.

Thanks Peter.

Best regards.