IOS 15.0(2)SE5 DHCP Snooping Problem

[pwtn]

I have just upgraded a single production switch from IOS 12.2(50)SE1 to 15.0(2)SE5 to test out new ipv6 security features that we will soon require for our deployment. upon booting into the newer IOS the DHCP snooping feature stopped working, this caused ARP inspection to start dropping traffic so we had to disable it. after going through the normal troublehsooting procedures (check config, reboot, re-apply config, check clients, renew IP address etc) it still is not working.

has anyone else experience this problem or anything similar?

I would be interested to hear from people on recent experiences when upgrading software as we have been having a bad time recently with cisco software across a range of products.

[AURELIEN MERE · Accepted solution]

Hi

I just encountered the same issue after upgrading various 2960 switches from 15.0(2)SE4 to 15.0(2)SE5.

Absolutely no trace in debug, like if incoming dhcp packets were just dropped.

The only way I found to get everything back to normal was to downgrade all of them to 15.0(2)SE4 and it's working again.

Regards

[pwtn]

Yes this sounds like the exact same issue we had, even with all debug enabled for the feature all you see is a periodic message about checking expired entries.. but no entries populating the snooping database.

we were advised by our hardware vendor to downgrade to 15.0(2)SE4, this fixed the problem for us too. I've requested our hardware vendor raise this with TAC as it has the potential to cause massive amounts of downtime in production networks for people who also run dynamic ARP inspection like we do. Had we rolled this software out to our 300+ access switches without testing we would have taken thousands of clients offline.

I would advise anyone experiencing this issue to also report it to TAC.

[Colby Beam]

Aurelien,

Prior to SE5 code there was https://tools.cisco.com/bugsearch/bug/CSCui65252, but this only pretained to ARP reply being dropped by the 3750 and ONLY if the ARP request was coming in on a port-channel. DAI was required to be on, and even though the ports were trusted, it would drop

If you can elaborate more on the DHCP issue you were seeing I can test in the lab. Would need your running-config, and is this only IPv6 DHCP request? However I would get a TAC case as well opened so we can investigate further and track it.

[AURELIEN MERE]

Hi

I encountered the issue with IPv4 dhcp snooping on 3560-8PC, 2960-48PST, 2960-24PC and 2960-24TT. DAI is not engaged on the switches, and the trusted port can either be a port-channel or a simple gigabit port. I'll proceed with an upgrade and give you the show tech in 15.0(2)SE4 and SE5.

Aurélien

After upgrade, you can see in the console output that I shut all ports, all phones and AP are supposed to renew their lease, but table stays empty and no DHCP snooping log.

Ce message a été modifié par: AURELIEN MERE

[Colby Beam]

Aurelien

I see from the SE5 that the DHCP binding table was not populating. I am looking to test this in the lab. However, I would recommend at this point opening a TAC case so can track this.

[Colby Beam]

Aurelien

I just tested this on a 2960-S running SE5 with no issues.

2960-1#debug ip dhcp snooping packet

DHCP Snooping Packet debugging is on

2960-1#

Mar 30 01:30:23.963: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Po1 for pak.  Was Vl1

Mar 30 01:30:23.963: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Vl1 for pak.  Was Po1

Mar 30 01:30:23.963: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Po1 for pak.  Was Vl1

Mar 30 01:30:23.963: DHCP_SNOOPING: received new DHCP packet from input interface (Port-channel1)

2960-1#

Mar 30 01:30:23.968: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Po1, MAC da: ffff.ffff.ffff, MAC sa: 3037.a696.3640, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 3037.a696.3640

Mar 30 01:30:23.968: DHCP_SNOOPING_SW: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (1)

Mar 30 01:30:23.968: DHCP_SNOOPING_SW: bridge packet send pac

2960-1#ket to cpu port: Vlan1.

Mar 30 01:30:25.976: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Gi0/24 for pak.  Was Vl1

Mar 30 01:30:25.976: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Vl1 for pak.  Was Gi0/24

Mar 30 01:30:25.976: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Gi0/24 for pak.  Was Vl1

Mar 30 01:30:25.976: DHCP_SNOOPING: received new DHCP packet from input interface (GigabitEthernet0/24)

Mar 30 01:30:25.976: DHCP_SNOOPING: process new DHCP packet, message type: DHCPOFFER, inpu

2960-1#t interface: Gi0/24, MAC da: ffff.ffff.ffff, MAC sa: 001c.0e86.6f4a, IP da: 255.255.255.255, IP sa: 172.16.156.33, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 172.16.156.47, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 3037.a696.3640

Mar 30 01:30:25.981: DHCP_SNOOPING: direct forward dhcp replyto output port: Port-channel1.

Mar 30 01:30:25.987: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Po1 for pak.  Was Vl1

Mar 30 01:30:25.987: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Vl1 for pak.  W

2960-1#as Po1

Mar 30 01:30:25.987: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Po1 for pak.  Was Vl1

Mar 30 01:30:25.987: DHCP_SNOOPING: received new DHCP packet from input interface (Port-channel1)

Mar 30 01:30:25.987: DHCP_SNOOPING: process new DHCP packet, message type: DHCPREQUEST, input interface: Po1, MAC da: ffff.ffff.ffff, MAC sa: 3037.a696.3640, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 3037.a696.3

2960-1#640

Mar 30 01:30:25.987: DHCP_SNOOPING_SW: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (1)

Mar 30 01:30:25.987: DHCP_SNOOPING_SW: bridge packet send packet to cpu port: Vlan1.

Mar 30 01:30:25.987: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Gi0/24 for pak.  Was Vl1

Mar 30 01:30:25.987: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Vl1 for pak.  Was Gi0/24

Mar 30 01:30:25.987: DHCPSNOOP(hlfm_set_if_input): Setting if_input to Gi0/24 for pak.  Was Vl

2960-1#1

Mar 30 01:30:25.987: DHCP_SNOOPING: received new DHCP packet from input interface (GigabitEthernet0/24)

Mar 30 01:30:25.992: DHCP_SNOOPING: process new DHCP packet, message type: DHCPACK, input interface: Gi0/24, MAC da: ffff.ffff.ffff, MAC sa: 001c.0e86.6f4a, IP da: 255.255.255.255, IP sa: 172.16.156.33, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 172.16.156.47, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 3037.a696.3640

Mar 30 01:30:25.992: DHCP_SNOOPING: direct forward dhcp replyto output port:

2960-1#Port-channel1.

2960-1#sh ip dhc

2960-1#sh ip dhcp no

2960-1#sh ip dhcp sno

2960-1#sh ip dhcp snooping b

2960-1#sh ip dhcp snooping binding

MacAddress          IpAddress        Lease(sec)  Type           VLAN  Interface

------------------  ---------------  ----------  -------------  ----  --------------------

30:37:A6:96:36:40   172.16.156.47    86387       dhcp-snooping   1     Port-channel1

Total number of bindings: 1

2960-1#sh ver | in IOS  

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 15.0(2)SE5, RELEASE SOFTWARE (fc1)

2960-1#

[Colby Beam]

I am looking to test a normal 2960, to see if there is any difference. This may take a day or two.

[AURELIEN MERE]

I just tested with a 2960C-8PC and I got the same problem. The DHCP binding table stays empty so IPSG blocks everything.

15.2(1)E is not impacted btw.

I will test a 2960-S in lab tomorrow with existing config and from zero

[borutlape]

We also encountered the same problem with the 15.0(2)SE5, on WS-C2960-24PC-L and WS-C3560G-48PS.

Luckily we spotted the problem very soon after the upgrade.

The other switches with 15.0(2)SE4 have no issues so far.

[AURELIEN MERE]

A customer with thousand switch seems not to be big enough for Cisco to be able to have a TAC case opened.

I can't understand we can't even report a major bug like this, furthermore introduced in a maintenance release...

[Alexander Pickar]

Hi all,

I could confirm the behaviour in my lab :

(and unfortunatelly, we run into that issue at customer's site as well)

Result summary:

On 3750G running 15.0(2)SE5, DHCP snooping didn't work.

On 3750G running 15.0(2)SE4 or 12.2(55)SE8, DHCP snooping was working well.

On 3750G running 15.0(2)SE5, DHCP snooping was working well.

I had this test setup:

3750-X with 15.0(2)SE5 - Gi 1/0/14 ----- Gi 3/0/1 - 3750-G with different software - Gi 3/0/2 ----- Fa 0/0 - Router

Regards,

Alexander

[AURELIEN MERE]

Hi

Has someone found any kind of workaround for this problem, except staying in 15.0(2)SE4 for 2960/3560/3750G?

Thanks for your help

[Matthew Smee]

FYI,

bug CSCug52922 : fix is available in 15.0(2)SE6. CCO date for 15.0(2)SE6 is 04-23-2014.

[Alexander Balakin]

Having same problem on 3560G and 3560V2.

I hope that this will be really fixed in 15.0(2)SE6