DHCP server works simultaneously with DHCP relay

Florin Barhala · ‎10-12-2015

Hi guys,

I have the 2x WS-C3560X-48P running 12.2(55)SE5, RELEASE SOFTWARE (fc1)

Both are configured with VRRP:

3560X-cr01#show standby brief
P indicates configured to preempt.
|
Interface Grp Pri P State Active Standby Virtual IP
Vl10 10 105 P Active local 172.17.224.3 172.17.224.1
Vl60 60 105 P Active local 172.17.226.3 172.17.226.1

Now I did all tests and troubleshoot on the current active: cr01 which holds the following DHCP configuration (standby switch currently has no DHPC config):

show run i | ip dhcp

ip dhcp excluded-address 172.17.224.0 172.17.224.10
ip dhcp excluded-address 172.17.224.240 172.17.224.250
ip dhcp excluded-address 172.17.226.0 172.17.226.10
ip dhcp pool CORPORATE224_DATA
ip dhcp pool CORPORATE226_DATA

Trouble started when I tried migrating DHCP service from the switch to one of our HQ AD servers.

In order to avoid full site outage I did the following:

1. One day in advance I switched default lease time:

ip dhcp pool CORPORATE226_DATA

lease 0 2

2. Next day I have configured AD server with the pool options required, then went on switch

no ip dhcp pool CORPORATE226_DATA

interface vlan 60

ip helper-address 172.17.120.71

3. Went on a workstation, shut/no shut the port on switch and then reviewed the ipconfig /all output

Trick was that it was showing the switch as DHCP server:

DHCP Server . . . . . . . . . . . : 172.17.226.2

Still on my AD server I could see some leases including my test workstation. About 2h later entire segment (vlan 60) had no connectivity as no device connected there was receiving any DHCP IP lease.

As an workaround I had to copy/paste and enable old DHCP pool on the switch as it seemed Microsoft AD setup (Windows 2008) was not working.

After adding everyone back online I redid steps 2&3 and ran a firewall sniffer capture.

Put that capture on Wireshark and here's what I could see (on the attachments).

Basically capture shows DHCP Discover and DHCP Offer but NO DHCP Request or DHCP Ack. Since these 4 messages are using same sniffing ports I assume the latter two messages never reach the firewall. By now you figured it out that DHCP server is reachable through an IPSEC tunnel. Still that server holds IP pools for other remote sites, which clears my concerns about DHCP servers' configuration. Another step I did was to use another DHCP server on another location (so another IPSEC tunnel):

ip helper-address 172.17.140.71

and to my surprise FW sniffer file looked the same: it traced just DHCP Discover and Offer messages for most of the time and some scarce DHCP Ack.

Wireshark file was captured using the following filter: tcpdump -nn host 172.17.120.71 port 67 or port 68 and was ran on each of the DHCP server location firewall. It didn't cross my mind at that time to run it also on the location where this issue sits. Also I checked on all FWs for any dropped packet and couldn't see any.

One more step I did I attempted to catch all DHCP traffic and PBR it through our MPLS line between site and HQ. Now I could not do a sniffer session here as I would have had to mirror one port there, but the result was the same which brings me to the point of suspecting as culprits either DHCP server (although I tried two different machines) or the switch itself.

Now please share your thoughts or things to check for this. My only idea sits on the thread's title: this happens because I did NOT completely disable DHCP service on the switch. You can also consul the live debug dhcp I ran on the switch.

Thanks in advance,

Florin.

L.E. here's SDM config if it matters

3560X-cr01#show sdm prefer
The current template is "desktop routing" template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

number of unicast mac addresses: 3K
number of IPv4 IGMP groups + multicast routes: 1K
number of IPv4 unicast routes: 11K
number of directly-connected IPv4 hosts: 3K
number of indirect IPv4 routes: 8K
number of IPv4 policy based routing aces: 0.5K
number of IPv4/MAC qos aces: 0.5K
number of IPv4/MAC security aces: 1K

Richard Bradfield · ‎10-12-2015

Hi Florin,

It looks like you have done everything right. A couple of questions

You say other remote sites are working ok using the remote DHCP servers, so are all sites pretty much configured the same ( same switches etc)?

Did you put the helper address on both VRRP switches?

How about clearing all DHCP config on the switches, then doing a reload of the switches. this will ensure no DHCP on the switches.

HTH

Richard

paul driver · ‎10-12-2015

Hello

Can you ping the AD DHCP server from the access layer?
Is the DHCP server authorised in AD and the scope made active
Do you have DHCP snooping enabled on the switches? - sh ip dhcp snooping
Are you using secondary addressing on the SVI's?

res

Paul

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Florin Barhala · ‎10-13-2015

Paul,

yes I can ping DHCP server from my switch:

3560X-cr01#ping 172.17.120.71 so vlan 60

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.17.120.71, timeout is 2 seconds:
Packet sent with a source address of 172.17.226.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 42/45/50 ms

There is no DHCP snooping configuration in place

3560X-cr01#show ip dhcp snooping
Switch DHCP snooping is disabled
DHCP snooping is configured on following VLANs:
none
DHCP snooping is operational on following VLANs:
none
DHCP snooping is configured on the following L3 Interfaces:

Insertion of option 82 is enabled
circuit-id default format: vlan-mod-port
remote-id: 2834.a2a5.4980 (MAC)
Option 82 on untrusted port is not allowed
Verification of hwaddr field is enabled
Verification of giaddr field is enabled
DHCP snooping trust/rate is configured on the following Interfaces:

Interface Trusted Allow option Rate limit (pps)

Richard,

Yes, we are using similar setup on other sites. There's no VRRP config here, just HSRP on the two 3560X. Still there's no DHCP config present on the standby switch.

Indeed as we speak I am left with this option: clear all DHCP config on active switch and test again by adding ip helper address on each SVI. If not maybe an active switch reload or even a switchover to the other switch.

Thanks for the heads up, gentlemen!