ASA-PIX: DHCP relay through VPN tunnel

Blog

Jan 5, 2011 10:43 AM
Jan 5th, 2011

Happy New Year to everyone!

I spent an entire day working on this issue that  I decided to discuss with everyone today.

(click image to enlarge)
kusankar-dhcp-blog.jpg

As you can see in the above topology, clients hanging off the inside interface of Local ASA are trying
to acquire an IP address from the DHCP server that is located on the other side of the tunnel behind the
Remote ASA.

Let us quickly run through how DHCP relay works.

1. Client  starts the DHCP process by sending a DHCPDISCOVER message to destination address
    255.255.255.255  - UDP port 67

2. ASA sees the broadcast and based on the dhcprelay server config it forwards the DHCPDISCOVER
    message as a unicast packet to the server's IP sourcing from the interface IP close to the server.
    In this case the outside IP address.

3. Server sends back DHCPOFFER as a unicast packet to the ASA - UDP port 67

    Server will send it to the destination IP address of the inside interface (giaddr) which is the
    dhcprelay enabled interface.

4. This packet will arrive on the outside interface and will be broadcast out the inside interface - UDP port 68.

5. Client receives the DHCPOFFER, and sends a DHCPREQUEST message to the server, that it accepts the offer.

6. The server will send back a DHCPACK message to the client.

7. Client upon receiving the DHCPACK, it will start communicating on the network.

Here is the partial config on the Local ASA:

dhcprelay server 10.48.65.13 outside
dhcprelay enable inside

DHCP relay packets will be sent to the server with the source address  of the outside interface IP
address and we need to encrypt that.

crypto acl includes:

access-list L-VPN extended permit ip host 10.252.17.6 host 10.48.65.13

access-list L-VPN extended permit ip host 10.150.1.1 host 10.48.65.13

Here is the partial config on the Remote ASA:

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255 (this is the problematic static)

static (inside,outside) 10.150.1.0 10.150.1.0 netmask 255.255.255.0

The first static 1-1 is a requirement because the DHCP server will only talk to IP addresses in
the 10.150.1.0/24 network.

crypto acl includes:

access-list R-VPN extended permit ip host 10.48.65.13 host 10.252.17.6

access-list R-VPN extended permit ip host 10.48.65.13 host 10.150.1.1

Technically the offer packets from the server will be sent to the interface IP address that has
dhcprelay enabled which is the inside interface 10.150.1.1 but, since this Remote ASA has
static 1-1 NAT configured it changes the destination IP address to 10.252.17.6 upon receiving

these packets destined to 10.150.1.1

Everyone with me so far? Now comes the interesting part, the problem.

The clients off the Local ASA never get an IP address.

According to debugs on the Local ASA the DHCP server seems to be sending offers but,
these packets do not egress the inside interface of the Local ASA.

dhcpd_forward_request: request from 0024.7e03.e3bf forwarded to 10.48.65.13.
DHCPRA: Received a BOOTREPLY from interface 1
DHCPRA: relay binding found for client 0024.7e03.e3bf.
DHCPRA: Adding rule to allow client to respond using offered address 10.150.1.10
DHCPRA: forwarding reply to client 0024.7e03.e3bf.

Now, where could they be going? On the outside we cannot capture because the traffic is
encrypted. Debugs and syslogs don't indicate any problem. I started to wonder if the ASA
was silently dropping these packets. Well, that should show in the asp drop captures shouldn't it?
Nope.  They weren't there either.

It was a mystery. I decided to throw a capture on the outside interface for udp 67 and udp 68
packets and wasn't I surprised to see the ASA broadcasting the offer packets off the outside interface!
Oh boy!! Why would it do that? The clients are on the inside so, why would this ASA send these packets
off the outside interafce and also create arp entries on the outside?

sh arp | i outside
arp outside 10.151.1.19 0024.7e03.d45d

arp outside 10.151.1.10 0024.7e03.e3bf

This was extremly strange. MAC address belongs to clients on the inside. DHCP offered IP address is
mapped to the MAC address properly but the interface is wrong.
No wonder the inside clients don't
get the IP address!

Well, now we need to investigate why the ASA is doing what it is doing.  Well, time to read the DHCP RFC:

According to section
4.1.2 BOOTREPLY Messages
...

The 'giaddr' field can be used to identify the logical interface from which the reply must be sent (i.e., the host
or router interface connected to the same network as the BOOTP client). If the content of the 'giaddr' field does
not match one of the relay agent's directly-connected logical interfaces, the BOOTREPLY messsage MUST
be silently discarded
....

The ASA is supposed to receive these offer packets destined to the inside interface IP on the outside interface
but, due to the static 1-1 on the Remote ASA this is not the case. The offer packet's destinated IP address gets
changed to the outside IP address. Upon receiving these packets, without looking at the "giaddr" in the packet the
ASA chooses to broadcast it out on the outside interface thinking the clients are off of the outside interface which
is clearly not enabled for dhcprelay.

I know what everyone is thinking. Time to file a bug. Right. But, we need to fix the probelm at hand.  What can
we do resolve the problem right now?Using static IP address on the client PCs is not an option. Somehow we
need to make sure that the destination IP address in the offer packets sent by the server does not get changed.
How can we accomplish this? If we can do that then, things will start to work and we can file a bug later.

We decided to change the static 1-1 on the Remote ASA as follows:
static (inside,outside) 10.150.1.6 10.252.17.6 netmask 255.255.255.255

We just picked this address
10.150.1.6 that was not used anywhere. Since these are just udp packets, when going to
the server these will have the source address of 10.150.1.6 but, when the server responds back it will respond back to the
"giaddr" which is 10.150.1.1, will take the identity static and reach the Local ASA.

Once we did the above, DHCP relay started to work as expected.
I have filed this defect CSCtj68732.(NOTE: you will need to login with your Cisco Connection Online "CCO" credentials to access)

I hope this was informative and thanks for reading.

View sample dhcp captures
http://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=view&target=dhcp.pcap

-Kureli


Average Rating: 5 (2 ratings)

Comments

Mirza Arsalan Baig Tue, 01/11/2011 - 00:11 (reply to Poonguzhali Sankar)

Hi Kureli,

static (inside,outside) 10.150.1.6 10.252.17.6 netmask 255.255.255.255

Will it work for any broadcast to send inside to outside (remote user over ip secc VPN tunnel)

Regards

Arsalan Baig

Poonguzhali Sankar Sat, 01/15/2011 - 16:06 (reply to Mirza Arsalan Baig)

Arsalan,

IPsec VPN does not support broadcast or multicast.  It supports only unicast traffic.

DHCP is broadcast but, the ASA acts as a relay, taking in the broadcast and sending a unicast packet out towards the DHCP server.

-Kureli

mayrojas Wed, 01/12/2011 - 14:20

Hi Kureli,

Isn't this static on the remote ASA backwards?

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255

Cheers

Mike

lcaruso Sun, 01/16/2011 - 08:25 (reply to Poonguzhali Sankar)

Great discussion and diagram.

Does this discussion apply to a situation where the tunnel terminates on the outside interface of both ASAs? Isn't that the more typical scenario, or to ask it this way, why would you want to terminate a tunnel on the inside interface instead of the outside interface?

I have a problem open with TAC for dhcp relay using the outside-to-ouside tunnel configuration, so I'm wondering if your discussion will solve this case since they said my configuration was correct.

Thanks.

Poonguzhali Sankar Sun, 01/16/2011 - 10:33 (reply to lcaruso)

Thank you. I put in a lot of effort. Seems like it has paid off. Yes, it does apply to tunnels terminated on the outside. In this dicussion the tunnel is terminating on the inside of the remote ASA and for some odd requirement they chose to translate the outside interface IP of the Local ASA to the dhcp relay enabled interface IP address on the Remote ASA which caused the whole problem.

Pls. send this link to the TAC engineer so, they can investigate if this applies in your case or may be reach out to me internally.

Well, the reason for terminating the tunnel on the inside in this case is that the network behind the Remote ASA is untrusted or customer network.

-Kureli

smailbouabdallah Fri, 08/05/2011 - 07:00

Hi,

How we configure the DHCPrelay if we have only one ASA for IPSEC remote VPN? I tried to configure the DHCPrelay and I could not receive an IP address.

Thanks,

stuartpatton Thu, 09/15/2011 - 05:13

This article is a great find because lots of other people on the internet have said rather confidently that you add the local firewall's inside and outside IP address into the cryptomap ACL to tunnel the DHCP relay traffic but have obviously never tested  that it works.

Has anyone implemented this with the remote ASA (per the diagram) using 8.3 NATs?  I have tried everything I can think of but end up getting "Deny inbound UDP" messages.  These are not ACL drop messages, but indicate there is something wrong with the NAT...just can't figure it out.

Thanks

Poonguzhali Sankar Thu, 09/15/2011 - 05:29 (reply to stuartpatton)

Thank you Stuart.  I am not sure your scenario is exactly the same as the one that I mentioned above. The tunnel is terminating on the inside (high sec interface) of the remote ASA. Whether 8.3 or pre 8.3 this should work. You are correct Deny inbound UDP messages are not the ones that are denied due to ACL.  The interface for some reason is rejecting the packets. asp-drop captures should tell you why.

Open a thread on our discussion forum

https://supportforums.cisco.com/community/netpro/security/firewall

and we can discuss this further and get you up and running. Another thought is to open a TAC case which would be better because of the complexity involved in the topology and all the data that needs to be gathered simultaneously and analyzed.

-Kureli

vijayasankar Sun, 12/04/2011 - 12:57

Hi,

We have the same problem observed in one of the working setup which is exactly similar to the setup that you have explained.

The observations remains the same.All of the sudden, the dhcp relay feature is broken.

WE have changed the static nat to another free ip at the remote firewall, and still the problem persists.

Here's what i have found more on this.

When i SPAN the local firewall outside interface ip and took the sniffer trace, i could see that all the DHCP offer messages from remote DHCP server is getting brodcasted on the outside interface of the firewall.

The DHCP offer packet is supposed to be transmitted by the local firewall to its inside interface, strangely for some reasons it is now not doing it.

As we lost almost 1 day with Business escalation, we have removed the dhcp relay configurations from local firewall and turned on DHCP relay in one of a L3 switch behind the local firewall. All is working well now as expected.

Certainly Not sure, why this feature should suddenly break, in a working setup..Any ideas.....

Poonguzhali Sankar Sun, 12/04/2011 - 15:03 (reply to vijayasankar)

Vijayasankar,

I believe the offer packets were encrypted so, you couldn't have possibly seen the desination IP address for these unicast offer packets.  I believe the NAT change may not have taken effect. Do you recall issuing a "sh xlate debug | i o.o.o.o for the local ASA's outside IP on the remote ASA and make sure it was taking the newly configured static to the un-used address in the server's preferred subnet? Did you also issue "sh xlate debug | i i.i.i.i for the inside IP address of the remote ASA so, it looked itself and DID NOT get changed back to the outside IP o.o.o.o?  Anytime you change NAT you would also have to clear all xlates and connections by issuing "clear local o.o.o.o" "clear local i.i.i.i" on the remote ASA where o.o.o.o is the IP address of the outside interface and i.i.i.i is the inside IP of the local ASA. I did discuss this issue with your engineer today. This kind of translation is not recommended - I mean translating the outside IP to inside IP of one ASA on another ASA.  NOT a good idea at all.  Always use an un-used address. 

-Kureli

sansarav720e Fri, 12/16/2011 - 20:51 (reply to vijayasankar)

Hi Vijay ,

          Could you please brief your problem clealry , I do have a same problem in my setup , below to my firewall i have L3 switch where i have L3 SVI under this i have defined IP helper address (remote dhcp server IP)  ,where my end machine on this SVI VLAN will direct DHCP request to this IP helper address . But i cant see end user machine is offered IP address from remote dhcp server .

              when i define IP helper address on interface VLAN ,it will be unicast dhcp message or broadcast message ?? how it will because i could not see any hits on my firewall inside interface .. I have done capture also ,even i have defined specific UDP rule for DHCP i could nt see any hits on ACL .

I guess the helper address on interface VLAN is once again UDP broadcast messgae , in my scenario how this DHCP service is achievable over VPN tunnel ??


vijayasankar Sun, 12/18/2011 - 22:38 (reply to sansarav720e)

Hi Sarvanan,

My setup was quite to accomodate a client restriction and hence it had challenges for DHCP relay. The workaround that i have implemented for that scenario, was infact moving the DHCP relay feature from the Firewall to L3 switch in Inside segment. It will work perfectly fine.

You need to ensure that appropriate rules are available in the firewall to allow this traffic ( and the encryption ACLs).

The flow will be as follows

1) MAchines broadcast DHCP discover message.

2) Switch SVI will listen to DHCP broadcast and relay the same onbehalf of the desktop as a unicast request to the DHCP servers defined in the helper address.

3) This unicast packet will flow through the firewall and reach the DHCP server.

4) DHCP server will send a offer message to your L3 SVI unicast

You can enable packet capture in the Firewalls inside interface for the packets originates/destined to the SVI. You should be able to see the DHCP discover unicast and offer unicast here. If not, some thing is wrong in the way the firewall and relay is setup.

Regards

Vijay

sansarav720e Sun, 12/18/2011 - 23:41 (reply to vijayasankar)

Hi Vijay ,

    Thank for your reply note ,  As part of security measure we have disabled service dhcp on our network switches , due to this dhcp relay function was not working . IP helper address under SVI L3 VLAN was not redirecting dhcp request as unicast . due to service dhcp deactivation .

       I have turned on service dhcp along with

ip forward-protocol udp bootpc

ip forward-protocol udp bootps , then also service havent started properly .Then have given hardboot to network switches with service dhcp turned on , after reloading service dhcp was performing relay functioning properly , where i can see dhcp request as a unicast to remote dhcp server . Then my end machine was able to recieve ip address seamlessly .

one strange thing about this IOS code is ,when i have enabled service dhcp , relay function was not turned on , after rebooting it was working automatically . i have given out 15 hours without rebooting by enabling service dhcp & no dhcp .At last i had no chance to troubleshoot then adopted for rebooted , luck turned on .. having the services offering.

Quite good one...

below is debug for dhcp events.. dhcp server ip  address (10.84.100.218)

end client ip address from dhcp : 10.4.110.21/24

Dec 18 23:24:21.768 IST: DHCP: SDiscover: sending 306 byte length DHCP packet

Dec 18 23:24:21.768 IST: DHCP: SDiscover 306 bytes

Dec 18 23:24:21.768 IST:             B'cast on Vlan280 interface from 0.0.0.0

Dec 18 23:24:22.548 IST: DHCP: Received a BOOTREP pkt

Dec 18 23:24:22.548 IST: DHCP: offer received from 10.84.100.218

Dec 18 23:24:22.548 IST: DHCP: SRequest attempt # 1 for entry:

Dec 18 23:24:22.548 IST: DHCP: SRequest- Server ID option: 10.84.100.218

Dec 18 23:24:22.548 IST: DHCP: SRequest- Requested IP addr option: 10.4.110.21

Dec 18 23:24:22.556 IST: DHCP: SRequest placed lease len option: 86400

Dec 18 23:24:22.556 IST: DHCP: SRequest: 324 bytes

Dec 18 23:24:22.556 IST: DHCP: SRequest: 324 bytes

Dec 18 23:24:22.556 IST:             B'cast on Vlan280 interface from 0.0.0.0

Dec 18 23:24:22.833 IST: DHCP: Received a BOOTREP pkt

Dec 18 23:24:25.836 IST: DHCP Client Pooling: ***Allocated IP address: 10.4.110.21

Dec 18 23:24:25.954 IST: Allocated IP address = 10.4.110.21 255.255.255.0

Thanks for your time .........

vijayasankar Sun, 12/18/2011 - 23:58 (reply to sansarav720e)

Oops. That was a terrible problem, when the features dont work as expected. Good that it is resolved now. Thanks for the update.

vijayasankar Sun, 12/04/2011 - 23:11

Hi,

Thanks a lot for the update.

The NAT changed was indeed taking effect and we could verify that using the packet capture taken in the remote firewall. The offer packets were coming from DHCP to the new NATTed id.

We could see all the DHCP discovery and DHCP offer packets when we do the pacekt capture in remote firewall.

The anomaly is observed on the local firwall.

The local firewall is receiving the DHCP offer from remote firewall,

however instead  of transmitting the offer packets to inside interface, the local firewall is sending the offer packet to the outside interface LAN. When we clear the ARP table for the outside interface in the local firewall and observe for a while, we could see slow the ARP table starts showing more inside ip address learned in the outside interface.

Apparently changing the NAT ip address had not changed the problem behavior. Whatever IP address we use to NAT, the local firewall was consistenly transmitting the DHCP offer only to the outside and not the inside interface.

We have erased the configuration completely on the local firwall, reloaded it, reconfigured it the same way and still the problem persists with the same behavior. Changed the NAT once more and collected the packet capture and logs and the problem symptom was consistent.

In my  network, i dont have a similar setup. This is very unique requirement for this setup.

Im not sure if such setups are  supported in the ASA firewall. May be im trying to make something work which is not expected to work (or) not as per a standard implementation... Whats your comment on this...

I have plenty of more setup like this which works perfectly fine. The only difference is in those setup, IP SEC vpn is not involved. DHCP relay feature would be turned on in teh inside interface and the relay is configured to reach a DHCP server which is located on a routed network after the firewalls outside interface. These setups works perfectly fine with no glitch.

Is this behavior of ASA firewall very unique when it comes to using DHCP relay with a IP Sec VPN tunnels and DHCP server which is routed over a IpSec Tunnel?

thanks/ Regards

Vijaya Sankar.K

Poonguzhali Sankar Mon, 12/05/2011 - 05:57 (reply to vijayasankar)

Vijaya Sankar,

The NAT changed was indeed taking effect and we could verify that using  the packet capture taken in the remote firewall. The offer packets were  coming from DHCP to the new NATTed id.

That NAT change to an un-used address is only for relay packets (request) going to the DHCP server - is only for one way traffic. DHCP server should send offer packets to the GI-ADDR (which is the IP of the inside int address of the lcoal ASa) and NOT the newly NAT-ed address. Pls. clarify your above statement.

Added to that you can only gather clear captures on the remote ASA on the outside interface. On the inside interface you cannot because traffic will be encrypted.  The offer packet's destination IP address should be the inside interface IP address of the local ASA while egressing the inside interface towards the local ASA.  Did you validate that?  How?

This setup is supported. As you can read in the comments that another reader had a similar setup in 8.3+ NAT and we resolved the issue right here on our forum.

The KEY here is that L-3 destination IP of the offer packets from the DHCP server when arriving on the local ASA, should remain un-changed all the way from the DHCP server THROUGH the remote ASA TO the local ASA and remain as the local ASA's inside interface IP address. 

The outstanding issue on Cisco side is the defect CSCtj68732 that we filed.  It is still not resolved.  So long as the outside interface on the local received offer packets destined to the inside int. IP, the ASA will process the offers properly - as the defect states.

<B>Symptom:</B>
DHCP Relay functionality on the PIX/ASA may incorrectly make a broadcast decision based the 
Layer-3 Destination IP instead of the embedded GI-Addr address. This does not conform to
BOOTP Relay RFC.. <B>Conditions:</B> The behavior is the same on all ASA/PIX codes. <B>Workaround:</B> Ensure that the BOOTP replies arrive at the ASA with a Layer-3 Destination IP address of the
interface facing the DHCP clients.

-Kureli

jurgendendas Mon, 02/06/2012 - 06:15

Hello I did everythting you said, but it is still not working.

What can I do more to let it work???

Kind regards.

Poonguzhali Sankar Mon, 02/06/2012 - 06:33 (reply to jurgendendas)

Sorry to hear that.  If you have smartnet, I'd suggest opening a case and refering this doc.  This gets really involved and we need to gather captures and debugs in multiple points to figure out what may be causing this.

Let me know the SR no. once opened.

Thanks,

Kureli

sansarav720e Mon, 02/06/2012 - 21:42 (reply to jurgendendas)

Hi Jurgen ,

    Can you explain your problem along with your exsiting network diagram . Simialrly you need to make sure all basic settings are up running ..

stuartpatton Mon, 10/08/2012 - 01:24

Incidentally, I just noticed that defect CSCtj68732 filed by Poohguzhali Sankar was fixed in 8.4.4(5)

ju_mobile Thu, 10/25/2012 - 11:11 (reply to stuartpatton)

Kureli,

Please can I ask in what circumstances would you need to apply the NAT on the REMOTE ASA.

Here is the partial config on the Remote ASA:

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255 (this is the problematic static)

static (inside,outside) 10.150.1.0 10.150.1.0 netmask 255.255.255.0

is this a mistype and do you mean local

many thanks

Julian

Poonguzhali Sankar Fri, 10/26/2012 - 05:41 (reply to ju_mobile)

Julian,

Like Stuart mentioned, the defect CSCtj68732 has been resolved in 8.4.4(2) and above.  I listed one possible reason right below the static NAT lines on the remote ASA.

Here is the partial config on the Remote ASA:

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255 (this is the problematic static)

static (inside,outside) 10.150.1.0 10.150.1.0 netmask 255.255.255.0

The first static 1-1 is a requirement because the DHCP server (on the remote network) will only talk to IP addresses in the 10.150.1.0/24 network.

-Kureli

ju_mobile Fri, 10/26/2012 - 07:34 (reply to Poonguzhali Sankar)

I guess that the external address of the local ASA has to be in the interesting traffic crypto-map to pass it down the tunnel.

The nat on the remote ASA is defined for the return traffic. I've looked at this with the latest interim code 8.44(9) and I couldn't get the helper request to go down the tunnel. In a packet capture wizard you can see it leave the outside interface on the local ASA. Was there any additional configuration required apart from the normal route ACL translation (RAT)?

Best Regards

Julian

ju_mobile Fri, 10/26/2012 - 01:33 (reply to stuartpatton)

Hi Stuart,

Many thanks for your response,  I've download the interim release and I'm just about to test and will post my findings.

Best Regards

Julian

william_niceman... Wed, 11/21/2012 - 07:44

I did not have any problem setting a DHCP relay across the L2L tunnel with ASA version 8.4(5).  The only problem I experienced is the delay time when the wireless clients tried to get the DHCP IP addresses.

Poonguzhali Sankar Wed, 11/21/2012 - 08:08 (reply to william_niceman...)

William,

The ASA where we identified the problem was running 8.2.3 at the time.  I have not seen any problem with 8.4.5.  If it works with delay may mean some bottle neck along the line.  Captures taken at various points would benefit in this situation so we can see where the delay is introduced.  With additional ipsec headers, MTU comes into picture.  Pls. check that on all L-3 devices along the way and lower MSS if needed.

-Kureli

mikael.sveden Wed, 01/09/2013 - 12:52

Hi and thanks for a good and helpful article.

However I have problems to get this to work. In my case I have two PIX 506E (Version 6.3(5)) and 515E (Version 8.0(4)). 506E is my local PIX where the DHCP clients is located and 515E is the remote PIX where the DHCP server (192.168.13.11) is located. My configuration looks like this:

Local device (inside subnet 192.168.20.0/24, outside ip 95.109.123.218):
dhcprelay server 192.168.13.11 outside
dhcprelay enable inside

access-list L-VPN line 1 permit ip host 95.109.123.218 host 192.168.13.11
access-list L-VPN line 2 permit ip host 192.168.20.1 host 192.168.13.11

Remote device (inside subnet 192.168.13.0/24, outside ip 217.208.207.43):

static (inside,outside) 192.168.20.6 95.109.123.218 netmask 255.255.255.255
static (inside,outside) 192.168.20.0 192.168.20.0 netmask 255.255.255.0

access-list R-VPN line 1 extended permit ip host 192.168.13.11 host 95.109.123.218
access-list R-VPN line 2 extended permit ip host 192.168.13.11 host 192.168.20.1

Can you please help me?

Regards Mikael

gauravg2 Wed, 08/21/2013 - 05:28

Hello ,

I have my customer who is running 8.4.5 code and has it running i addded all the above NAT statement

As i am using different subnet so could you tell me what is 10.150.1.6 ip address i don't see it as any of the interface ip address.

Also i have added identity NAT on remote ASA  for DHCP server to outside interface ip address and to inside interface ip address.

So would it be required .

regards

Gaurav

Actions

Login or Register to take actions

This Blog

Posted January 5, 2011 at 10:43 AM
Stats:
Comments:38 Avg. Rating:5
Views:22733   
Shares:0

Related Content

Blogs Leaderboard