VPN Clients not getting response from internal server

ewellsie07 · ‎08-19-2009

I have a client that is using an IPSec VPN, with the Cisco VPN Client.

The clients connect fine, can map drives, use printers, get e-mail, etc.

However, they have one server that is not able to respond back to the clients.

According the server logs, it sees the clients connecting and appears to act like there is no problem, only the responses aren't being returned to the client.

I can't figure out what's going on, can anyone help?

Attached is the edited config, which contains a few capture access lists I was using for troubleshooting and a test VPN of "TownNamePD"

The "SPD" policy is the one with the problem here.

Also pasted below is what my packet capture shows as the problem.

The internal server is 192.0.0.249 and the clients are using 172.16.1.0

1: 13:43:47.040509 172.16.1.180.1041 > 192.0.0.249.7002: udp 4

2: 13:43:47.040662 192.0.0.249 > 172.16.1.180: icmp: 192.0.0.249 udp port 7002 unreachable

Please help.

Thanks.

ewellsie07 · ‎08-19-2009

I have confirmed that the client can ping this server by hostname + IP, and I can also browse this server using a UNC path and it works.

Really not sure why this program isn't working, unless it's a client/server setting with their program.

Yudong Wu · ‎08-19-2009

It looks like IP connectivity between VPN client and server is good. But your server is not listening to UDP port 7002. You need check the application setting on the server to see if it is listening to port udp 7002.

ewellsie07 · ‎08-19-2009

That's the thing, the clients ARE getting into the server, and it's sending the reply to the client IP 172.16.1.xx:7002 and that is what appears to be failing.

Yudong Wu · ‎08-19-2009

The vpn client was trying to talk to server on udp port 7002. But server was not listenning to UDP port 7002. You need check the application about why client would like to send the traffic to internal server on UDP 7002 and why internal server did not listen to this udp port.

sziaulla · ‎08-19-2009

Is the problem happening with any protocol or just the UDP/TCP.

I am interesting to know if you can ping this server from the ASA and also from the Client?

If you are able to see the response of icmp coming from internal server and going towards vpn client on the inside interface then you can check the syslog to see if the packets are getting dropped by ASA.

thanks

-Syed

ewellsie07 · ‎08-19-2009

It's just this particular program they are using. It's a client program connecting to a server program.

Pings work fine to the server, from the ASA, the client, etc.

Can even browse the server's files by using a UNC path.

Just this program that can't get responses from the server.

I've asked them to look at the server/client setup again just to make sure nothing is wrong there.

sziaulla · ‎08-19-2009

i am sorry. i was not able to see the response from Kevin earlier.

Pls ignore my comment on this issue.

sorry about the confusion.

-Syed

ewellsie07 · ‎08-19-2009

He sent me a log file from the client and the server.

It appears that the client is sending the request on port 7001, which reaches the server, and then the server tries this response:

8/19/2009 4:21:20 PM:UDP Sent Packet to Remote<172.16.1.187:7002, Len: 139> PG: 168 PID=1 of 1 Msg=

8/19/2009 4:21:20 PM:UDP Broadcast too:172.16.1.187

That's what is not reaching the client.

Client log:

<8/19/2009 4:18:52 PM> Resend: {8}<192.0.0.249:7001>1:1

<8/19/2009 4:19:01 PM> Resend: {8}<192.0.0.249:7001>1:1

<8/19/2009 4:19:08 PM> Resend: {8}<192.0.0.249:7001>1:1

<8/19/2009 4:19:16 PM> Resend: {8}<192.0.0.249:7001>1:1

<8/19/2009 4:19:24 PM> Report to user: MSG FAILURE Owner Msg ID=<172.16.1.187> QRID<1>

Server Log:

8/19/2009 4:21:05 PM:********* MSG FROM <172.16.1.187:3 bytes> ********

8/19/2009 4:21:20 PM:********* MSG FROM <172.16.1.187:291 bytes> ********

8/19/2009 4:21:20 PM:Found STX

8/19/2009 4:21:20 PM:Sent ACK to:172.16.1.187:7002

8/19/2009 4:21:20 PM:******** << MSG PACKET REC AND ACK SENT! GROUP ID=1 Packet 1 of 1 TimeOut=1 Len=272 **********

8/19/2009 4:21:20 PM:Entered Encryption AES 2

8/19/2009 4:21:20 PM:Done with Encryption AES 2

8/19/2009 4:21:20 PM:PACKETS TO SEND:1

8/19/2009 4:21:20 PM:Message Owner:

8/19/2009 4:21:20 PM:PACKINFO: PacketStart 168 ID 1Packets: 1

8/19/2009 4:21:20 PM:UDP Sent Packet to Remote<172.16.1.187:7002, Len: 139> PG: 168 PID=1 of 1 Msg=

8/19/2009 4:21:20 PM:UDP Broadcast too:172.16.1.187

8/19/2009 4:21:20 PM:PACKETS TO SEND:1

8/19/2009 4:21:20 PM:Message Owner:

8/19/2009 4:21:20 PM:PACKINFO: PacketStart 169 ID 1Packets: 1

8/19/2009 4:21:20 PM:UDP Sent Packet to Remote<172.16.1.187:7002, Len: 83> PG: 169 PID=1 of 1 Msg=

8/19/2009 4:21:20 PM:Deleted Que Msg: Start Packet=1 Packet ID:1 Packet Type: REC

8/19/2009 4:21:20 PM:Deleted: Start Packet=1 Packet ID:1 Packet Type:=5 RECOK

8/19/2009 4:21:21 PM:Checking Unit: 56

8/19/2009 4:21:21 PM:PACKETS TO SEND:1

8/19/2009 4:21:21 PM:Message Owner:

8/19/2009 4:21:21 PM:PACKINFO: PacketStart 170 ID 1Packets: 1

8/19/2009 4:21:21 PM:UDP Sent Packet to Remote<172.16.1.187:7002, Len: 43> PG: 170 PID=1 of 1 Msg=

8/19/2009 4:21:22 PM:Resend: <172.16.1.187>170:1

8/19/2009 4:21:22 PM:Resend: <172.16.1.187>168:1

8/19/2009 4:21:26 PM:Checking Unit: 56

8/19/2009 4:21:27 PM:********* MSG FROM <172.16.1.187:291 bytes> ********

I removed some sensitive information from those logs, but that's the gist of it.

Yudong Wu · ‎08-19-2009

Can you run a testing with a client in the same subnet of the server? (not go through vpn).

Capture the traffic on both client and server to see what the traffic flow should be.

Then run the testing through VPN again and capture traffic on both client and server.

Since you have tested IP connectivity between the server and vpn client is good, I am suspecting the issue should be on the client/server config of this specific application unless something in the middle to block traffic from server to client.

Compare both testing to see where is broken.

ewellsie07 · ‎08-20-2009

The client works fine on the internal LAN subnet, communication works both ways.

I have run nmap and confirmed that ports 7000-7006 are open on the server.

Yudong Wu · ‎08-20-2009

Well, in that case, we need figure out where the packet got dropped.

Do a capture on ASA and check ASA log as well when doing a testing.

ewellsie07 · ‎08-20-2009

Finally got with the client, and gained access to the application server, which happens to have 2 NICs, one on the internal subnet and one connected to another private state network, which the police department uses for information lookups.

After loading WireShark on the server, we noticed that packets destined for the VPN Subnet were heading out the wrong interface.

Added a persistent route to 172.16.1.0 out the internal firewall in the server's route table and now all is working fine.

Thanks for the help, turns out it wasn't really the VPN that was the issue.

Yudong Wu · ‎08-20-2009

glad you found the issue.