ASA 8.4(1) ftp passive problem with NAT

Unanswered Question
Mar 4th, 2011

Hi !

We have 2 ASA 5580 with a cluster active/standby configuration

We have updated to version 8.4.(1) since version 8.3(1) but since then it is impossible to establish the FTP connection in passive mode with NAT.

Before this update, all was OK.

Here our configuration :

class-map global-class
match default-inspection-traffic
!
!
policy-map global-policy
class global-class
  inspect dns
  inspect http
  inspect icmp
  inspect icmp error
  inspect sunrpc
  inspect tftp
  inspect pptp
  inspect rtsp
  inspect ftp
!
service-policy global-policy global

Do you know if it's a bug or you can fixed this problem ?

Thank you very much for your help.

Regards,

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
gdelavenne@inte... Fri, 03/04/2011 - 15:24

Thanks for your response.

We have no errors in syslog messages and "show service-policy" display :

Inspect: ftp, packet 650742, lock fail 0, drop 0, reset-drop 8

Paul Gilbert Arias Fri, 03/04/2011 - 15:35

ok, there are some drop-resets on your service-policy that could be the cause. Is it possible for you to test the FTP connection and check the logs and the service-policy to see if the number increases?

gdelavenne@inte... Sat, 03/05/2011 - 04:39

Inspect: ftp, packet 771540, lock fail 0, drop 0, reset-drop 8

The reset-drop does not increase.

Why inspection work without NAT?

We are the only ones with this behavior (version 8.4.1)?

If you have any ideas, thank you for your help!

mayrojas Sat, 03/05/2011 - 09:03

Hi,

Would you please answer the following questions? I know this is suppose to work on this version as well, but I want to analyze some data:

Where is the server located?

Were is the client located?

What are the security levels for the interfaces?

Can you get the logs when the connection doesnt work?

Would you please get a packet capture with all TCP between the client and the server?

Cheers

Mike.

gdelavenne@inte... Sat, 03/05/2011 - 15:06

Hello Mike,

Thank you for the interest.

Here our answer :

Where is the server located?

The server is behind our ASA 5580 connected on an Vlan interface.

Were is the client located?

The client comes from Internet.

What are the security levels for the interfaces?

All our interaces are in security level 100.

Can you get the logs when the connection doesnt work?

We have no log when the connection doesnt work.

Would you please get a packet capture with all TCP between the client and the server?

sh capture ftp

48 packets captured

   1: 00:00:19.577011 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: S 2847225137:2847225137(0) win 5840
   2: 00:00:19.577225 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: S 1447625788:1447625788(0) ack 2847225138 win 5792
   3: 00:00:19.605635 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625789 win 92
   4: 00:00:19.607527 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625789:1447625799(10) ack 2847225138 win 46
   5: 00:00:19.637860 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625799 win 92
   6: 00:00:26.124779 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225138:2847225154(16) ack 1447625799 win 92
   7: 00:00:26.125039 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: . ack 2847225154 win 46
   8: 00:00:26.125054 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625799:1447625833(34) ack 2847225154 win 46
   9: 00:00:26.152518 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625833 win 92
  10: 00:00:29.892226 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225154:2847225170(16) ack 1447625833 win 92
  11: 00:00:29.914823 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625833:1447625856(23) ack 2847225170 win 46
  12: 00:00:29.941601 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625856 win 92
  13: 00:00:29.943173 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225170:2847225176(6) ack 1447625856 win 92
  14: 00:00:29.943447 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625856:1447625875(19) ack 2847225176 win 46
  15: 00:00:30.011428 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625875 win 92
  16: 00:00:32.052746 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225176:2847225182(6) ack 1447625875 win 92
  17: 00:00:32.053097 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625875:1447625921(46) ack 2847225182 win 46
  18: 00:00:32.082500 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625921 win 92
  19: 00:00:32.083629 802.1Q vlan#832 P0 82.239.4.178.48382 > 10.34.4.37.23061: S 2327685571:2327685571(0) win 5840
  20: 00:00:32.083796 802.1Q vlan#832 P0 10.34.4.37.23061 > 82.239.4.178.48382: S 1457888673:1457888673(0) ack 2327685572 win 5792
  21: 00:00:32.109781 802.1Q vlan#832 P0 82.239.4.178.48382 > 10.34.4.37.23061: . ack 1457888674 win 92
  22: 00:00:32.111505 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  23: 00:00:32.287186 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625875:1447625921(46) ack 2847225182 win 46
  24: 00:00:32.314757 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625921 win 92
  25: 00:00:32.340695 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  26: 00:00:32.755072 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625875:1447625921(46) ack 2847225182 win 46
  27: 00:00:32.781865 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: . ack 1447625921 win 92
  28: 00:00:32.803287 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  29: 00:00:32.803806 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46
  30: 00:00:32.803867 802.1Q vlan#832 P0 10.34.4.37.23061 > 82.239.4.178.48382: . 1457888674:1457889998(1324) ack 2327685572 win 46
  31: 00:00:32.803898 802.1Q vlan#832 P0 10.34.4.37.23061 > 82.239.4.178.48382: P 1457889998:1457890049(51) ack 2327685572 win 46
  32: 00:00:32.803898 802.1Q vlan#832 P0 10.34.4.37.23061 > 82.239.4.178.48382: F 1457890049:1457890049(0) ack 2327685572 win 46
  33: 00:00:32.842241 802.1Q vlan#832 P0 82.239.4.178.48382 > 10.34.4.37.23061: . ack 1457889998 win 137
  34: 00:00:32.842973 802.1Q vlan#832 P0 82.239.4.178.48382 > 10.34.4.37.23061: . ack 1457890049 win 137
  35: 00:00:32.882019 802.1Q vlan#832 P0 82.239.4.178.48382 > 10.34.4.37.23061: . ack 1457890050 win 137
  36: 00:00:32.882232 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625960:1447625984(24) ack 2847225188 win 46
  37: 00:00:33.038007 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46
  38: 00:00:33.508793 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46
  39: 00:00:33.729484 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  40: 00:00:34.448508 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46
  41: 00:00:35.581695 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  42: 00:00:36.327955 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46
  43: 00:00:39.281632 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  44: 00:00:40.087809 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46
  45: 00:00:46.688517 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  46: 00:00:47.607512 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46
  47: 00:01:01.505497 802.1Q vlan#832 P0 82.239.4.178.58805 > 10.34.4.37.21: P 2847225182:2847225188(6) ack 1447625921 win 92
  48: 00:01:02.647030 802.1Q vlan#832 P0 10.34.4.37.21 > 82.239.4.178.58805: P 1447625921:1447625960(39) ack 2847225188 win 46

Our customers are impacted by this problem and all worked well before the update.

Thank you so much for you help.

Best regards,

mayrojas Sat, 03/05/2011 - 15:14

Hi,

Would it be possible for you to get the capture on a pcap format? I want to take a look at the payload of the packets, from what I can see, there is a secondary connection being opened

  19: 00:00:32.083629 802.1Q vlan#832 P0 82.239.4.178.48382 > 10.34.4.37.23061: S 2327685571:2327685571(0) win 5840
  20: 00:00:32.083796 802.1Q vlan#832 P0 10.34.4.37.23061 > 82.239.4.178.48382: S 1457888673:1457888673(0) ack 2327685572 win 5792
  21: 00:00:32.109781 802.1Q vlan#832 P0 82.239.4.178.48382 > 10.34.4.37.23061: . ack 1457888674 win 92

Makes me thing that this can be the data channel but not sure. In order to download them do the following:

Enable HTTP server on the interface where the management station is


HTTP server enable

Enable access via HTTP to that host

HTTP x.x.x.x y.y.y.y.y inside

And then put the following url

https:///capture//pcap

Also, if you can please take a capture of ASP drop just to make sure that the ASA is not dropping anything:

capture asp type asp-drop all

If you can get the logs using ASDM or a syslog server that would be great.

Mike.

mayrojas Sun, 03/06/2011 - 12:35

Hello,

Based on the captures it seems that the server is sending again the "Entering to passive mode message" which is causing the retransmissions. However, I can see the data channel being opened. If you try to do the command list or try to pull a file, does it work?

Let me know.

Mike

gdelavenne@inte... Sun, 03/06/2011 - 13:59

Mike,

We observed many log :

<163>%ASA-3-210005: LU allocate connection failed

Can there be a relationship between the log and our FTP connection problem?

Thanks
mayrojas Sun, 03/06/2011 - 14:13

Hello,

Not really, since the connection is estsblished with no problems. we may need to do in deep troubleshooting on this case. Clearly we are missing packets on the connection, however, I am unsure if the data channel worked fine. I can see that when you did the listing of the directory it completely. So my big question is, what is it that is not working? Is it polling a file?

If it is so, please take a captures on the server, inside and outside of the firewall and get them on pcap format.

By any change, do you have a CSC module attached to this firewall?

Let me know.

Mike Rojas

ryan.palamara Mon, 06/20/2011 - 13:30

I know that this has been dead for a while, but I have similar problems with passive FTP and 8.4.1 for the ASA. Since upgrading to this version, passive FTP drops consantly from some servers. I have identified that it is an issue with IPS on the firewall. With my issue, the firewall is creating out of order packets. Cisco TAC has been working on it for weeks and has no idea.

I know that a new version of the ASA software just came out today, and I am planning on upgrading it tonight. I have had nothing but issues with the 8.4.1 version. Garbage if you ask me.

I have had the following issues:

1. FTP dropping issues

2. IPSEC L2L VPN tunnel drops. packets get dropped at random. , although a small percent. TAC has not been able to track down the problem.

3. A lot of asp drops. TAC as of yet unable to determine why.

All of these issues are not major by themselves, but after people started reporting issues within a few days, I reviewed my log server and found that the problems maifested themselves the second that I switched over to 8.0.3. My config has been verified 3 seperate times by TAC and has no issues.

I was about to downgrade to a previous version before 8.4.2 was released today. I will see if that corrects the issues.

Poonguzhali Sankar Mon, 06/20/2011 - 16:20

Pls. provide case numbers if you have them. Unless TAC pointed out documented defects that were resolved in 8.4.2, the probelms that you are seeing in 8.4.1 might still be there in 8.4.2.

-KS

mayrojas Mon, 06/20/2011 - 20:42

Thanks a lot Kureli, you are right. Maybe more eyes could help to determine what the problem is, upgrading may not be the desire path cuz the issue may remain.  Maybe taking a look at the documentation on the case will help us to check and see what the root cause can be.

Mike

ryan.palamara Tue, 06/21/2011 - 07:00

yes,  the issue was not corrected by the software upgrade.

The ticket open on this is 618020667.

dporod Tue, 06/21/2011 - 07:49

We also seem to be running into this problem with a 5585x 8.4.1 with IPS. One server works and the other does not. The non-working servers is NATed to an internal number, the working server is not.

mayrojas Tue, 06/21/2011 - 09:11

Dprod,

Please verify if you are not hitting the following defect CSCto09465. Ryan, I checked on the case and the captures were taken only on port 21. I have asked this question before, what is it that does not work? What happens when you try to pull a file?

Mike

dporod Tue, 06/21/2011 - 09:26

Hi, we are running into CSCto09465 also, my understanding is that CSCto09465 applies to active ftp. We also seem to have issues with passive ftp.

mayrojas Tue, 06/21/2011 - 09:54

Hi,

Ok, can you take captures and paste them here as pcap format? I want you to take the captures with IP instead of just the control channel of FTP.

******* Capture configuration ******

{Enable GUI interface:}

http 0 0 inside

http server enable

{For outside interface:}

access-list capture1 permit ip host   host

access-list capture1 permit ip host host

{For inside interface:}

access-list capture2 permit ip host host

access-list capture2 permit ip host host

capture tcpin access-list capture1 interface outside

capture tcpout access-list capture2 interface inside

****** To download the files then… *****

Open the browser

https:///capture/tcpin/pcap

https:///capture/tcpout/pcap

Note:

Username: blank = no name

Password: {enable password}

********* To delete them *********

clear access-list capture1

clear access-list capture2

no capture tcpin

no capture tcpout

********** End *********

Let me know.

Mike

Poonguzhali Sankar Tue, 06/21/2011 - 10:17

I do not believe that this ASA5510 is being hit by this bug CSCto09465.

The RN clearly says ASA5510s are NOT affected.

Conditions:
To hit this bug, the following conditions must be met:
1) The ASA must be running version 8.4(1) or greater
2) The ASA must have multiple CPUs. ASA 5580 and 5585 platforms are affected by this problem. The ASA 5505, 5510, 5520, 5540 and 5550 platforms are NOT affected by this problem
3) The FTP connection must be subjected to port address translation (PAT) on the ASA. Connections subjected to static NAT, or connections that do not hit any NAT rule on the ASA will not encounter this problem.

Control channel stalls meaning data channel cannot be open. In this case data starts to flow as the text based captures show but, seem to break later.  We need to see the entire capture until the very end and may be a few seconds later after the break so, we what might be going on.

-KS

mayrojas Tue, 06/21/2011 - 10:57

Yeah, but dprod was running with an ASA 5585 and the customer on the main threat has an ASA 5580. There is no ASA 5510 mentioned.

Anyways, we asked for the capture, and we still have none, so it is going to be very difficult to troubleshoot with no Info.

Mike.

ryan.palamara Tue, 06/21/2011 - 11:05

I am agreeing with you. I dont think that this is the same bug that was affecting dprod.

I can get a packet capture a little latter on. I have some already running from the ASA, and spanned ports before and after the ASA.

mayrojas Tue, 06/21/2011 - 11:30

Dprod,

In your particular case, you will need to take a capture on the server, as I am able to see the list command passing thru the device, but as for the retransmissions, it seems like the LIST command is not getting to the server, would you be able to take a capture there?

Mike

dporod Tue, 06/21/2011 - 11:35

FYI, when I try this internally (directly to the internal IP address of the server) rather than going through the firewall there is no problem.

mayrojas Tue, 06/21/2011 - 11:40

Nice to hear that, however, I need to make sure that the server is able to see a packet coming from 67.52.213.103 with the list command on it. As per the firewall, the packet is passing thru.

Mike

mayrojas Tue, 06/21/2011 - 11:27

Unfortunately no, but you can create a temporary username and password for the session then delete it.

Mike

r.robins Wed, 06/22/2011 - 12:58

Hi All,

I too have hit this problem with ASA 5520 running 8.4.1 in transparent mode.

Does anyone have any idea what the fix is for this, if there is a fix at all.

mayrojas Wed, 06/22/2011 - 13:44

Hi,

Same thing, there is no issue reported with FTP and ASA 5520, please paste the configuration and get the captures as described earlier.

Mike

r.robins Thu, 06/23/2011 - 00:34

Already had a case open with TAC and provided config and captures.

Took so long that we had to put a workround in and it difficult to replicate now.

TAC case SR 617830879 - We would like to see if their is a resolution to this issue so we can take out the workround in place.

ryan.palamara Fri, 06/24/2011 - 16:11

Sorry that I never posted any packet captures, but when troubleshooting was being done on the ASA with cisco, it caused all sorts of other problems, so work was halted until the weekend.

Well here is what I have found. to try and resolve this, since it was dragged on for 3 weeks with not even knowing what is causing the problem, I gave up on our TAC ticket and decided to try the new software 8.4.2.

Well that was a big mistake. I dont know if anyone else is having issues, but it is a complete disaster. It may fix FTP issues with the 5580, but it broke all FTP on my 5510. Active and passive FTP were broken on all external servers. Now instead of just a few ftp servers not working, none of them work.

Also to top it off, our 50 Mbps connection started running at 1/5 of that speed. Looking at the outside router with netflow, the ASA was supplying 10Mbps max. Again I dont know if I am a unique case, but what a piece of garbage update. Again, my config has been reviewed for the past 3 weeks by Cisco and has been declared fine.

So now 8.4.1  and 8.4.2 are working horribly, so I downgrade to 8.3.2. Guess what? Everything works perfectly now. FTP has no issues. What used to be running at 10Kbps and timing out constantly now runs at 1 MBps, 800x the speed isnt a bad improvement. I get no out of order packets. the speed issues for all IP transfers that were there is 8.4.2, are gone.

Overall, I am very displeased with how this was handled. Cisco went back and forth with me that the firewall was not supposed to do that, and so it could not be the firewall. Well that is why I am calling support, the firewall is doing something that it is not supposed to. I provided many packet captures and for some reason no one could figure anything out.

After 3 weeks of degraded service and wasting probably around 20-30 hours on this, I am not going to persue it. I will stick with the older version until a version that works properly is released.

Mike thanks for the offer and I wish that I had seen this post earlier, but I cant waste more time on this. I offered Cisco TAC all the captures that I have and the config if they want to try and troubleshoot their bugs.

mayrojas Sun, 06/26/2011 - 12:16

Ryan,

Fair enough. I am sorry you did not get the desired support, Ill check what I can do from here.

Mike

mukeshbhai Wed, 08/28/2013 - 08:11

I  have same issue, and I am using 5510 with version 8.4.(4) 9, I am trying to ftp intervlan and I get same Problem, has any one find a solution for it yet?

Actions

Login or Register to take actions

This Discussion

Posted March 4, 2011 at 10:33 AM
Stats:
Replies:35 Avg. Rating:
Views:7795 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard

Rank Username Points
1 7,861
2 6,140
3 3,165
4 1,473
5 1,446