cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
47566
Views
14
Helpful
97
Replies

ASK THE EXPERTS - TROUBLESHOOTING ASA, PIX AND FWSM

ciscomoderator
Community Manager
Community Manager

Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to learn how to address and troubleshoot common problems with Adaptive Security Appliances, Private Internet Exchange and Firewall Service Modules with Kureli Sankar.  Kureli is an engineer supporting Cisco's firewall team in Research Triangle Park, North Carolina. Her team supports the Cisco Adaptive Security Appliance, Firewall Services Module, Cisco Security Manager, the Content Security and Control module, and the Zone Based Firewall module in Cisco IOS Software.

Remember to use the rating system to let Kureli know if you have received an adequate response.

Kureli might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through January 14, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

97 Replies 97

The input keyword enables policing of traffic flowing in the input direction.

The output keyword enables policing of traffic flowing in the output direction.

You can refer this link (step 3): http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_qos.html#wp1071334

IP address private or public depends on the which interface policing is applied.

If the flow is going to initiated from the inside and if the policy is applied globally then the initial packet of the flow will hit the inside interface so, private IP address.

If the same policy is applied on the outside interface then you need to use the translated address. Simply know what the ip address will look like on each of the interfaces and combine that with the policy that is applied on that particular interface.

http download is on port 80 so everything that happens on port 80 will be policed if configured to do so.

Pls. refer this QoS link: https://supportforums.cisco.com/docs/DOC-1230

Keep this show command handy

show service-policy flow tcp host 192.168.1.1 host 209.85.149.99 eq 80

This will tell you all the inspections and policing that the inside host 192.168.1.1 has to go through when going to load google.com

Simple rules for QoS.

1. Police on the egress interface close to the source.

2. Policing input may not work because the traffic simply arrives on the interface and we have no control over it.

3. Always apply policing on the outside interface because that is the one with the bottleneck. The reason is because inside interface is hooked up to 100MB or Gig speed by default.

4. Apply QoS using a separate policy-map and apply it to the specific outside interface.

-Kureli

kureli,

thanks for answering ths.

1) how can we control or limit the amount of bandwidth for a single host in both inbound and outbound direction . also if the same IP Address is getting PATTED to the external interface IP , will ASA assume the other (entire range) inside hosts (getting PATTED to that) as well for policing .if yes , do we need to apply a seperate static for the single host?

2) also if the traffic is incoming to the ASA to a public FTP server hosted inside and we want that outside users should not exceed a particular limit and apply the policing then in which direction and on which interface we should do that ?(considering that we may have active or passive ftp clients so there might be a scenario where the FTP control channel is from outside but data channel frm inside to outside)

cchughes
Level 1
Level 1

Hello,

I am having an issue involving a Cisco ASA that has an IPSec tunnel to a Fortigate firewall.  In brief, the issue is that P1 establishes and most of the P2 SA's establish but at least 2 subnet pairs defined in the crypto map ACL will not form an SA.  The destination for the P2 SA is a DMZ based subnet.  Other SA's for the DMZ subnet work, just not the ones that originate from the 192.168.13.0/24 subnet.

I have troubleshot on the Fortigate and I can see the packets get encrypted and placed in the tunnel.  On the ASA all I see in the log for the packets is:

Jan 05 2011 23:56:29: %ASA-7-609001: Built local-host outside:192.168.13.1

Jan 05 2011 23:56:29: %ASA-7-609002: Teardown local-host outside:192.168.13.1 duration 0:00:00

I have run "debug ipsec 200 "  and while the traffic for the subnet pair is generated I see no attempt to negotiate an SA.  I've reviewed theACL for the crypto map on both devices to validate that the subnet and mask are identical.  Other subnet pairs are working fine on the same P1 SA.

I wanted to troubleshoot this further so I tried a packet capture but no packets are displayed.  I'm looking for other troubleshooting steps to perform in order to find the problem.  Any suggestions?

Thanks in advance

Message was edited by: cchughes  Added that the destination for the SA is a dmz on the ASA.

kathy-kat
Level 1
Level 1

Hello Kureli!!

I have some problems when I tried to access an ASA through SSH, I can active this protocol but the version 1 because the client does not have the licence VPN-3DES-AES , if I try to access at the device the session is closed and appears a message like unattainable.

I deleted the old key and generate another one and make the configuration again, but the problem does not fix it.

Here is a debug of conections´s ssh:

SA-Firewall# Device ssh opened successfully.
SSH0: SSH client: IP = '172.17.200.32'  interface # = 2
SSH: host key initialised
SSH0: starting SSH control process
SSH0: Exchanging versions - SSH-1.5-Cisco-1.25

SSH0: send SSH message: outdata is NULL

server version string:SSH-1.5-Cisco-1.25SSH0: receive SSH message: 83 (83)
SSH0: client version is - SSH-1.5-TTSSH/2.49 Win32

client version string:SSH-1.5-TTSSH/2.49 Win32SSH0: begin server key generation
SSH0: complete server key generation, elapsed time = 720 ms
SSH0: declare what cipher(s) we support:
00  0x00  0x00  0x04  0xSSH0: send SSH message: SSH_SMSG_PUBLIC_KEY (2)
SSH0: SSH_SMSG_PUBLIC_KEY message sent
SSH0: receive SSH message: SSH_MSG_DISCONNECT (1)
SSH0: invalid SSH_CMSG_SESSION_KEY msg - msg type 0x01, length 270
SSH0: Session disconnected by SSH server - error 0x01 "Invalid message type"

Any idea?

Regards,

Kathy

Kathy,

How are you? VPN-3DES-AES license is actually free.

You simply have to go to cisco.com/go/license

please click                          here for available licenses.

Cisco ASA 3DES/AES License

Can you try that and let me know if ssh works for you with 3DES?

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/access_management.html#wp1042023

Let me look up these debug messages and see what might cause this.

-Kureli

Thanks Kureli!!!

Let me try!!!

Kathy

Apply the activation key that you receive with the command activation-key and copy and paste the 4-tuple or 5-tuple key.

conf t

activation-key

wri mem

exit

-KS

rkalia1
Level 1
Level 1

Hi Kureli,

I have come across strange issue with ASA failover.  The ASA software version does not matter whether 7.2x or 8.x.  The issue is that if there is an ASA failover pair at one site having a tunnel to a remote site (ASA or cisco router) sometimes the Phase II stalls.  The data does not seem to pass through the ASA failover pair end though Phase I is up and hence tunnel shows up.  I have seen this at altogether different networks for different companies.  Same thing I came across on PIX failover pair too.  The fix, however, is either rebooting the primary ASA or failing over.  Also, I have tried upgrading a couple of ASA pairs to no effect.  Sometimes it so happens that only one particular subnet (in interesting traffic) stops working.  Can you please help explain this issue and suggest a fix?  Please note that when the issue occurs I try everything from clearing the Phase I/II on both ends to rebooting the remote ASA/Router.  But things start working only after failover pair at headend is failed over or the active ASA rebooted.

thanks

Hello,

Could you pls. verify if you might have overlapping addreses (dest addresses)  in the crypto acl between diff. cypto maps?

-Kureli

No there are no overlapping subnets.  When I say different companies and different networks it means IPSec VPN from Company A to Company B and Company C to Company D.  We are managed services company and manage variety of networks.  I work extensively on VPNs on PIX/ASA and have advanced knowledge of IPSec VPNs.  This issue keeps haunting me on different networks wherever I have PIX/ASA failover pairs.  Phase II stops working (works only from remote to headend ASA pair but not in other direction).  No matter what you do (clear Phase I on both end devices or reboot remote device) the only fix is to failover the ASA or reboot the Active.  Usually I do not see any error in logs but luckily this time I saw the following on one customers' ASA pair :

   

f1fc0)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

%ASA-5-713137: Group = 142.166.121.254, IP = 142.166.121.254, Reaper overriding refCnt [0] and tunnelCnt [0] -- deleting SA!

%ASA-3-713232: Group = 142.166.121.254, IP = 142.166.121.254, SA lock refCnt = 0, bitmask = 00000000, p1_decrypt_cb = 0, qm_decrypt_cb = 0, qm_hash_cb = 0, qm_spi_ok_cb = 0, qm_dh_cb = 0, qm_secret_key_cb = 0, qm_encrypt_cb = 0

%ASA-7-715065: Group = 142.166.121.254, IP = 142.166.121.254, IKE MM Initiator FSM error history (struct &0x941d4c8)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

%ASA-5-713137: Group = 142.166.121.254, IP = 142.166.121.254, Reaper overriding refCnt [0] and tunnelCnt [0] -- deleting SA!

%ASA-3-713232: Group = 142.166.121.254, IP = 142.166.121.254, SA lock refCnt = 0, bitmask = 00000000, p1_decrypt_cb = 0, qm_decrypt_cb = 0, qm_hash_cb = 0, qm_spi_ok_cb = 0, qm_dh_cb = 0, qm_secret_key_cb = 0, qm_encrypt_cb = 0

%ASA-7-715065: Group = 142.166.121.254, IP = 142.166.121.254, IKE MM Initiator FSM error history (struct &0x94c01d0)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

%ASA-5-713137: Group = 142.166.121.254, IP = 142.166.121.254, Reaper overriding refCnt [0] and tunnelCnt [0] -- deleting SA!

%ASA-3-713232: Group = 142.166.121.254, IP = 142.166.121.254, SA lock refCnt = 0, bitmask = 00000000, p1_decrypt_cb = 0, qm_decrypt_cb = 0, qm_hash_cb = 0, qm_spi_ok_cb = 0, qm_dh_cb = 0, qm_secret_key_cb = 0, qm_encrypt_cb = 0

%ASA-7-715065: Group = 142.166.121.254, IP = 142.166.121.254, IKE MM Initiator FSM error history (struct &0x956b2c8)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

The ASA 5510 pair in question runs ver 7.2(1).  Both ASAs are identical in hardware too.  Cisco says this is a bug for 8.0.2.  Exact Cisco words are:

Seen on 8.0.2 Active in FO pair but suspected to have been corrupted while device was Standby.

In the above scenario I have ASA 5510 (running 7.2(1)) and Cisco 1811 router on the other end running 12.3 IOS.

My question is that if this is a bug then it should have been taken care of in versions subsequent to 7.2.  Why Cisco says that it is taken care of after 8.0.2 version?  Other thing is that even if  I upgrade I do not think the issue will go away since I have done that for 2 other customers already and the issue still happens - PIX or ASA does not matter.  I have started to believe that Cisco's failover has an issue which either went undetected or has not been resolved.  We had once (a couple of years back) raised this issue with Cisco TAC too and were advised to upgrade the PIX pair which was done.  But that did not resolve the issue.  It still happens.

However, for this case I believe I will have to advise the customer for an upgrade because the error message matches with the bug though the bug is believed to be reported for 8.0.2.  I do not seem to have any choice there.  If you have any suggestions they are welcome. 

thanks

Check these two defects.

CSCtd36473    IPsec: Outbound context may be deleted prematurely

CSCtb53186    Duplicate ASP crypto table entry causes firewall to not encrypt traffic

http://tools.cisco.com/Support/BugToolKit/
you can go to the above link login with your CCO ID and then key in this defect ID

Pls. provide the case number if you have them.

-KS

cchughes
Level 1
Level 1

Kureli,

You may have missed my post.  earlier just before Kathy.  Can you suggest next steps for me?

cchuges,

Sorry about that. I discussed this with our vpn specialist, we would have to gather debugs for isakmp and ipsec. Basically bring the tunnel down completely and then bring it up while the debug is enabled. It would be a good idea to open a case with our VPN team.

-Kureli

well thats the problem. i ran the debugs at the 200 level and see nothing. the log messages i included in my original post are the only indication i have that i am receiving packets. is there any other way to see what packets are coming in?

Chris Hughes

Layer8 Consulting

Chughes@l8c.com

(240)460-7283

BTW, the debugs I used are the basic :

Deb cry isa 200

Deb cry ips 200

Are there other debugs I can use? These work and show other tunnel activity but nothing for the subnet pairs in question. Its like my acl is wrong but I have checked it over and over and cannot find a problem with the way they are setup. Because I have a Fortigate fw on the remote end I made sure I checked the way it defines the sa.

Question: does the order of the subnet pairs in the acl need to be the same on both endpoints?

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: