This discussion is locked

ASK THE EXPERTS - TROUBLESHOOTING ASA, PIX AND FWSM

Unanswered Question
Jan 3rd, 2011

Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to learn how to address and troubleshoot common problems with Adaptive Security Appliances, Private Internet Exchange and Firewall Service Modules with Kureli Sankar.  Kureli is an engineer supporting Cisco's firewall team in Research Triangle Park, North Carolina. Her team supports the Cisco Adaptive Security Appliance, Firewall Services Module, Cisco Security Manager, the Content Security and Control module, and the Zone Based Firewall module in Cisco IOS Software.

Remember to use the rating system to let Kureli know if you have received an adequate response.

Kureli might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through January 14, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

I have this problem too.
1 vote
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 4.7 (3 ratings)
edoig.admin Mon, 01/03/2011 - 09:48

My group does log analyis on a large organizations PIX/ASA logs. Currently we recieve a dump of the daily logs and put them into our system for our analysis. I'm working on a script to process these logs into a CSV or other readable format for investigators. Is there a standard script or tool that can be used to process these logs? We have log analysis tools, but we want to convert them to CSV or other format so that they can be manipulated more easily by non-techies.

Thanks!

edoig.admin Tue, 01/04/2011 - 05:36

we are getting the raw dump from a syslog server and just pushing that to a linux share... no specific syslog app... I want to change it to a CSV file that has fields for dst address, dst ip, src pt, src ip, etc... but since the logs are specific on the type of message, can't do a simple script... my question was, are there any other solution for converting them to csv or other easily readable format...

if i did the kiwi method, it'd parse first few lines (date, message type) then probably dump rest into on field.

Poonguzhali Sankar Tue, 01/04/2011 - 18:29

Tim,

Pls. let me know if there are any specific syslog messages in particular that you are interested in seeing the source interface, source ip, source port, dest interface, dest ip, dest port.

Since all these messages have unique text in them it will be hard for one particular script to spit out the format as a .csv format.

Are you interested only in 302014 and 302015 and 302016 built and teardown messages?

If so you can use shell script to do what you like to do. Let me know and I shall send a sample.

-KS

edoig.admin Tue, 01/04/2011 - 20:37

The things we care most about are builds and teardowns, but for our purpose, we also care about deny's, icmp's, etc...

so i did a count of each message type for a single day and got what is pasted below... my thinking was to create a script that captured most of the data in the fields not italicized, then throw the data from the others into another field (or wear appropriate)... (if you want to talk offline, please message me.)

Count      Log Type      LogFormat
18132395     %ASA-6-302015: Built {inbound|outbound} UDP connection number for interface_name:real_address/real_port (mapped_address/mapped_port) to interface_name:real_address/real_port (mapped_address/mapped_port) [(user)]
18123239     %ASA-6-302016: Teardown UDP connection number for interface:real-address/real-port to interface:real-address/real-port duration hh:mm:ss bytes bytes [(user)]
9098811      %ASA-6-302014: Teardown TCP connection id for interface:real-address/real-port to interface:real-address/real-port duration hh:mm:ss bytes bytes [reason] [(user)]     
9097915      %ASA-6-302013: Built {inbound|outbound} TCP connection_id for interface:real-address/real-port (mapped-address/mapped-port) to interface:real-address/real-port (mapped-address/mapped-port) [(user)]     
4017138      %ASA-4-106023: Deny protocol src [interface_name:source_address/source_port] dst interface_name:dest_address/dest_port [type {string}, code {code}] by access_group acl_ID     
2646225      %ASA-6-305012: Teardown {dynamic|static} {TCP|UDP|ICMP} translation from interface_name [(acl-name)]:real_address/{real_port|real_ICMP_ID}to interface_name:mapped_address/{mapped_port|mapped_ICMP_ID} duration time     
2645583      %ASA-6-305011: Built {dynamic|static} {TCP|UDP|ICMP} translation from interface_name:real_address/real_port to interface_name:mapped_address/mapped_port     
768037      %ASA-6-302020: Built {in | out}bound ICMP connection for faddr {faddr | icmp_seq_num} gaddr {gaddr | cmp_type} laddr laddr     
767977      %ASA-6-302021: Teardown ICMP connection for faddr {faddr | icmp_seq_num} gaddr {gaddr | cmp_type} laddr laddr     
468749      %ASA-6-106015: Deny TCP (no connection) from IP_address/port to IP_address/port flags tcp_flags on interface interface_name.     
141597      %ASA-3-305006: {outbound static|identity|portmap|regular) translation creation failed for protocol src interface_name:source_address/source_port dst interface_name:dest_address/dest_port     
9917      %ASA-4-733100: Object drop rate rate_ID exceeded. Current burst rate is rate_val per second, max configured rate is rate_val; Current average rate is rate_val per second, max configured rate is rate_val; Cumulative total count is total_cnt
6095      %ASA-3-305005: No translation group found for protocol src interface_name: source_address/source_port dst interface_name: dest_address/dest_port     
1267      %ASA-6-106100: access-list acl_ID {permitted | denied | est-allowed} protocol interface_name/source_address(source_port) - interface_name/dest_address(dest_port) hit-cnt number ({first hit | number-second interval}) hash codes     
219      %ASA-6-314001: Pre-allocated RTSP UDP backconnection for src_intf:src_IP to dst_intf:dst_IP/dst_port.     
164      %ASA-5-111008: User user executed the command string     
143      %ASA-6-302010: connections in use, connections most used     
138      %ASA-4-313005: No matching connection for ICMP error message: icmp_msg_info on interface_name interface. Original IP payload: embedded_frame_info icmp_msg_info = icmp src src_interface_name:src_address dst dest_interface_name:dest_address (type icmp_type, code icmp_code) embedded_frame_info = prot src source_address/source_port dst dest_address/dest_port     
95      %ASA-6-110002: Failed to locate egress interface for protocol from src interface:src IP/src port to dest IP/dest port     
90      %ASA-6-602303: IPSEC: An direction tunnel_type SA (SPI=spi) between local_IP and remote_IP (username) has been created.     
90      %ASA-6-602304: IPSEC: An direction tunnel_type SA (SPI=spi) between local_IP and remote_IP (username) has been deleted.     
88      %ASA-6-303002: FTP connection from src_ifc:src_ip/src_port to dst_ifc:dst_ip/dst_port, user username action file filename     
59      %ASA-4-419002: Received duplicate TCP SYN from in_interface:src_address/src_port to out_interface:dest_address/dest_port with different initial sequence number.     
48      %ASA-5-713041: IKE Initiator: new or rekey Phase 1 or 2, Intf interface_number, IKE Peer IP_address local Proxy Address IP_address, remote Proxy Address IP_address, Crypto map (crypto map tag)     
45      %ASA-5-713049: Security negotiation complete for tunnel_type type (group_name) Initiator/Responder, Inbound SPI = SPI, Outbound SPI = SPI     
45      %ASA-3-713020: No Group found by matching OU(s) from ID payload: OU_value     
28      %ASA-3-313001: Denied ICMP type=number, code=code from IP_address on interface interface_name     
27      %ASA-6-611101: User authentication succeeded: Uname: user     
26      %ASA-1-709003: (Primary) Beginning configuration replication: Sending to mate.     
18      %ASA-6-113004: AAA user aaa_type Successful: server = server_IP_address, User = user     
18      %ASA-6-113008: AAA transaction status ACCEPT: user = user     
11      %ASA-6-315011: SSH session from IP_address on interface interface_name for user user disconnected by SSH server, reason: reason     
9      %ASA-5-502103: User priv level changed: Uname: user From: privilege_level To: privilege_level     
9      %ASA-5-611103: User logged out: Uname: user     
9      %ASA-6-605005: Login permitted from source-address/source-port to interface:destination/service for user “username”     
6      %ASA-5-713050: Connection terminated for peer IP_address. Reason: termination reason Remote Proxy IP_address, Local Proxy IP_address     
5      %ASA-4-313004:Denied ICMP type=icmp_type, from source_address on interface interface_name to dest_address:no matching session     
5      %ASA-5-111007: Begin configuration: IP_address reading from device.
4      %ASA-5-111001: Begin configuration: IP_address writing to device     
4      %ASA-5-111004: IP_address end configuration: {FAILED|OK}     
4      %ASA-5-111005: IP_address end configuration: OK     
4      %ASA-6-611102: User authentication failed: Uname: user     
3      %ASA-4-713903:descriptive_event_string     
3      %ASA-5-713119: PHASE 1 COMPLETED     
3      %ASA-6-713172: Automatic NAT Detection Status: Remote end is|is not behind a NAT device This end is|is not behind a NAT device     
2      %ASA-6-113005: AAA user authentication Rejected: reason = string: server = server_IP_address, User = user     
1      %ASA-6-110003: Routing failed to locate next-hop for protocol from src interface:src IP/src port to dest interface:dest IP/dest port

Poonguzhali Sankar Mon, 01/10/2011 - 05:48

I can't think of a way to use a script to separate all the fields that you are looking to separate specially all these syslogs have unique messages.

If you can grep for certain syslog messages and then try to separate the 4th column to get all the individual port, ip etc out, it might be easier.  Seems like this might be a lot of work. I am attaching the script that we came up with. Give it a shot.

-Kureli

gdspa Tue, 01/04/2011 - 07:21

We have some problems with a couple of cisco ASA 5510 with stateful failover.

This is the situation.

This host: Primary - Standby Ready
        Active time: 10790719 (sec)
        slot 0: ASA5510 hw/sw rev (1.1/8.2(1)11) status (Up Sys)
          Interface inside (10.11.5.101): Normal
          Interface management (10.12.6.247): Normal
          Interface dmz (10.249.5.2): Normal
          Interface outside (x.x.x.x): Normal
          Interface CircoloAziendale (172.30.1.2): Normal
          Interface Sindacato (172.30.2.3): Normal
          Interface vodafone (10.49.5.2): Normal
          Interface videoconferenza_SalaConsiglio (10.18.5.2): Normal
          Interface GD_guests (10.50.1.52): Normal
        slot 1: empty
Other host: Secondary - Active
        Active time: 6766056 (sec)
        slot 0: ASA5510 hw/sw rev (2.0/8.2(1)11) status (Up Sys)
          Interface inside (10.11.5.100): Normal
          Interface management (10.12.6.246): Normal
          Interface dmz (10.249.5.1): Normal
          Interface outside (x.x.x.y): Normal
          Interface CircoloAziendale (172.30.1.1): Normal
          Interface Sindacato (172.30.2.1): Normal
          Interface vodafone (10.49.5.1): Normal
          Interface videoconferenza_SalaConsiglio (10.18.5.1): Normal
          Interface GD_guests (10.50.1.51): Normal
        slot 1: empty

When I use the primary as active, I have a lot of overruns on the inside interface, withous any other errors.

Now secondary ASA is the active one and we don't have any overrun.

For both firewalls, inside interface speed is 1000Mbps.

Firewalls are different on hardware version, can it cause problems?

Poonguzhali Sankar Tue, 01/04/2011 - 07:28

gdspa,

This should not cause this problem.  Does the switch port show any errors? When the secondary unit is active, I suggest to move the Primary units inside interface to another port on the switch and see if this goes away. Compare the switch port config between these two inside interfaces and make sure they are configured exactly the same way. For 1 GB usually the recommendation is to set it to auto auto on both ends and not to specify the speed.

-Kureli

gdspa Tue, 01/04/2011 - 08:34

I forgot to write that on the switch I don't have any error on the port of the primary firewall.

Speed is configured on auto.

From Cisco docs I read, overruns are caused by too much traffic and not from cable problems. Do you confirm?

Poonguzhali Sankar Tue, 01/04/2011 - 08:53

That is correct.

An Overrun is when an incoming (ingress) packet hits the firewall's NIC, and the rx ring is full.  This is generally caused by elevated CPU, or cpu hogs or infected hosts.

An Underrun is when part of the packet is in the tx ring, and the driver starts transmitting it on the wire, but is unable to get the remaining part of the packet by the time it has finished transmitting the first part.

What doesn't add up is that this doesn't seem to be a problem when the secondary unit is active. That is the reason I suggested to look at the swtichport config for both ports to see if they are any diff.

-Kureli

Harinirina_2 Wed, 01/05/2011 - 04:08

Hello,

I would like to ask if it is possible to configure AIP-SSM for redundancy.

We have 2 ASA with AIP-SSM each. the ASA is configured for failover. What should be the configuration of the AIP-SSM so that it can work for failover.

Poonguzhali Sankar Wed, 01/05/2011 - 06:39

Harinirina,

As far as the SSM module is concerned there is no particular failover config for that.  If the module in one ASA fails then that ASA is considered less healthy and it will failover to the other unit and the SSM module in the other unit will do all the scanning per the configuration.

You can read about the failover guidelines here: http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/csc.html#wp1107307

Failover Guidelines

Does not support sessions in Stateful Failover. The CSC SSM does not  maintain connection information, and therefore cannot provide the  failover unit with the required information. The connections that a CSC  SSM is scanning are dropped when the adaptive security appliance in  which the CSC SSM is installed fails. When the standby adaptive security  appliance becomes active, it forwards the scanned traffic to the CSC  SSM and the connections are reset.

-Kureli

jvardhan29 Wed, 01/05/2011 - 06:19

Hi

Please explain the input and output in the ASA for QOS policing . i have never been able to understand this and usually end up in making this work by trial and error .i have gone thru the below cisco guide but thought that this platform is good to get answer from experts

http://www.cisco.com/en/US/docs/security/asa/asa72/configuration/guide/mpc.html#wp1116522

also if u can tell what IP Address to use (private or public) while applying an ACL to a class-map (using ASA 8.2 ).

take an example if i want to police the user for a specific website downloads (traffic going from inside to outside) where and how police will be applied .i.e i just want to restrict the downloads but not the HTTP site .now the confusion is that download is also part of browsing that HTTP page,  so how will ASA determine what to police ?

Poonguzhali Sankar Wed, 01/05/2011 - 09:45

The input keyword enables policing of traffic flowing in the input direction.

The output keyword enables policing of traffic flowing in the output direction.

You can refer this link (step 3): http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_qos.html#wp1071334

IP address private or public depends on the which interface policing is applied.

If the flow is going to initiated from the inside and if the policy is applied globally then the initial packet of the flow will hit the inside interface so, private IP address.

If the same policy is applied on the outside interface then you need to use the translated address. Simply know what the ip address will look like on each of the interfaces and combine that with the policy that is applied on that particular interface.

http download is on port 80 so everything that happens on port 80 will be policed if configured to do so.

Pls. refer this QoS link: https://supportforums.cisco.com/docs/DOC-1230

Keep this show command handy

show service-policy flow tcp host 192.168.1.1 host 209.85.149.99 eq 80

This will tell you all the inspections and policing that the inside host 192.168.1.1 has to go through when going to load google.com

Simple rules for QoS.

1. Police on the egress interface close to the source.

2. Policing input may not work because the traffic simply arrives on the interface and we have no control over it.

3. Always apply policing on the outside interface because that is the one with the bottleneck. The reason is because inside interface is hooked up to 100MB or Gig speed by default.

4. Apply QoS using a separate policy-map and apply it to the specific outside interface.

-Kureli

jvardhan29 Thu, 01/13/2011 - 04:29

kureli,

thanks for answering ths.

1) how can we control or limit the amount of bandwidth for a single host in both inbound and outbound direction . also if the same IP Address is getting PATTED to the external interface IP , will ASA assume the other (entire range) inside hosts (getting PATTED to that) as well for policing .if yes , do we need to apply a seperate static for the single host?

2) also if the traffic is incoming to the ASA to a public FTP server hosted inside and we want that outside users should not exceed a particular limit and apply the policing then in which direction and on which interface we should do that ?(considering that we may have active or passive ftp clients so there might be a scenario where the FTP control channel is from outside but data channel frm inside to outside)

cchughes Thu, 01/06/2011 - 06:44

Hello,

I am having an issue involving a Cisco ASA that has an IPSec tunnel to a Fortigate firewall.  In brief, the issue is that P1 establishes and most of the P2 SA's establish but at least 2 subnet pairs defined in the crypto map ACL will not form an SA.  The destination for the P2 SA is a DMZ based subnet.  Other SA's for the DMZ subnet work, just not the ones that originate from the 192.168.13.0/24 subnet.

I have troubleshot on the Fortigate and I can see the packets get encrypted and placed in the tunnel.  On the ASA all I see in the log for the packets is:

Jan 05 2011 23:56:29: %ASA-7-609001: Built local-host outside:192.168.13.1

Jan 05 2011 23:56:29: %ASA-7-609002: Teardown local-host outside:192.168.13.1 duration 0:00:00

I have run "debug ipsec 200 "  and while the traffic for the subnet pair is generated I see no attempt to negotiate an SA.  I've reviewed theACL for the crypto map on both devices to validate that the subnet and mask are identical.  Other subnet pairs are working fine on the same P1 SA.

I wanted to troubleshoot this further so I tried a packet capture but no packets are displayed.  I'm looking for other troubleshooting steps to perform in order to find the problem.  Any suggestions?

Thanks in advance

Message was edited by: cchughes  Added that the destination for the SA is a dmz on the ASA.

kathy-kat Thu, 01/06/2011 - 11:06

Hello Kureli!!

I have some problems when I tried to access an ASA through SSH, I can active this protocol but the version 1 because the client does not have the licence VPN-3DES-AES , if I try to access at the device the session is closed and appears a message like unattainable.

I deleted the old key and generate another one and make the configuration again, but the problem does not fix it.

Here is a debug of conections´s ssh:

SA-Firewall# Device ssh opened successfully.
SSH0: SSH client: IP = '172.17.200.32'  interface # = 2
SSH: host key initialised
SSH0: starting SSH control process
SSH0: Exchanging versions - SSH-1.5-Cisco-1.25

SSH0: send SSH message: outdata is NULL

server version string:SSH-1.5-Cisco-1.25SSH0: receive SSH message: 83 (83)
SSH0: client version is - SSH-1.5-TTSSH/2.49 Win32

client version string:SSH-1.5-TTSSH/2.49 Win32SSH0: begin server key generation
SSH0: complete server key generation, elapsed time = 720 ms
SSH0: declare what cipher(s) we support:
00  0x00  0x00  0x04  0xSSH0: send SSH message: SSH_SMSG_PUBLIC_KEY (2)
SSH0: SSH_SMSG_PUBLIC_KEY message sent
SSH0: receive SSH message: SSH_MSG_DISCONNECT (1)
SSH0: invalid SSH_CMSG_SESSION_KEY msg - msg type 0x01, length 270
SSH0: Session disconnected by SSH server - error 0x01 "Invalid message type"

Any idea?

Regards,

Kathy

Poonguzhali Sankar Thu, 01/06/2011 - 13:06

Kathy,

How are you? VPN-3DES-AES license is actually free.

You simply have to go to cisco.com/go/license

please click                          here for available licenses.

Cisco ASA 3DES/AES License

Can you try that and let me know if ssh works for you with 3DES?

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/access_management.html#wp1042023

Let me look up these debug messages and see what might cause this.

-Kureli

kathy-kat Fri, 01/07/2011 - 06:08

Thanks Kureli!!!

Let me try!!!

Kathy

Poonguzhali Sankar Mon, 01/10/2011 - 08:42

Apply the activation key that you receive with the command activation-key and copy and paste the 4-tuple or 5-tuple key.

conf t

activation-key

wri mem

exit

-KS

rkalia1 Thu, 01/06/2011 - 11:49

Hi Kureli,

I have come across strange issue with ASA failover.  The ASA software version does not matter whether 7.2x or 8.x.  The issue is that if there is an ASA failover pair at one site having a tunnel to a remote site (ASA or cisco router) sometimes the Phase II stalls.  The data does not seem to pass through the ASA failover pair end though Phase I is up and hence tunnel shows up.  I have seen this at altogether different networks for different companies.  Same thing I came across on PIX failover pair too.  The fix, however, is either rebooting the primary ASA or failing over.  Also, I have tried upgrading a couple of ASA pairs to no effect.  Sometimes it so happens that only one particular subnet (in interesting traffic) stops working.  Can you please help explain this issue and suggest a fix?  Please note that when the issue occurs I try everything from clearing the Phase I/II on both ends to rebooting the remote ASA/Router.  But things start working only after failover pair at headend is failed over or the active ASA rebooted.

thanks

Poonguzhali Sankar Thu, 01/06/2011 - 15:07

Hello,

Could you pls. verify if you might have overlapping addreses (dest addresses)  in the crypto acl between diff. cypto maps?

-Kureli

rkalia1 Thu, 01/06/2011 - 15:38

No there are no overlapping subnets.  When I say different companies and different networks it means IPSec VPN from Company A to Company B and Company C to Company D.  We are managed services company and manage variety of networks.  I work extensively on VPNs on PIX/ASA and have advanced knowledge of IPSec VPNs.  This issue keeps haunting me on different networks wherever I have PIX/ASA failover pairs.  Phase II stops working (works only from remote to headend ASA pair but not in other direction).  No matter what you do (clear Phase I on both end devices or reboot remote device) the only fix is to failover the ASA or reboot the Active.  Usually I do not see any error in logs but luckily this time I saw the following on one customers' ASA pair :

   

f1fc0)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

%ASA-5-713137: Group = 142.166.121.254, IP = 142.166.121.254, Reaper overriding refCnt [0] and tunnelCnt [0] -- deleting SA!

%ASA-3-713232: Group = 142.166.121.254, IP = 142.166.121.254, SA lock refCnt = 0, bitmask = 00000000, p1_decrypt_cb = 0, qm_decrypt_cb = 0, qm_hash_cb = 0, qm_spi_ok_cb = 0, qm_dh_cb = 0, qm_secret_key_cb = 0, qm_encrypt_cb = 0

%ASA-7-715065: Group = 142.166.121.254, IP = 142.166.121.254, IKE MM Initiator FSM error history (struct &0x941d4c8)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

%ASA-5-713137: Group = 142.166.121.254, IP = 142.166.121.254, Reaper overriding refCnt [0] and tunnelCnt [0] -- deleting SA!

%ASA-3-713232: Group = 142.166.121.254, IP = 142.166.121.254, SA lock refCnt = 0, bitmask = 00000000, p1_decrypt_cb = 0, qm_decrypt_cb = 0, qm_hash_cb = 0, qm_spi_ok_cb = 0, qm_dh_cb = 0, qm_secret_key_cb = 0, qm_encrypt_cb = 0

%ASA-7-715065: Group = 142.166.121.254, IP = 142.166.121.254, IKE MM Initiator FSM error history (struct &0x94c01d0)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

%ASA-5-713137: Group = 142.166.121.254, IP = 142.166.121.254, Reaper overriding refCnt [0] and tunnelCnt [0] -- deleting SA!

%ASA-3-713232: Group = 142.166.121.254, IP = 142.166.121.254, SA lock refCnt = 0, bitmask = 00000000, p1_decrypt_cb = 0, qm_decrypt_cb = 0, qm_hash_cb = 0, qm_spi_ok_cb = 0, qm_dh_cb = 0, qm_secret_key_cb = 0, qm_encrypt_cb = 0

%ASA-7-715065: Group = 142.166.121.254, IP = 142.166.121.254, IKE MM Initiator FSM error history (struct &0x956b2c8)  , :  MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE-->MM_FREE, NullEvent-->MM_FREE, EV_TERMINATE

The ASA 5510 pair in question runs ver 7.2(1).  Both ASAs are identical in hardware too.  Cisco says this is a bug for 8.0.2.  Exact Cisco words are:

Seen on 8.0.2 Active in FO pair but suspected to have been corrupted while device was Standby.

In the above scenario I have ASA 5510 (running 7.2(1)) and Cisco 1811 router on the other end running 12.3 IOS.

My question is that if this is a bug then it should have been taken care of in versions subsequent to 7.2.  Why Cisco says that it is taken care of after 8.0.2 version?  Other thing is that even if  I upgrade I do not think the issue will go away since I have done that for 2 other customers already and the issue still happens - PIX or ASA does not matter.  I have started to believe that Cisco's failover has an issue which either went undetected or has not been resolved.  We had once (a couple of years back) raised this issue with Cisco TAC too and were advised to upgrade the PIX pair which was done.  But that did not resolve the issue.  It still happens.

However, for this case I believe I will have to advise the customer for an upgrade because the error message matches with the bug though the bug is believed to be reported for 8.0.2.  I do not seem to have any choice there.  If you have any suggestions they are welcome. 

thanks

Poonguzhali Sankar Sun, 01/09/2011 - 12:54

Check these two defects.

CSCtd36473    IPsec: Outbound context may be deleted prematurely

CSCtb53186    Duplicate ASP crypto table entry causes firewall to not encrypt traffic

http://tools.cisco.com/Support/BugToolKit/
you can go to the above link login with your CCO ID and then key in this defect ID

Pls. provide the case number if you have them.

-KS

cchughes Thu, 01/06/2011 - 21:33

Kureli,

You may have missed my post.  earlier just before Kathy.  Can you suggest next steps for me?

Poonguzhali Sankar Fri, 01/07/2011 - 05:15

cchuges,

Sorry about that. I discussed this with our vpn specialist, we would have to gather debugs for isakmp and ipsec. Basically bring the tunnel down completely and then bring it up while the debug is enabled. It would be a good idea to open a case with our VPN team.

-Kureli

cchughes Fri, 01/07/2011 - 05:32

well thats the problem. i ran the debugs at the 200 level and see nothing. the log messages i included in my original post are the only indication i have that i am receiving packets. is there any other way to see what packets are coming in?

Chris Hughes

Layer8 Consulting

Chughes@l8c.com

(240)460-7283

cchughes Fri, 01/07/2011 - 06:01

BTW, the debugs I used are the basic :

Deb cry isa 200

Deb cry ips 200

Are there other debugs I can use? These work and show other tunnel activity but nothing for the subnet pairs in question. Its like my acl is wrong but I have checked it over and over and cannot find a problem with the way they are setup. Because I have a Fortigate fw on the remote end I made sure I checked the way it defines the sa.

Question: does the order of the subnet pairs in the acl need to be the same on both endpoints?

Poonguzhali Sankar Fri, 01/07/2011 - 07:19

cchuges,

The crypto acl on one side is usually a mirror image of the other.

debug cry isa

debug cry ips

(level 1) should be good engough. Once done you should start some interesting traffic by pinging from one side to the other.

If the debugs don't even show anything then, it looks like crypto isakmp is not even enabled on the interface.

Could you pls. post the output of

"sh run crypto" from the firewall

-KS

cchughes Fri, 01/07/2011 - 07:42

while the acls include all the same subnet pairs, they are not lsted in the fortigate config in the same order. i dont think that is an issue but thought i'd ask.

generating interesting traffic yields no output no matter what debug level i use up to 200.

crypto isakmp is enabled and other sa's are establishing.

i will send the output you requested. is there a way to capture the encrypted interesting traffic and view the headers to look for corruption and such?

Chris Hughes

Layer8 Consulting

Chughes@l8c.com

(240)460-7283

Poonguzhali Sankar Fri, 01/07/2011 - 09:10

This interesting traffic is listed in the nat 0 acl correct?

The oder of the access-list should not matter. More and more it looks like of us should really ge on the box and look at it.

Pls. do open a case and let me know the case number. You can open a case here: https://www.cisco.com/tac

If you captured on the outside interface you will only see esp and udp 500 packets so, that may not help.

You can capture on the inside interface and see if you do see clear traffic arriving.

-KS

jsluzewski Fri, 01/07/2011 - 07:27

Is it possible to NAT multiple source addresses to a single IP using policy NAT?

Will the following config translate any 10.x.x.x address to 172.16.1.250 while accessing the /24 ?

access-list policy-nat-acl extended permit 10.0.0.0 255.0.0.0 152.220.108.0 255.255.255.0

static (inside,outside) 172.16.1.250 access-list policy-nat-acl

Thank you,

Jarek

Poonguzhali Sankar Fri, 01/07/2011 - 07:53

No. You will get an error message that will talk about mask being inconsistent with the global address.

You can do dynamic policy nat

access-list policy-nat-acl extended permit ip 10.0.0.0 255.0.0.0 152.220.108.0 255.255.255.0

nat (inside) 100 access-l policy-nat-acl

global (outside) 100 172.16.1.250

-KS

dianewalker Fri, 01/07/2011 - 13:09

Kureli,

Welcome back!!!  You did a tutorial on Troubleshooting Common Firewall Problems in July 2010.  I learned a lot from this tutorial.  Not everyone can teach or explain firewall in basic terms.  You did very well on this tutorial.   Have you done more tutorials since July 2010 or do you plan to do more tutorials in the future?  Do you have any recommendations on learning the basics on ASA VPN/firewall?

Thanks.

Diane

Poonguzhali Sankar Fri, 01/07/2011 - 13:43

Diane,

I do remember you! Glad to hear that you learned a lot from my webcast.  I haven't  done another one since then.  May be it is time now.

Let me know if you have any questions that I can answer.

I just posted this blog today that you can read: https://supportforums.cisco.com/community/netpro/security/firewall/blog/2011/01/07/asa-pix-dhcp-relay-through-vpn-tunnel

Let me know what you think.

Hmm...ASA/VPN basics....Best thing to do is to purchase an asa5505 and some small routers and try out different topologies. Does your job involve maintaining a network? Take some classes. I can send you some information regarding that.

There is no place (that I know of) better than Cisco TAC to learn! We learn something new every single day!

-KS

dianewalker Mon, 01/10/2011 - 09:42

Thanks Kureli.  Another great, helpful documents.  I hope to see more of these or web casts in the future. Thanks again.

Diane

mkashifashraf Sat, 01/08/2011 - 15:01

Dear Experts,

I have ASA5520, Configured Subinterface on inside for different VLANS with same security level. But i'm not able to communicate with same security level subinterface with VLAN1. I can communicate between other subinterfaces (with same security level) and different VLANS.

Waiting for your expert recommendation ASAP.

Regards,

rkalia1 Sat, 01/08/2011 - 15:06

Configure same-security-traffic intra-interface also

Sent from my iPhone

mkashifashraf Sat, 01/08/2011 - 15:14

Dear,

I already configured both same-security-traffic intra-interface & Inter.

I have one Catalyst 3560 with different VLANS. Configured one port as TRUNK which is connected to ASA for INSIDE. We configured interface Vlans for different Vlans. All Other Vlans can communicate with ASA Subinterfaces except VLAN1 & our native Vlan also Vlan 1.

Regards,

Poonguzhali Sankar Sun, 01/09/2011 - 06:30

I have suggested a few options in thread: https://supportforums.cisco.com/message/3265079#3265079

This is not recommended practice and that is the reason I didn't suggest this. Many people do configure it and it does work.

According to our documentation: http://www.cisco.com/en/US/docs/security/asa/asa72/configuration/guide/intrface.html#wp1044006

Note: If   you use subinterfaces, you typically do not also want the physical   interface to pass traffic, because the physical interface passes   untagged packets. Because the physical interface must be enabled for  the  subinterface to pass traffic, ensure that the physical interface  does  not pass traffic by leaving out the nameif command. If you want to let the physical interface pass untagged packets, you can configure the nameif command as usual.

So, pls. configure the main interface wtih an IP address in vlan 1 and security level and remove the sub-interface that you configured for vlan1.

-Kureli

cchughes Sat, 01/08/2011 - 15:10

Are you using ASDM? If so theres a check box on the interface configuration screen to enable traffic between two interfaces with the same security level. Or you can use the command "same-security-traffic permit inter-interface"

mkashifashraf Sat, 01/08/2011 - 15:25

Dears,

I didn't configured any Access-list, Routing and NAT for same security level INSIDE Subinterfaces.and My all Vlans can communicate except VLAN1.

Regards,

rkalia1 Sat, 01/08/2011 - 15:27

Can you post ASA config pls?

Sent from my iPhone

mkashifashraf Sun, 01/09/2011 - 10:28

Dear Experts,

after removing sub-interface for vlan1 and using physical interface for that. My vlan1 is working. But still my intervlan communication is not working.

I already used same security level inter & intra both.

please reply soon, it's critical for me.

Regards,

Poonguzhali Sankar Sun, 01/09/2011 - 10:52

Ashraf,

If this is very critical, I'd suggest opening a TAC case.

Now that vlan1 is working and all these interfaces have the same security level we need to look at

permission and translation to see if they are configured correctly.

What traffic is breaking now?

source vlan source IP

destination vlan desatination IP

Watch what the logs say.

conf t

logging on

logging buffered 7

exit

sh logg | i x.x.x.x where x.x.x.x. is the host in question.

-Kureli

mkashifashraf Sun, 01/09/2011 - 15:19

interface gi 0/1 (VLAN1 & NATIVE VLAN)

nameif inside

security-level 100

ip add 192.168.0.x 255.255.255.0

no shut

interface gi 0/1.20

vlan 20

nameif inside20

security-level 100

ip add 192.168.20.x 255.255.255.0

no sh

interface gi 0/1.30

vlan 30

nameif inside30

security-level 100

ip add 192.168.30.x 255.255.255.0

no sh    

Can you send me configuration example for permission and Translation with Same security level on sub-interfaces.

I already opened a TAC case, but your response is fast and accurate than TAC on this case.

Waiting for your earliest response.

Regards,

Poonguzhali Sankar Sun, 01/09/2011 - 16:02

Do you have nat control enabled or not? Do you have any translation configured on this ASA?

If you use "no nat-control" then, all you need to do is restrict who can access what via acl.

If you do have nat-control enabled then you need to provide translation.

How about nat 0 with acl? Depending on which interface got created first you may need just one set or all of them.

nat (inside) 0 access-list inside-to-vlans

access-list inside-to-vlans permit ip 192.168.0.0 255.255.255.0 92.168.20.0 255.255.255.0

access-list inside-to-vlans permit ip 192.168.0.0 255.255.255.0 92.168.30.0 255.255.255.0

** Test with just the above, if you have trouble and see some no translation messages then add all the ones below **

nat (inside20) 0 access-list 20-inside-30

access-list 20-inside-30 permit ip 192.168.20.0 255.255.255.0 192.168.0.0 255.255.255.0

access-list 20-inside-30 permit ip 192.168.20.0 255.255.255.0 192.168.30.0 255.255.255.0

nat (inside30) 0 access-list 30-20-inside

access-list  30-20-inside per ip 192.168.30.0 255.255.255.0 92.168.0.0 255.255.255.0

access-list  30-20-inside per ip 192.168.30.0 255.255.255.0 92.168.20.0 255.255.255.0

Make sure you have permission allowed in the acl applied on the interfaces (all three of them).

Give it a shot. Problems like this can be solved quickly once TAC has access to the device.

-KS

mkashifashraf Sun, 01/09/2011 - 23:13

As I understand, Permission for all INSIDE (different VLANs) should be outbound. Please correct me, If i'm wrong.

Regards,

mkashifashraf Mon, 01/10/2011 - 00:57

NAT FOR SAME SECURITY LEVEL

access-list inside1-inside10 extended permit ip 192.168.0.0 255.255.255.0 192.168.10.0 255.255.255.0
access-list inside10-inside1 extended permit ip 192.168.10.0 255.255.255.0 192.168.0.0 255.255.255.0

nat (inside1) 0 access-list inside1-inside10
nat (inside10) 0 access-list inside10-inside1

PERMISSION FOR SAME SECURITY LEVEL

access-list 100 extended permit ip any any

access-group 100 out interface inside1
access-group 100 out interface inside10

i'm pinging from host reside inside1---to---inside10

source VLAN1 source IP ADD = 192.168.0.244 ------>destination vlan10 destionation ip add 192.168.10.4 (ASA's sub-interface IP)

RESULT is below..........!

%ASA-6-302020: Built inbound ICMP connection for faddr 192.168.0.244/1 gaddr 192.168.10.4/0 laddr 192.168.10.4/0

%ASA-6-110003: Routing failed to locate next hop for icmp from inside10:192.168.10.4/0 to inside10:192.168.0.244/0

ICMP echo request from 192.168.0.244 to 192.168.10.4 ID=1 seq=5 len=32

%ASA-6-302021: Teardown ICMP connection for faddr 192.168.0.244/1 gaddr 192.168.10.4/0 laddr 192.168.10.4/0

Please reply soon......!

Regards,

Actions

Login or Register to take actions

This Discussion

Posted January 3, 2011 at 8:03 AM
Stats:
Replies:97 Avg. Rating:4.66667
Views:36557 Votes:1
Shares:0

Related Content

Discussions Leaderboard

Rank Username Points
1 7,861
2 6,140
3 3,165
4 1,473
5 1,446