CSCuf46296 - Unable to add static NAT/PAT after upgrade to 8.4.5
Let me start by saying that the BugID mentioned in the topic is not with 100% certainty the bug we are facing but it was the only one that I could find even remotely matching our situation.
Let me give some background on what has happened so far (some thats not related directly to this problem)
We had 2x ASA5585-X SSP20 running 8.4(1)9 and 8.4(2) in Multiple Context mode (with around combined Security Context amount of 300) and the latter one started expiriencing random reboots
We opened a TAC Case and the latest Interim realease of 8.4(7)3 was suggested
On the first attempt directly to the 8.4(7)3 failed due to the ASA not even accepting to load the image file to the Flash (related to some known bug)
We upgarded the ASA to 8.4(6) and jumped to 8.4(7)3 to overcome the previously mentioned bug. The ASA became nearly unusable and we had to revert back to 8.4(6) where we noticed that the change related to ARP (ARP for non connected networks) broke some of our device monitoring. The ASA also seemed to be unable to classify packets to the correct Security Context even though a Static NAT was configured for the monitored host (which according to documentation should satisfy the requirements to classify the packet correctly)
We had to revert back to the 8.4(2).
Before the next update to 8.4(6) we added the following configurations to correct the problems
arp permit-nonconnected (System Context)
mac-address auto (Systen Context)
Since that upgrade the ASA in question had been running several weeks on the 8.4(6) software
Last night we also upgraded the ASA running 8.4(1)9 to the same 8.4(6) software
We had a small customer migration set up for this morning on the other device (not the one upgraded last night). The migration included changing the local address of a couple of Section 2 Auto NAT Static NAT translations
As one of our people changed the "host <local ip>" address setting under the "object network" the first change was accepted and after the next change the ASA printed the following error messages
ERROR: NAT Policy is not downloaded
ERROR: object (<our object name>) updation failed due to internal error
After this configuration change the management connection was lost and the ASA unit rebooted. After the reboot the same changes were attempted again and it rebooted again.
One of our people decided to load the configuration from Flash to his computer, modify the configuration, remove the customer Security Context and load it with the newly created configuration file to get the changes done
So at the moment we find ourselves running over 300 Security Contexts on this software level 8.4(6) while potentially introducing a risk that a NAT configurations change will reload the device. Needles to say, reboot of around 200 customer Security Context doesn't really increase customer happines.
We will open a TAC Case about this to have a further look into the actual cause of the reloads
I was wondering if anyone has had the same problem or if any Cisco employee could provide insight to this situation. Are we facing this bug (it does not mentiond reload/reboot so I am in doubt) or is there some other bug that might explain this?
EDIT: Managed to forget to add a comment that after the NAT ERROR messages the unit reboot, doh (which is the actual problem)
Basically in the problem situation new routers were installed on the customer site while the LAN network was also changed. This required that we change the local IP address for the customers Static NAT configuration which was done as a Section 2 Auto NAT configuration.
The only thing one of our people managed to change is to go under the second "object network " and issue the "host " command which would essentially change the real IP address for the Static NAT translation. While the second "object network " "host" configuration was changed the ASA gave the ERROR messages posted in my original post.
I was told that after this the unit rebooted.
After the reboot the user attempted to first remove the "host" configuration under the "object network " and then enter the new "host" which resulted yet in another reboot of the unit. After that the TAC case was opened.
Basically the Security Context had this kind of configurations.
object network STATIC-x-x-x-x
nat (sourceint,destint) static x.x.x.x
object network STATIC-y-y-y-y
nat (sourceint,destint) static y.y.y.y
The user attempted to insert the new "host" command with the following commands
object network STATIC-x-x-x-x
object network STATIC-y-y-y-y
I think I have done these changes on many of the other ASA units running 8.4(2) software and also my home ASA running 8.4(5) which I use both for work purposes and simulating/lab setups that people ask questions about here on the CSC and I have not faced this problem ever.
The above commands are to my knowledge the only commands entered to the unit.
At the moment Cisco TAC is in the process or trying to recreate the situation and also test if another software level truly corrects this bug.
I will have to make an addition to the customer NAT configuration in about 10min since they need the change today.
This NAT change will be done to a Section 3 Manual NAT rule so will have to see if the effect will be the same
We configure Dynamic PAT for multiple LAN/DMZ interfaces in the following way
If adding the new network results in a reboot I will probably have to download the current configuration from Flash, modify it and upload it to the Flash and recreate the whole firewall with the changes already added to get this working.
The BugID I mentioned doesnt really specify if this problem is with all the different NAT types and if for example removing the NAT rule in question and recreating it would avoid the bug. It does say While ASA NAT configuration so should I understand that as so that removing the rule completely and recreating it with new information would not cause this bug to occur.
There are few interoperability issues between 2960x and other vendors.
In this article we will go over the steps on how to debug these issues.
Check the phy id from “show controllers ethernet-controller phy” output.
It appears that you are trying to access end of life information from
Cisco search. We have detected a problem with your script ("eol...) and
would like to help. Please contact email@example.com if you would
Cisco Licensing is pleased to announce the new Licensing Portal. It
provides customers with an enhanced self-service experience by
stream-lining and automating many licensing activities. Please take a
moment to familiarize yourself with the features that ...