ASA5515X mgmt port stops working in active/standby setup

Answered Question

To whoever can help:


I've setup a pair of ASA 5515X firewalls in an active/standby failover group.  When I issue the command ip address 172.16.1.2 255.255.255.0 standby 172.16.1.3 from the management interface configuration mode, I'm able to ping and connect via ssh or asdm for period of time.  If I let the interface go idle for any extended period of time and then attempt to ping it again, I get one ping off and then "Request timed out."  The only way to get the interface to begin accepting traffic again is to change the IP address again, or reload each device in sequence.


I've tested this effect from 2 different workstations on the same network and both end up with the same results: a single ping and then "request timed out."  I've disbled the firewalls on each of the workstations (not that it would matter) just to make sure I had an open line of communication with the ASA, yet the problem persists.  This occurs even if I set an IP with no standby on each unit.


The weird thing is, the switches do not experience this behavior when attempting to ping or ssh to the ASAs.  My workstation will not be able to ping it, but the switch that is connected to the management is able to ping and SSH to the ASAs with no problems.  My current setup is like this:


C3750 Switch

Gi1/0/18 (VLAN 999) = Management Interface on ASA1 172.16.1.2

Gi2/0/18 (VLAN 999) = Management Interface on ASA2 172.16.1.3

Gi1/0/10 (VLAN 999) = Workstation 172.16.1.100

SVI 999 = 172.16.1.1


I've set console logging to debug on the ASAs and watched as the first ping came in...and then nothing.  So it almost seems as though as if the problem is on the C3750 because the packets just aren't making it to the management interface.  Just writing this out has sorta given me a logical perspective to maybe analyze the switch a bit more in depth.  I'd still like to see if anyone has any ideas as to what might be causing this behavior.


Thanks for any insight anyone might be able to provide.


John H

.

Correct Answer by Jouni Forss about 4 years 1 month ago

Hi,


The information you have given does seem to point to a wierd situation.


Essentially when in Failover pair the ASAs should to my understanding have a MAC address that is alway son the Active unit and a MAC address that is always on the Standby unit. Same naturally goes for the IP address therefore even though the switch of the active unit happens the ARP should stay the same.


I am not aware of any other IP/MAC like there might be with Cisco Routers HSRP implementations.


To my understanding the Management interface has had some changes when comparing the original ASA5500 series and the new ASA5500-X series.


One big change is ofcourse the fact that the interface has been disabled as a Data interface and can only be used for management purposes. And there have been mentions on Cisco documents that it couldnt be used in any high availability setups.


Here is a quote from a Cisco document explaining migration from ASA5500 series to ASA5500-X series



Management Port Configuration Changes


The  ASA 5500-X Series introduced a shared management port for firewall and  IPS services.,There are certain caveats to follow during migration from  the ASA 5500 Series.

•  The shared management port cannot be used as a data port. All  through-the-box traffic arriving at the management port will be dropped  implicitly. This cannot be disabled.

• The shared management port cannot be used as a part of a high availability configuration.


If  the ASA management port (M0/0) on the ASA 5500 Series appliance was  being used as a data port, the configuration associated with that port  should be moved to one of the gigabit data ports numbered above G0/3.


Source:

http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5708/ps5709/ps6120/guide_c07-727453.html


The ASA 8.6 configuration guide also states the following


Management 0/0 Interface on the ASA 5512-X through ASA 5555-X



The Management 0/0 interface on the ASA 5512-X through ASA 5555-X has the following characteristics:


No through traffic support


No subinterface support


No priority queue support


No multicast MAC support


The  IPS SSP software module shares the Management 0/0 interface. Separate  MAC addresses and IP addresses are supported for the ASA and IPS module.  You must perform configuration of the IPS IP address within the IPS  operating system. However, physical characteristics (such as enabling  the interface) are configured on the ASA.




What I am wondering here is if this could have anything to do with the fact that the ASA5500-X series ASA CX or IPS management is also handled through this management port. I am not sure if there is even a chance that the management IP address of the CX or IPS could be configured to overlap with the actual IP address of the Management interface.


I was personally quite confused at first when I got my ASA5515-X with ASA CX. The Management interface had its IP address and also ASA CX had a management IP address on the same interface but this didnt show on the interface configuration. It could be reset on the ASDM side through the Startup Wizard atleast. You could skip all the other phases and just apply the CX settings if you wanted to change that management IP address.


You have not mention if you have either IPS or CX. To be honest I am not 100% sure but I think if the ASA comes with the SSD HD then atleast in CX case there should be an evaluation license so you could actually use it and therefore there would be a management IP address also.


One thing you could naturally do on the ASAs is to "debug arp" and monitor that traffic and compare it to what you are seeing on the switch and the hosts.


I dont think you should be seeing 2 different MAC addresses on a single switch port unless its maybe somehow related to the CX/IPS setup


On my ASA5515-X for example I could use these commands to view the IP addresses configured for CX or IPS management (I have CX)


show module ips details


show module cxsc details


For example in my case


some output omitted


ASA-CX# show module cxsc details

Getting details from the Service Module, please wait...


Card Type:          ASA CX5515 Security Appliance

Model:              ASA CX5515


Software version:   9.1.1

MAC Address Range:  f872.ea24.ed03 to f872.ea24.ed03

App. name:          ASA CX

App. Status:        Up

App. Status Desc:   Normal Operation

App. version:       9.1.1

Data Plane Status:  Up

Status:             Up

Mgmt IP addr:       10.0.250.251

Mgmt Network mask:  255.255.255.0

Mgmt Gateway:       10.0.250.2

Mgmt web ports:     443

Mgmt TLS enabled:   true



While my Management interface configuration is


interface Management0/0

management-only

nameif MGMT

security-level 100

ip address 10.0.250.250 255.255.255.0



As for your question about the Management interface and its IP addresses. To my understanding you are unable to separate any interface from the actual Failover process. You can only specify that some interfaces are not monited with regards to Failover but their configuration will still change depending on the Failover state.


So I would presume that even though you manually set the interface IP address separately on the units that some Failover event might trigger that the other units interface configuration would be overwritten by the other unit.


I would imagine that the Management interface should work in a Failover setup because they dont give an option to use the Management interface in the way you are attempting. Though again the documents are pretty vague with regards to this information.


- Jouni

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.

I have some new information.  I started to look through the arp entries on my PC and the 3750 and I'm not at all certain how to interpret what's going on other than to say, the switch is simply giving the wrong MAC address.


C:>arp -d *

C:>ping 172.16.1.2 -t



Pinging 172.16.1.2 with 32 bytes of data:

Reply from 172.16.1.2: bytes=32 time<1ms TTL=255

Request timed out.



Ping statistics for 172.16.1.2:

    Packets: Sent = 2, Received = 1, Lost = 1 (50% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C

M:\>arp -a


Interface: 172.16.1.10 --- 0x11

  Internet Address      Physical Address      Type

  172.16.1.2          6c-20-56-bd-f8-33     dynamic (This also shows up as 6c-20-56-bd-f6-23 occasionally)

  224.0.0.22            01-00-5e-00-00-16     static


DistSW01# show ip arp

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  172.16.1.10            0   0010.1884.69d2  ARPA   Vlan100

Internet  172.16.1.1             -   e4d3.f155.6042  ARPA   Vlan100

Internet  172.16.1.3            46   6c20.56bd.f625  ARPA   Vlan100

Internet  172.16.1.2             0   6c20.56bd.f835  ARPA   Vlan100


I would expect this kind of behavior from a high availability routing protocol, but I honestly know little to nothing about how the ASAs hand out MAC information in a high availability cluster.  Upon further review of the mac address table on the switch I see the following:


DistSW01# show mac address-table

.

.

.

100    6c20.56bd.f623    DYNAMIC     Gi1/0/18 (No clue what this is because the real MAC of ASA2 is 6c20.56bd.f625)

100    6c20.56bd.f833    DYNAMIC     Gi2/0/18 (virtual MAC?)

100    6c20.56bd.f835    DYNAMIC     Gi2/0/18 (real MAC of ASA1)


There are two MAC addresses tied to one port, which suggests that the high availability protocol does indeed use a virtual MAC address.  I'm not sure what the formula is for that, but I'm sure I'll figure that out later.  What bugs me is, why does the switch's arp table not show the same MAC address that my work station's arp table shows?  The fact that the switch has the real MAC address of the ASA explains why it is able to ping it and why my workstation (which does not have the same MAC in its arp table) cannot.


I've taken this investigation probably as far as I (that is my level of knowledge) can go at this point.  Can anyone fill in the blanks as to why, if this ASA is supposed to answer for that virtual MAC, does it not answer?  I've read in several locations that the management interface cannot be included in the failover process.  I'm not entirely sure if that relates since that same documentation outlines how to add an IP to the management interface on the standby unit (ip address standby .


Any help is much appreciated.


John H


Message was edited by: John Holmes.  Edited for accuracy of MAC address information.

So at this point, I'm pretty sure that the whole standby management interface is just a wrong idea on my part.  So I would like to assign unique IP addresses to each management interface on the ASAs.  However when I attempt to assign a unique IP to the standby unit I get the following:


ASA01# conf t

**** WARNING ****

        Configuration Replication is NOT performed from Standby unit to Active unit.

        Configurations are no longer synchronized.

ASA01(config)# int management 0/0

**** WARNING ****

        Configuration Replication is NOT performed from Standby unit to Active unit.

        Configurations are no longer synchronized.


And even if I do assign an IP address to this standby unit, the next time one of the devices reboots, it's going to delete its configuration (with exception of the failover configuration) and download the config from the active node.  This means it's going to get the management IP of the active node.  Is there a way to exclude the management interface config from the deletion?  I thought this would have been a default behavior given that management interface can't be included in HA.


John H

Correct Answer
Jouni Forss Fri, 07/12/2013 - 17:05
User Badges:
  • Super Bronze, 10000 points or more

Hi,


The information you have given does seem to point to a wierd situation.


Essentially when in Failover pair the ASAs should to my understanding have a MAC address that is alway son the Active unit and a MAC address that is always on the Standby unit. Same naturally goes for the IP address therefore even though the switch of the active unit happens the ARP should stay the same.


I am not aware of any other IP/MAC like there might be with Cisco Routers HSRP implementations.


To my understanding the Management interface has had some changes when comparing the original ASA5500 series and the new ASA5500-X series.


One big change is ofcourse the fact that the interface has been disabled as a Data interface and can only be used for management purposes. And there have been mentions on Cisco documents that it couldnt be used in any high availability setups.


Here is a quote from a Cisco document explaining migration from ASA5500 series to ASA5500-X series



Management Port Configuration Changes


The  ASA 5500-X Series introduced a shared management port for firewall and  IPS services.,There are certain caveats to follow during migration from  the ASA 5500 Series.

•  The shared management port cannot be used as a data port. All  through-the-box traffic arriving at the management port will be dropped  implicitly. This cannot be disabled.

• The shared management port cannot be used as a part of a high availability configuration.


If  the ASA management port (M0/0) on the ASA 5500 Series appliance was  being used as a data port, the configuration associated with that port  should be moved to one of the gigabit data ports numbered above G0/3.


Source:

http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5708/ps5709/ps6120/guide_c07-727453.html


The ASA 8.6 configuration guide also states the following


Management 0/0 Interface on the ASA 5512-X through ASA 5555-X



The Management 0/0 interface on the ASA 5512-X through ASA 5555-X has the following characteristics:


No through traffic support


No subinterface support


No priority queue support


No multicast MAC support


The  IPS SSP software module shares the Management 0/0 interface. Separate  MAC addresses and IP addresses are supported for the ASA and IPS module.  You must perform configuration of the IPS IP address within the IPS  operating system. However, physical characteristics (such as enabling  the interface) are configured on the ASA.




What I am wondering here is if this could have anything to do with the fact that the ASA5500-X series ASA CX or IPS management is also handled through this management port. I am not sure if there is even a chance that the management IP address of the CX or IPS could be configured to overlap with the actual IP address of the Management interface.


I was personally quite confused at first when I got my ASA5515-X with ASA CX. The Management interface had its IP address and also ASA CX had a management IP address on the same interface but this didnt show on the interface configuration. It could be reset on the ASDM side through the Startup Wizard atleast. You could skip all the other phases and just apply the CX settings if you wanted to change that management IP address.


You have not mention if you have either IPS or CX. To be honest I am not 100% sure but I think if the ASA comes with the SSD HD then atleast in CX case there should be an evaluation license so you could actually use it and therefore there would be a management IP address also.


One thing you could naturally do on the ASAs is to "debug arp" and monitor that traffic and compare it to what you are seeing on the switch and the hosts.


I dont think you should be seeing 2 different MAC addresses on a single switch port unless its maybe somehow related to the CX/IPS setup


On my ASA5515-X for example I could use these commands to view the IP addresses configured for CX or IPS management (I have CX)


show module ips details


show module cxsc details


For example in my case


some output omitted


ASA-CX# show module cxsc details

Getting details from the Service Module, please wait...


Card Type:          ASA CX5515 Security Appliance

Model:              ASA CX5515


Software version:   9.1.1

MAC Address Range:  f872.ea24.ed03 to f872.ea24.ed03

App. name:          ASA CX

App. Status:        Up

App. Status Desc:   Normal Operation

App. version:       9.1.1

Data Plane Status:  Up

Status:             Up

Mgmt IP addr:       10.0.250.251

Mgmt Network mask:  255.255.255.0

Mgmt Gateway:       10.0.250.2

Mgmt web ports:     443

Mgmt TLS enabled:   true



While my Management interface configuration is


interface Management0/0

management-only

nameif MGMT

security-level 100

ip address 10.0.250.250 255.255.255.0



As for your question about the Management interface and its IP addresses. To my understanding you are unable to separate any interface from the actual Failover process. You can only specify that some interfaces are not monited with regards to Failover but their configuration will still change depending on the Failover state.


So I would presume that even though you manually set the interface IP address separately on the units that some Failover event might trigger that the other units interface configuration would be overwritten by the other unit.


I would imagine that the Management interface should work in a Failover setup because they dont give an option to use the Management interface in the way you are attempting. Though again the documents are pretty vague with regards to this information.


- Jouni

Jouni,


This is fantastic information.  We're upgrading a pair of ASA 5510s that I never really touched the IPS on.  So it didn't occur to me to check the IPS.  To be perfectly honest, I a little new to IPS so it didn't occur to me to check this.  I had read all of the resources you were talking about, and I must have seen that statement about the IPS and the ASA sharing the same management port at least 5 times.  It never occurred to me to check the IPS (and tbh, I didn't realize we had one).  Thanks a ton for your information.  I won't be back in the office until Monday to check the solution on the active unit (console is currently plugged into the standby unit), but as soon as I can confirm that changing the IPS's IP address will resolve this issue, I'll mark your answer as correct.


THANK YOU so much for your insight.


Cheers,


John H

Jouni,


It was indeed the IPS.  Thank's a million for your insight!  Now I guess the only thing left to determine is, why would Cisco assign the same IP address 2 to different devices with distict MAC addresses using the same port.  Perhaps there's a valid reason, but for the time being I will just be content to know that I can now use the management interface.  I can't say for certain that it's going to work, and I'll monitor it in the coming weeks, but I believe this resolves the failover functionality of the mgmt interfaces as well.  I'm now able to use the command ip addr (ip) (mask) standby (standby ip).  I will perform some reboots and make sure the configurations stick across both devices and then test failover access.


Thanks again for your excellent explanation.


Cheers,


John H

Jouni Forss Mon, 07/15/2013 - 09:57
User Badges:
  • Super Bronze, 10000 points or more

Hi,


Glad to hear you found the problem.


To be honest I was almost confident that it could NOT be the IPS management IP address but I guess it was.


I cant remember what the management interface and the ASA-CX management IP address was when I first unboxed my ASA unit. I guess I could always return it to factory default and check at some point but it does seem somehow unbelievable that the IP address would be set to overlap.


Hopefully there are no more surprises for your failover setup


Thank you for marking the correct reply


- Jouni

Actions

This Discussion

Related Content