Solved: ACE 4710 Duplicate MAC Addresses

tonybourke · ‎05-13-2009

I have two pairs of ACEs, both are running redundancy. I've got shared-vlan-hostid set to different pools on both. However, it wasn't set initially, and the Admin context administrative alias IPs, as well as user contexts alias IPs, are still using the same VMAC addresses on both pairs.

ACE pair 1: 00.0b.fc.fe.1b.01

ACE pair 2: 00.0b.fc.fe.1b.01

Rebooting, removing the alias and re-adding it, removing a context, nothing seems to prevent the MAC address collisions.

Version: system: Version A3(2.2) [build 3.0(0)A3(2.2) adbuild_20:56:50-2009/04/03_/a

uto/adbu-rel2/rel_a3_2_2_throttle/REL_3_0_0_A3_2_2]

Has anyone seen anything like this?

sachinga.hcl · ‎05-13-2009

Hi Tony,

When contexts share a VLAN, the ACE assigns a different MAC address to the VLAN on each context. The MAC addresses reserved for shared VLANs are 0x001243dc6b00 to 0x001243dcaaff, inclusive. All ACE modules derive these addresses from a global pool of 16k MAC addresses. This pool is divided into 16 banks, each containing 1,024 addresses. Thus, there can be 16 ACEs per subnet.

Each ACE supports 1,024 shared VLANs, and uses only one bank of MAC addresses out of the pool. Thus, a shared MAC address is associated with a shared VLAN interface.

By default, the bank of MAC addresses that the ACE uses is randomly selected at boot time. However, if you configure two ACE modules in the same Layer 2 network and they are using shared VLANs, the ACEs may select the same address bank, resulting in the use of the same MAC addresses. To avoid this conflict, you need to configure the bank that the ACEs will use.

To configure a specific bank of MAC addresses for an ACE, use the shared-vlan-hostid command in configuration mode in the Admin context. The syntax for this command is:

shared-vlan-hostid number

The number argument indicates the bank of MAC addresses that the ACE uses. Enter a number from 1 to 16. For example, to configure bank 2 of MAC addresses, enter:

host1/Admin(config)# shared-vlan-hostid 2

To remove the configured bank of MAC addresses and allow the ACE to randomly select a bank, use the no shared-vlan-hostid command. For example, enter:

host1/Admin(config)# no shared-vlan-hostid

Plz rate.

Kind Regards,

Sachin Garg

View solution in original post

sachinga.hcl · ‎05-13-2009

Hi Tony,

By default, ARP entry replication is enabled. To disable the replication of ARP entries, use the arp sync disable command in configuration mode.

The syntax of this command is as follows:

arp sync disable

For example, to disable the replication of ARP entries, enter:

host1/Admin(config)# arp sync disable

To reenable ARP entry replication, use the no arp sync disable command. For example, enter:

host1/Admin(config)# no arp sync disable

By default, for bridged traffic, the ACE learns MAC addresses from all traffic. For routed traffic, the ACE learns MAC addresses only from ARP response packets or from packets that are destined to the ACE (for example, a ping to a VIP or a ping to a VLAN interface). To enable the ACE to learn MAC addresses from traffic after the command has been disabled, use the arp learned-mode enable command in configuration mode. You configure this command per context. This command is enabled by default.

The syntax of this command is as follows:

arp learned-mode enable

For example, to enable the ACE to learn MAC addresses from traffic after the command has been disabled, enter:

host1/Admin(config)# arp learned-mode enable

To instruct the ACE to forward packets without learning the ARP information, use the no arp learned-mode enable command. For example, enter:

host1/Admin(config)# no arp learned-mode enable

Kindly use the url given below regarding your issue:

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/routing_bridging/guide/arp.html#wp1186276

sachin garg

tonybourke · ‎05-13-2009

Hi Sachin,

Thanks for the note, however that wasn't the problem I was having.

I'll do bit more explanation.

I have two pairs of ACEs. 4 in total.

ACE-1 partners ACE-2, and ACE-3 partners ACE-4

I configure a VIP in a context of the 1/2 pair, and a VIP in the context of the 3/4 pair

The problem is, ACE 1/2 and 3/4 sit on the same VLANs.

The VIP addresses were of course different. Let's say 192.168.0.200 (1/2 pair) and 192.168.0.201 (3/4 pair).

The MAC address for both IPs was 00.0b.fc.fe.1b.09

I think I've found a workaround, however. As I added new contexts, I was starting with FT group 2 and going up. By using *different* FT group numbers between the two paris of ACEs, the MAC addresses become unique.

The VMAC seems to be derived from the FT group number. FT group 1 is 00.0b.fc.fe.1b.01, FT group 2 is 00.0b.fc.fe.1b.02, etc.

sachinga.hcl · ‎05-13-2009

You are absolutely right,

One virtual MAC address (VMAC) is associated with each FT group. The format of the VMAC is: 00-0b-fc-fe-1b-groupID. Because a VMAC does not change upon a switchover, the client and server ARP tables does not require updating.

The ACE selects a VMAC from a pool of virtual MACs available to it.

You can specify the pool of MAC addresses that the local ACE and the peer ACE use by configuring the

shared-vlan-hostid command and the peer shared-vlan-hostid command, respectively.

To avoid MAC address conflicts, be sure that the two pools are different on the two ACEs.

Each peer uses a VMAC that is dependent on the FT group number. If you are using multiple ACEs in the same chassis, be careful when using the same FT groups in more than one module.

To display the VMAC for an FT group by entering the following command:

ACE_module5/Admin# show interface internal iftable vlan100

vlan100

--------

ifid: 6

Context: 0

ifIndex: 16777316

physid: 100

rmode: 0 (unknown)

iftype: 0 (vlan)

bvi_bgid: 0

MTU: 1500

MAC: 00:18:b9:a6:91:15

VMAC: 00:00:00:00:00:00 <------- Virtual MAC Address

Flags: 0x8a000800 (valid, down, admin-down, Non-redundant, tracked)

ACL In: 0

ACL Out: 0

Route ID: 0

FTgroupID: 0

Zone ID: 6

Sec Level: 0

L2 ACL: bpdu DENY, ipv6 DENY, mpls DENY, all DENY

LastChange: 0 (Thu Jan 1 00:00:00 1970)

iflookup index: 100

vlan-vmac index:0

Next Shared IF: 0

Lock: Unlocked, seq 5

Lock errors: 0

Unlock errors: 0

No. of times locked: 5

No. of times unlocked: 5

Current/last owner: 0x40a7fc

Check the FT group configuration on both devices. Make sure that both devices are associated with the same context. Enter the following command:

ACE_module5/Admin# show running-config ft

Also verify the FT peer status and configuration by entering the following command:

ACE_module5/Admin# show ft peer detail

Peer Id : 1

State : FSM_PEER_STATE_COMPATIBLE

Maintenance mode : MAINT_MODE_OFF

FT Vlan : 100

FT Vlan IF State : DOWN

My IP Addr : 10.1.1.1

Peer IP Addr : 10.1.1.2

Query Vlan : 110

Query Vlan IF State : DOWN

Peer Query IP Addr : 172.25.91.202

Heartbeat Interval : 300

Heartbeat Count : 20

Tx Packets : 318573

Tx Bytes : 66301061

Rx Packets : 318540

Rx Bytes : 66272840

Rx Error Bytes : 0

Tx Keepalive Packets : 318480

Rx Keepalive Packets : 318480

TL_CLOSE count : 0

FT_VLAN_DOWN count : 0

PEER_DOWN count : 0

SRG Compatibility : COMPATIBLE

License Compatibility : COMPATIBLE

FT Groups : 3

....contd 2

sachinga.hcl · ‎05-13-2009

page 2...

Verify the FT group status and configuration by entering the following command:

ACE_module5/Admin# show ft group detail

FT Group : 1

No. of Contexts : 1

Configured Status : in-service

Maintenance mode : MAINT_MODE_OFF

My State : FSM_FT_STATE_ACTIVE

My Config Priority : 110

My Net Priority : 110

My Preempt : Enabled

Peer State : FSM_FT_STATE_STANDBY

Peer Config Priority : 100

Peer Net Priority : 100

Peer Preempt : Enabled

Peer Id : 1

Last State Change time : Thu Apr 2 00:00:00 2009

Running cfg sync enabled : Enabled

Running cfg sync status : Running configuration sync has completed

Startup cfg sync enabled : Enabled

Startup cfg sync status : Running configuration sync has completed

Bulk sync done for ARP: 0

Bulk sync done for LB: 0

Bulk sync done for ICM: 0

Kind regards,

Kindly Rate.

sachin garg

tonybourke · ‎05-13-2009

Thanks for posting that.

My understanding from the documentation was the 1K pool of MAC addresses gets used for VMACs, but that seems to not be the case.

"The ACE selects a VMAC from a pool of virtual MACs available to it. You can specify the pool of MAC addresses that the local ACE and the peer ACE use by configuring the shared-vlan-hostid command and the peer shared-vlan-hostid command, respectively."

The shared-vlan-hostid doesn't seem to have any effect on the VMAC that gets used for alias and virtual IPs. Only the FT group does.

What does the 16K MAC pool get used for?

sachinga.hcl · ‎05-13-2009

Hi Tony,

When contexts share a VLAN, the ACE assigns a different MAC address to the VLAN on each context. The MAC addresses reserved for shared VLANs are 0x001243dc6b00 to 0x001243dcaaff, inclusive. All ACE modules derive these addresses from a global pool of 16k MAC addresses. This pool is divided into 16 banks, each containing 1,024 addresses. Thus, there can be 16 ACEs per subnet.

Each ACE supports 1,024 shared VLANs, and uses only one bank of MAC addresses out of the pool. Thus, a shared MAC address is associated with a shared VLAN interface.

By default, the bank of MAC addresses that the ACE uses is randomly selected at boot time. However, if you configure two ACE modules in the same Layer 2 network and they are using shared VLANs, the ACEs may select the same address bank, resulting in the use of the same MAC addresses. To avoid this conflict, you need to configure the bank that the ACEs will use.

To configure a specific bank of MAC addresses for an ACE, use the shared-vlan-hostid command in configuration mode in the Admin context. The syntax for this command is:

shared-vlan-hostid number

The number argument indicates the bank of MAC addresses that the ACE uses. Enter a number from 1 to 16. For example, to configure bank 2 of MAC addresses, enter:

host1/Admin(config)# shared-vlan-hostid 2

To remove the configured bank of MAC addresses and allow the ACE to randomly select a bank, use the no shared-vlan-hostid command. For example, enter:

host1/Admin(config)# no shared-vlan-hostid

Plz rate.

Kind Regards,

Sachin Garg

tonybourke · ‎05-14-2009

Hi Sachin,

I think the confusion on my part was the terminology. In the documentation, shared MACs are different than VMACs. And the ACE treats them differently.

Only interfaces that share a VLAN with another context get those 00:12 MAC addresses. Otherwise, they use the burnt-in. (Is this correct?)

VMACs, the MAC addresses that do not change during fail-over, and are used for Virtual Servers as well as alias IPs, are not derived from this MAC pool. The http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_2_2/configuration/device_manager/guide/UGha.html#wp1053157">documentation mentions: "The ACE selects a VMAC from a pool of virtual MACs available to it." The pool part seemed to imply that it was derived from one of those 16K pools. But that doesn't seem to be case, the FT group number that you use decides the VMAC. So there are only 21 VMACs available.

Thanks for your help clearing this up Sachin.