Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

Nexus 7K F2e module mac address limits

Hi

In the Cisco docs it says that the F2e card supports 16k MAC addresses per SoC and there are 12 per linecard implying a 192K MAC address limit PER linecard.

When I read further it says that MAC address tables are synchronized across ports in the same VDC.  Does that mean that if I only use 1 VDC then the limit of MAC addresses across all linecards is 16K in total or as I have read somewhere else does limiting the VLANs on each port also reduce the MAC addresses held on each card?

So if on card 1 I have a vPC carrying VLANs 10, 20 and 30.  On another port I have vlans 10, 40 and 50 and on a further linecard port I have VLANs 60, 70 and 80 what MAC addresses are where ?

Are all MAC addresses from VLANs 10 - 80 on all cards or do the relevant card and it's SoC only hold the MAC addresses for the VLANs that exist on that cards trunks ?

Would the second SoC on the 1st linecard see VLAN 10, 20 and 30 MAC addresses as well as 40 and 50 ?

Would the second linecard see a copy of MAC addresses in vlans 10 to 50 if they only get trunked on ports on the 1st linecard?

So in short are all MACs copied to all SoC's in the same VDC or only to those ones with shared VLANs.

Many thanks for any replies.

St.

Everyone's tags (5)
1 ACCEPTED SOLUTION

Accepted Solutions

Nexus 7K F2e module mac address limits

Hi Stuart,

Your summary is correct. The trick is to group your VLANs to the same 4 port group which is served by the same SOC.

With an empty line card this should be easy with good planning, but those 48 ports will soon fill up!

The command to check your SOC ulitization per linecard:

attach module x

show hardware internal forwarding l2 table utilization instance all

cheers,

Seb.

11 REPLIES

Nexus 7K F2e module mac address limits

Hi there,

We have hit the 16K MAC limit in a production campus environment; the Cisco docs and TAC have been pretty poor at discribing why!

From what we have seen when an F2 only VDC is configured to operate at L3 and has SVIs present the MAC address limit is at its worst.

Each sequential 4 ports (port-group) share the same SOC. Any MACs which appear on a VLAN are stored on the respective SOC MAC table. The Nexus implements MAC-sync, so if a VLAN appears on two different SOCs they will sync their MAC table entires for that VLAN. The worst case scenario is on vPC links which trunk multiple VLANs where the 16K limit is quickly hit.

There are two methods to mitigate this limitation.

  1. Implement the F2 VDC as layer 2 aggregation only (this is what it was designed in the DC for). Use vPC links to an M-series VDC which can handle the Later 3 routing. Finally configure 'conversational mac learning', this will further reduce the impact of MAC-sync.
  2. F2e's can be combined with a M-series cards. This is a good position, as when combined with a M card the F2e operates at Layer2 only and the resources on the M card (MAC tables) are used by it; 128K. This also has the benefit of using the FAB backplane between the linecards as opposed to the vPC link in the above solution which is a potential bottleneck (160Gbps across two Nexus chassis - 8 x 10Gb fibres per chassis).

In short, the Nexus is not a great campus solution!

cheers,

Seb.

New Member

Nexus 7K F2e module mac address limits

Many thanks Seb

You say "The Nexus implements MAC-sync, so if a VLAN appears on two different SOCs they will sync their MAC table entires for that VLAN."

Are you saying that MACs are only sync'd between SoCs that share VLANs ?  So if VLANs are distributed around the SoCs and rarely appear on multiple SoCs then the replication of MACs will me minimised ?

So on line card 1 port 1 we have a trunk with VLANs A, B and C and on the same linecard on port 48 we have a trunk with VLANs A, D and E then only MACs on VLAN A are sync'd across the SoCs ?

Thanks, Stuart.

Nexus 7K F2e module mac address limits

Hi Stuart,

Your summary is correct. The trick is to group your VLANs to the same 4 port group which is served by the same SOC.

With an empty line card this should be easy with good planning, but those 48 ports will soon fill up!

The command to check your SOC ulitization per linecard:

attach module x

show hardware internal forwarding l2 table utilization instance all

cheers,

Seb.

New Member

Re:Nexus 7K F2e module mac address limits

Seb

attach module x

show hardware internal forwarding l2 table utilization instance all

That command is very useful but I found when I upgraded to 6.1.4a the command is no longer valid.  I've reverted to 6.1.2 in the meantime to do some testing because I like the output format from that command.  No doubt there is a similar command that gives similar output in 6.1.4a but I've not found it yet.

What I've found is that I have a few cards with many vPC's on them.  Each vpc is a trunk with VLANs limited to very specific ranges and they don't tend to duplicate across vPC's apart from one VLAN.  So vPC X on SoC 1 may have vlan 10,20 and 30 and vPC Y on SoC 5 may have vlan 10, 40 and 50

What I have confirmed in testing is that any MAC address from VLANs 20 to 50 appears in the count for ALL SoCs on all cards.  So limiting VLANs to a SoC does not seem to limit the MAC replication.

I found something on conversational mac learning that may help the situation but I believe that requires Fabric path to operate on ethernet VLANs.

Do you have any knowledge of conversational mac learning/fabric path ?

Thanks again, Stuart.

Re: Nexus 7K F2e module mac address limits

Hi Stuart,

I finally got a response back from Cisco, and the piece of information that was most relevant was:

We  synchronize MAC addresses across all SOCs in a VDC or switch that have a  VLAN with a SVI defined, regardless if the SOC carries the VLAN or not  with one  exception of a VLAN with no SVI.

...which sort explains the numbers on the SOCs better than what I first proposed in my original post.

With regard to conversational MAC learning; this solution was suggested by the DC CCIE's at our supplier. When we tried it, our F2 was still operating at L3 so gave unexpectedly poor results.

We're currently in the process of creating an M1/2 VDC for layer3. Eventually we'll get round testing the F2 at L3 again...

Be good to hear what results you get.

cheers,

Seb.

New Member

Re:Nexus 7K F2e module mac address limits

Thanks Seb

Sorry for the late reply.  I've been on holiday.

I found similar.  Any MAC on any SoC is replicated across all cards unless we deploy VDC's which isn't really an option for me.  I hear there are F3 cards due out with significantly large MAC tables that we may look to use if the MAC limit is a problem.  Conversation MAC learning didn't help.  A very strangely low MAC table size for such a high end switch.

Stuart.

New Member

Nexus 7K F2e module mac address limits

You can use this command:

show hardware capacity forwarding

You will get a lot of ouput, but this one is whatyou want:

L2 Forwarding Resources

-------------------------

  L2 entries: Module inst   total    used   mcast   ucast   lines   lines_full

------------------------------------------------------------------------------

              3         0   16384    1183      38   10145     512            0

              3         1   16384    1308      16    1292     512            0

              3         2   16384      11       0      11     512            0

              3         3   16384      11       0      11     512            0

              3         4   16384      11       0      11     512            0

              3         5   16384      11       0      11     512            0

              3         6   16384      11       0      11     512            0

              3         7   16384      11       0      11     512            0

              3         8   16384      11       0      11     512            0

              3         9   16384      61       0      61     512            0

              3        10   16384    2380      20   10360     512            0

              3        11   16384    3380      20   10360     512            0

New Member

Nexus 7K F2e module mac address limits

Did you get any information how to use similar command to see mac table utilisation on 6.1.4?

Thx

Hubert

New Member

Nexus 7K F2e module mac address limits

Hi Seb,

Cisco documentation with regards to classical ethernet and dynamic mac address learning is rather sparse. Most of it is limited to "you can configure it and here is how:".We're facing simailr issues and are considering implementing this too. Do you have any experience with this? Especially with regards to impact on existing VPCs etc.

cheers!

New Member

Re:Nexus 7K F2e module mac address limits

Thanks again Web.

For the size of chassis this is it seems quite a restriction that needs carefully planned. To be honest I've never been concerned about MAC capacity on large switches before. I can't think there's been a situation when I've been close to a chassis capacity. It has been quite a surprise to find this in a very high end chassis like a 7k.

St.


Sent from Cisco Technical Support Android App

Cisco Employee

Next link is too useful:http:

Next link is too useful:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/best_practices/cli_mgmt_guide/cli_mgmt_bp/hw_resources.html

 

4075
Views
10
Helpful
11
Replies
CreatePlease to create content