Solved: Tcam resource exhausted on Nexus 7010 M1 card

ben.tan · ‎11-04-2011

Hi,

Anybody seen the issue below yet? I am on NX-OS 5.2.1. The Mod is M1 card. The FEX port (Nexus 2248) end up in err-disabled after TCAM exhauseted. I have to remove "QoS" policy on that port and bounce the port. Than it will come back up again.

2011 Nov 5 01:20:03 BNZ-AKDCM1-L01-DC-02 %ACLQOS-SLOT9-2-ACLQOS_OOTR: Tcam resource exhausted: tcam 0, bank 1

2011 Nov 5 01:20:03 BNZ-AKDCM1-L01-DC-02 %ACLQOS-SLOT10-2-ACLQOS_OOTR: Tcam resource exhausted: tcam 0, bank 1

2011 Nov 5 01:20:03 BNZ-AKDCM1-L01-DC-02 %ETHPORT-5-IF_SEQ_ERROR: Error ("Sufficient free entries are not available in TCAM bank") communicating with MTS_SAP_IPQOS_MGR for opcode MTS_OPC_ETHPM_PORT_BRINGUP (RID_PORT: Ethernet103/1/13)

2011 Nov 5 01:20:03 BNZ-AKDCM1-L01-DC-02 %ETHPORT-2-IF_DOWN_ERROR_DISABLED: Interface Ethernet103/1/13 is down (Error disabled. Reason:Sufficient free entries are not available in TCAM bank)

ACL Hardware Resource Utilization (Module 9)

--------------------------------------------

Used Free Percent

Utilization

-----------------------------------------------------

Tcam 0, Bank 0 2 16382 0.01

Tcam 0, Bank 1 16173 211 98.71<------Almost maxed out!

Tcam 1, Bank 0 7 16377 0.04

Tcam 1, Bank 1 155 16229 0.94

LOU 3 101 2.88

Both LOU Operands 2

Single LOU Operands 1

LOU L4 src port: 0

LOU L4 dst port: 1

LOU L3 packet len: 0

LOU IP tos: 0

LOU IP dscp: 0

LOU ip precedence: 0

LOU ip TTL: 0

TCP Flags 0 16 0.00

Protocol CAM 3 4 42.85

Mac Etype/Proto CAM 7 7 50.00

Non L4op labels, Tcam 0 35 6108 0.56

Non L4op labels, Tcam 1 1 6142 0.01

L4 op labels, Tcam 0 0 2047 0.00

L4 op labels, Tcam 1 1 2046 0.04

Ingress Dest info table 1 511 0.19

Egress Dest info table 0 512 0.00

BNZ-AKDCM1-L01-DC-02# show system internal access-list feature bank map interface ingress module 9

_________________________________________________________________________

Feature Rslt Type T0B0 T0B1 T1B0 T1B1

_________________________________________________________________________

PACL Acl X

RACL Acl X

DHCP Acl X

QoS Qos X

PBR Acl X

Netflow Acl X X X

Netflow Sampler Acc X

SPM WCCP Acl X X X

BFD Acl X

SPM OTV Acl X

FEX Acl X

SPM CBTS Acl X X

SPM LISP INST Acl X X

BNZ-AKDCM1-L01-DC-02# show system internal access-list feature bank map interface egress module 9

_________________________________________________________________________

Feature Rslt Type T0B0 T0B1 T1B0 T1B1

_________________________________________________________________________

QoS Qos X

RACL Acl X

Tunnel Decap Acl X

Netflow Acl X X

Netflow Sampler Acc X

Rbacl Acl X X

CTS implicit Tunnel Acl X

SPM WCCP Acl X

SPM OTV Acl X

SPM LISP Acl X

SPM ERSPAN (termination) Acl X X X

nkarpysh · ‎11-05-2011

Hi Ben,

are you using wccp? I have seen similar with it as it makes TCAM load be calculated with exponential values.

Let's say, we have the following:

X = amount of access-list entries

Y = the amount of bits in the WCCP DST IP MASK advertised (for example:

0.0.23.65 [00000000.00000000.00010111.01000001] = 6 bits).

Then the amount of TCAM entries will be:

X * 2 tcp ports * 2^Y * 2 = normal amount of TCAM entries.

For example if we have:

X = 60 access-list lines.

Y = 6 bits

This then gives a total of: 60 * 2 * 2^6 * 2 = 15360.

Make sure to minimize the following entries:

* Amount of ACL entries.

* Reduce the MASK send by and configured on the WCCP appliance.

as seen in show ip wccp vrf default detail

M A S K

Source Destination Sport Dport

------ ----------- ----- -----

0x00000000 0x00001741 0x0000 0x0000

Make sure to minimize the following entries:

* Amount of ACL entries.

* Reduce the MASK send by and configured on the WCCP appliance.

as seen in show ip wccp vrf default detail

M A S K

Source Destination Sport Dport

------ ----------- ----- -----

0x00000000 0x00001741 0x0000 0x0000

Nik

HTH,
Niko

View solution in original post

nkarpysh · ‎11-05-2011

Hi Ben,

are you using wccp? I have seen similar with it as it makes TCAM load be calculated with exponential values.

Let's say, we have the following:

X = amount of access-list entries

Y = the amount of bits in the WCCP DST IP MASK advertised (for example:

0.0.23.65 [00000000.00000000.00010111.01000001] = 6 bits).

Then the amount of TCAM entries will be:

X * 2 tcp ports * 2^Y * 2 = normal amount of TCAM entries.

For example if we have:

X = 60 access-list lines.

Y = 6 bits

This then gives a total of: 60 * 2 * 2^6 * 2 = 15360.

Make sure to minimize the following entries:

* Amount of ACL entries.

* Reduce the MASK send by and configured on the WCCP appliance.

as seen in show ip wccp vrf default detail

M A S K

Source Destination Sport Dport

------ ----------- ----- -----

0x00000000 0x00001741 0x0000 0x0000

Make sure to minimize the following entries:

* Amount of ACL entries.

* Reduce the MASK send by and configured on the WCCP appliance.

as seen in show ip wccp vrf default detail

M A S K

Source Destination Sport Dport

------ ----------- ----- -----

0x00000000 0x00001741 0x0000 0x0000

Nik

HTH,
Niko

ben.tan · ‎11-24-2011

Hi,

No, I am not running WCCP. Here is the solution by using the "hardware access-list resouce pooling" command.

##Enable TCAM resoruce sharing##

hardware access-list resource pooling module 9

hardware access-list resource pooling module 10

## Reload the module to take effect##

reload module 9

reload module 10

##Verify with this commands##

show system internal access-list globals module 9

sh hardware access-list resource utilization module 9

show system internal access-list globals module 10

sh hardware access-list resource utilization module 10