Recommended value for 7600 IPv4 FIB TCAM?

[Shawn Kleinart]

I'm simply asking if there is a "best practices" or widely accepted value people have settled on?

I understand that I'm asking a broad question, the best answer is "it depends" -- 
and each circumstance may have different values and needs.

Quick run-down of our network:
IPv6 is enabled but non-production, but we'd like the full IPv6 table by end of year.  No multicast and no need.

Full IPv4 table, multihomed, and private peering.  Some 75 ASN's we peer with.  No multi-path, no plans for it.

RSP720-3CXL

We're hoping to get another 4-5 years from the hardware, we're no where near fabric/backplane capacity.  We do VoIP, small packets, lower bps, higher pps.

We're at 95% with threshold warnings.  default value of 512k

I saw some discussion and people seem to be settling on 640k.

Any considerations not to use 640k?

show commands:

#show mls cef maximum-routes     
FIB TCAM maximum routes :
=======================
Current :-
-------
 IPv4 + MPLS         - 512k (default)
 IPv6 + IP Multicast - 256k (default)

#show mls cef su

Total routes:                     498430
    IPv4 unicast routes:          496353
    IPv4 Multicast routes:        4     
    MPLS routes:                  2049  
    IPv6 unicast routes:          22     
    IPv6 multicast routes:        5     
    EoM routes:                   0 

#

FIB TCAM usage:                     Total        Used     %Used
    72 bits (IPv4, MPLS, EoM)      524288      498420     95%
   144 bits (IP mcast, IPv6)      262144            33      1%

[gareth.bryan · Accepted solution]

We settled with the below as "standard" across out 7600 estate.

#show mls cef maximum-routes 

FIB TCAM maximum routes :
=======================
Current :-
-------
 IPv4 + MPLS         - 736k (default)
 IPv6                - 128k 
 IP Multicast        - 16k 

 Module              FIB TCAM usage:                     Total        Used     %Used
   4                     72 bits (IPv4, MPLS, EoM)      753664      540607     72%
                        144 bits (IP mcast, IPv6)      147456        17731     12%

[bhnegi]

Here is good write up on this subject.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/117712-problemsolution-cat6500-00.html

https://supportforums.cisco.com/document/12202206/size-internet-global-routing-table-and-its-potential-side-effects

Best Regards,

Bheem

[Shawn Kleinart]

I probably should have referenced doc ID 117712 in my original post, as I had just read that and it sort of prompted my question to begin with.

The doc only talks about the 1M IPv4 routes, and uses only the 1000k value in its example (useless to me).  It does nothing to discuss a suggested (real-world) value (I don't fault it, they don't want to try to predict the routing table growth).

It also does not suggest there are any other consolidations to factor when choosing your value (eg, memory block sizes, whereas if I chose X value, is space wasted?)  Or, is there ANY (engineering/technical) difference in using 704 vs 736 vs 768.

But it's good to know how that the full IPv6 table is 12% of 128k memory allocation.

I suppose I could have just posted, "hey all, what do you predict the IPv4 and IPv6 global tables will be on Jan 1 of 2018"

... I was reading a bunch of stuff on predictions... people in 2008 thought we'd hit 750k+ by now (500k in 2010).  One recent prediction, but more accurately a projection, was ~1M routes in 2020.  Looking at predictions from < 2009, which is only 5 years ago, they are funny... in how off most of them are.

[Shawn Kleinart]

Thanks Gareth, this is helpful.

I don't have a live SUP720/RSP720 in my lab right now to test with, but it'll be back online in a couple weeks...
what hardware was your output from?  IOS?

(we're running 12.2(33)SRE6 on RSP720-3CXL)
... I ask because my output is:
 IPv4 + MPLS         - 512k (default)
 IPv6 + IP Multicast - 256k (default)

and yours:
 IPv4 + MPLS         - 736k (default)
 IPv6                - 128k 
 IP Multicast        - 16k 

... the two totals differ (768 vs 864 or 880).
Which is confusing to me as the 3CXL's support 1M routes, so thats 960k - 992k of memory.

Anyway, using the formula of:  V * 1024 = routes (v being TCAM value)
 ... and I can't imagine how or why a bit bounday (eg, 32, 64, 128) would be relevant, but I don't undertand the TCAM memory, design or hardware... 32 (bits) * 20 = 640k, 21 = 672, then 704, 736... and using 128, you get 512, then 640, then 768.
... I suspect I am way over-thinking it.  (but want to see it in a lab I guess)

It seems all about predicting IPv4 and IPv6 tables... 

640k = 655,360   # I see that happening, but won't try to guess how soon (less than 4 years I'd imagine)
736k = 753,664   # that's more reasonable to last > 4 years
768k = 786,432

What will IPv6 grow to... and this is where I guess I need my lab to see what happens... as from your output 128k = 147,456
(but, 128*1024 = 131,072. My original output of 256k = 262,144 , which is 256*1024)

Which is simply curous, as 147k vs 131k is somewhat trivial, I'd guess the IPv6 table will remain under 100k for more than 5 years.  So 128k is fair.  In fact, depending on what I see in my testing, 768k (786k routes) for IPv4 and as little as 65k Ipv6 routes is ok with me.

So at least I talked myself out of the 640 value (as 655k Ipv4 routes is not enough).

[gareth.bryan]

"... the two totals differ (768 vs 864 or 880)."

TCAM entries are 72 bits, IPv4 and MPLS (inc EoMPLS) use a single entry - IPv6 and Multicast  use 2.

All adds up :)

As a side note, I can highly recommend reading "BRKSPG-2772"  (Anatomy of Internet Routers)

I think you may find it interesting reading.

[Shawn Kleinart]

FYI:

# source: https://www.nanog.org/sites/default/files/monday.general.huston.BGP.21.pdf

IPv4 BGP Table Size predictions

           Linear                  Exponential

2013 441,172 entries

2014 488,011 entries

2015 540,000 entries 559,000

2016 590,000 entries 630,000

2017 640,000 entries 710,000

2018 690,000 entries 801,000

2019 740,000 entries 902,000

v6 table..

2013 11,600 entries

2014 16,200 entries

2015 25,400 entries 19,000

2016 38,000 entries 23,000

2017 57,000 entries 27,000

2018 85,000 entries 30,000

2019 127,000 entries 35,000

....

other predictions of IPv6 in 2018 are >100k  (these are the high end of predictions)

... but if people treat IPv6 like they do IPv4 as far as routing policy, traffic engineering, and there is a adoption and move to IPv6 (when IPv4 actually runs out there will be some movement), then anything can happen.

 

[Shawn Kleinart]

FYI to all,

I settled on a 768k allocation....

768k IPv4 = 802,816 routes

120k IPv6 = 122,880   (system default)

(config: mls cef maximum-routes ip 768)