The Size of the Internet Global Routing Table and Its Potential Side Effects

Document

Nov 26, 2014 6:35 AM
May 12th, 2014

Since the early 1990s, we’ve watched as the number of entries on the Internet routing table has steadily grown. It wasn’t that long ago (2008) that the table reached 256k routes, triggering action by network administrators to ensure the continued growth of the Internet. Now that the table has passed 500,000 routes, it’s time to start preparing for another significant milestone – the 512k mark.
 

Looking Ahead to 512k

As an industry, we’ve known for some time that the Internet routing table growth could cause Ternary Content Addressable Memory (TCAM) resource exhaustion for some networking products. TCAM is a very important component of certain network switches and routers that stores routing tables. It is much faster than ordinary RAM (random access memory) and allows for rapid table lookups.
 

Networking Product Implications

No matter who provides your networking equipment, it needs to be able to manage the ongoing growth of the Internet routing table. We recommend confirming and addressing any possible impacts for all devices in your network, not just those provided by Cisco. The products that could be affected include those with a default configuration supporting 512k routes. From Cisco’s perspective, this includes:

  • Cisco Catalyst 6500 Switches
  • Cisco 7600 Series Routers
  • Cisco ASR 9000 Series Aggregation Services Routers configured with Trident-based line cards (typhoon-based line cards are not affected)
  • Cisco ASR 1000 Series Aggregation Services Routers with 4GB (devices with 8GB or RAM or higher can scale to up to 1,000,000 routes)

The Good News – Workarounds Are Available!

Cisco has published information on several workarounds that can be applied by our customers, including changing the default configuration for affected devices. In some cases this may require a reload of the device or line card. See below for the links to this customer information.

Cisco Catalyst 6500/Cisco 7600 Series Supervisor Engine 720

The following document describes how to customize the forwarding information base (FIB) ternary content addressable memory (TCAM) on Catalyst 6500 switches that run the Supervisor Engine 720:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116132-problem-catalyst6500-00.html

This guidance is specific to the Supervisor Engine models SUP720-3BXL and SUP720-3CXL. The “non-XL” versions do not support more than 256,000 IPv4 routes.

The following documents include additional information about the Cisco Catalyst 6500 and the Cisco 7600 Series Supervisor Engine 720 capabilities, respectively.

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6500-series-switches/product_data_sheet09186a0080159856.pdf

http://www.cisco.com/c/en/us/products/collateral/routers/7600-series-routers/product_data_sheet0900aecd8057f3b6.html
 

Cisco ASR 9000 Series Aggregation Services Routers

The following document describes workarounds available for the Cisco ASR 9000 Series Aggregation Services Routers. When a Trident-based line card reaches its prefix limit, the message %ROUTING-FIB-4-RSRC_LOW occurs, causing potential traffic loss on the line cards:

http://www.cisco.com/c/en/us/support/docs/routers/asr-9000-series-aggregation-services-routers/116999-problem-line-card-00.html

Cisco ASR 1000 Series Aggregation Services Routers

Cisco ASR 1000 Series Aggregation Services Routers with 4GB can scale to up to 500,000 IPv4 or IPv6 routes. Cisco ASR 1000 Series Aggregation Services Routers with 8GB of RAM or higher can scale to up to 1,000,000 routes. The following document provides an overview of the number of supported routes:

http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/data_sheet_c78-441072.html

 

Additional Workarounds

Route filtering and the use of a default route can also be used to decrease the number of routes in an affected device. Prefix lists can be used as an alternative to access lists in many BGP route-filtering commands. The use of prefix lists provides significant performance improvements when loading and performing route lookup of large routing tables. Additional information about BGP best practices and configuring prefix lists is available at:

http://www.cisco.com/web/about/security/intelligence/protecting_bgp.html#8

 

Security Considerations

The possibility of TCAM resource exhaustion at 512k routes is a known issue that we all know has been coming for some time. There is no related security vulnerability, and it cannot be easily triggered by a remote, untrusted user.

The following website is a great resource that provides the current state of the Internet routing table. This could help Cisco customers when configuring route filtering:

Implementing the recommended workarounds ahead of time will help your network avoid any performance degradation, routing instability, or impact to availability. Having just passed the 500,000 route milestone, now is the right time to ensure your network is prepared to manage a 512k entry internet routing table.

Overall Rating: 0 (0 ratings)
sean_evershed Mon, 05/12/2014 - 22:15

I thought the ASR1K was impacted by this as well? I guess however it is just a simple memory upgrade.

According to this link if you only have 4Gbps it will only support 500,000 IPv4 routes or 500,000 IPv6 routes. It states that you need to upgrade to 8Gbps if you want to scale beyond this limit.

http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/data_sheet_c78-441072.html

Edited by S. Evershed

Omar Santos Tue, 05/13/2014 - 09:28

Hi Sean,

You are correct. The Cisco ASR 1000 Series Aggregation Services Routers with 4GB can scale to up to 500,000 IPv4 or IPv6 routes. The ASR routers with 8GB or RAM or higher can scale to up to 1,000,000 routes. I have included additional references to the document.

Thank you!

Omar

 

 

 

matthew.taylor Wed, 08/20/2014 - 20:58

Omar,

Definitive answers we've gotten the BU indicate that the ASR 1000 is not affected where a minimum ESP-10 is deployed along with RP1 and 4GB RAM. This configuration will support 1M IPv4 routes. ESP-5 will however only support 500,000 IPv4 routes. The ASR1001 will support 1M routes if it has ESP5 and 8GB of RAM but not with 4GB.

We've been advised that the router will experience memory allocation errors at some point beyond these limits.

Matt

Omar Santos Tue, 08/26/2014 - 11:03

The number of supported routes in the ASR 1000 series is determined by the combination of both the RP and the ESP. When the system has an RP1 + ESP5, the limit is 500k (because of the ESP5).

When the system runs an RP1 + ESP10, the limit is 1 million routes (both RP1 and ESP10 reach the limit), and so on.  The following is the Cisco ASR 1000 Series Embedded Services Processors (ESP) Data Sheet, which lists the limits of each ESP model:

http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731640.html?cachemode=refresh 

hchien Wed, 08/20/2014 - 19:26

Could someone point me out how to workaroud on GSR?

matthew.taylor Wed, 08/20/2014 - 20:54

You don't need one. The GSR is not limited to 512,000 routes.

stphsu Wed, 08/20/2014 - 22:30

But customer may still asking: what's the Internet routing size should my GSR  with different hardware version(GRP, PRP/PRP2, E3/E5 LC...) need be upgraded?

(600k, 1M...?)

I can only find a GSR memory recommendation url as following:

http://www.cisco.com/en/US/products/hw/routers/ps167/prod_bulletin09186a0080186fd5.html

For networks with more than 200,000 routes, Cisco recommends using the Performance Route Processor with at least 1 GB of memory and configuring all linecards with the maximum amount of route memory available.

"200k route" wording is very old Internet scale and it is apprecated if a more updated version can be published.

mkazantsev Wed, 11/26/2014 - 06:35

Could someone provide the information related to NEXUS 7000's (SUP1 and/or SUP2E) limitations?

 

Thank you in advance.

Actions

Login or Register to take actions

This Document

Posted May 12, 2014 at 12:08 PM
Updated May 13, 2014 at 9:27 AM
Stats:
Comments:8 Overall Rating:0
Views:23842 Contributors:6
Shares:87
Tags: No tags.