netflow vs performace

ROBERTO TACCON · ‎11-27-2008

Need to utilize the netflow feature on a BGP peering internet service provider data center.

Need to use the netflow for analyze the customers IP subnets:

- on customer routed interfaces (FULL DUPLEX fast ethernet or Gigabit ethernet interfaces)

- or I can filter the netflow for specif stream (like ANY to CUSTOMER IP addresses).

The routers are Cisco Catalyst 6500 / 7600 with supervisor:

WS-X6K-SUP2-2GE Catalyst 6000 supervisor 2

WS-SUP720-3BXL Supervisor Engine 720

Need to see ALL the "raw packtes" routed with netflow feature for speciuc customers IP subnet.

Questions :

1) WHICH IS THE BEST SOLUTION/APPROACH TO DO IT ?

2) In a enviroment where I have 5 or 10 or 20 Gbps of throughput on the same router can I use the MLS harware netflow feature WITH the netflow filtering solution to see ALL the "raw packets" whithout lost any one ?

3) If I use the MLS hardware netflow feature can I see ALL the "raw packets" (or i can lost some streams !) ?

4) It's possibile to configure the netflow on hardware (MLS NETFLOW) BEFORE the IP sterms/flow hit the PFC without missing any packets ?

I know how to filter netflow AFTER when I configure the NDE (keeping CPU cycles to a minimum on the Control Plane CPU Router) with "Packet-based NetFlow Flow Sampling" and/or "flow filters"

Packet-based NetFlow Flow Sampling:

http://www.cisco.com/en/US/docs/routers/7600/ios/12.2SXF/configuration/guide/nde.html#wp1148270

I can also use flow filters to limit the flows being exported:

http://www.cisco.com/en/US/docs/routers/7600/ios/12.2SXF/configuration/guide/nde.html#wp1140829

Regards

Roberto Taccon

Giuseppe Larosa · ‎11-27-2008

Hello Roberto,

some considerations:

with 5 to 20 Gbps of routed traffic the device needs to work using MLS: it will be CEF based on both platforms.

So I think that in any case Neflow will be executed during the MLS operation.

Actually you configure netflow on the MSFC just to drive the MLS netflow accounting during MLS operation.

In this case you are exposed to the MLS netflow table finite size:

see Table 50-3

http://www.cisco.com/en/US/docs/routers/7600/ios/12.2SXF/configuration/guide/netflow.html#wp1106378

this means 128K rows for Sup2/PFC2, up to 256k rows on SUP7203BXL.

these are not so high numbers when compared to the traffic volume you say to appear.

So the risk to miss some flows is not negligible and can be small only in a data center when monitoring inter-servers flows like DB synchronizations.

On the other end the export filters you can configure doesn't prevent the undesired flows to use space in the netflow table: they are thought to reduce the cpu burden in building the export packets with the same logic that has driven the introduction of flow cache router aggregation on router platforms: filtering or aggregating data locally can be a way to reduce the number of accounting packets to be generated.

So these filters doesn't provide protection from table size limits.

Netflow router aggregation can provide some help because the flow aggregation cache is another table hosting aggregated data.

You wrote of customer ip subnets so some form of aggregation can be used to achieve this level of granularity.

"When you configure NetFlow aggregation on the MSFC, it is configured automatically on the PFC and DFCs (see the "Configuring NetFlow Aggregation on the PFC" section)."

http://www.cisco.com/en/US/docs/routers/7600/ios/12.2SXF/configuration/guide/netflow.html#wp1146734

This is a difficult challenge as you have already stated.

sampling :

In 12.2(18)SXF and later releases, the MSFC supports NetFlow sampling for software-routed traffic.

but you cannot SW route 5 to 20 Gbps so it isn't a viable option.

Hope to help

Giuseppe

ROBERTO TACCON · ‎12-01-2008

Thanks for your reply.

May I ask you if the following it's the correct command on IOS to check if there're MLS netflow table misses ?

"sh mls netflow table-contention detailed"

cat6500SUPII#sh mls netflow table-contention detailed

Earl in Module 1

Netflow Entry Creation is Enabled. Maximum number of flows is 128K

Detailed Table Contention Level Information

===========================================

Layer 3

-------

L3 Contention Level: 4

Page Hits on Virtual Page 1 = 58435

Page Hits on Virtual Page 2 = 30962

Page Hits on Virtual Page 3 = 20480

Page Hits on Virtual Page 4 = 14700

Page Hits on Virtual Page 5 = 12722

Page Hits on Virtual Page 6 = 8830

Page Hits on Virtual Page 7 = 6467

Page Hits on Virtual Page 8 = 4505

Page Misses = 82668

gix-sw65-0pd1#

gix-sw65-0pd1#sh mls netflow usage

Netflow table usage notification enabled at 85% every 300 seconds

Netflow table utilization of module 1 is 44%

gix-sw65-0pd1#

Thanks again.

RT.

Giuseppe Larosa · ‎12-01-2008

Hello Roberto,

if you sum all the entries in the first show for the hits you get a number that is roughly 157,000 that is 61% of 256,000 I'm afraid these counters are accumulated over time.

Because the other show says usage is 44%.

However, there are 82,668 misses that should be flows that didn't find place in the table.

you can compare the output of

sh mls netflow usage

and

sh mls netflow count

they can give you a more immediate measure of table usage.

I see from the output that you have enabled the mls netflow usage notify with a threshold of 85 and frequency of 300 seconds so you can get messages that tell you when the table is near to full.

This is a wise measure.

Aggregation caches should be investigated to see if they offer some way to collect data.

To be noted this should be configurable at the same time as main flow table.

Hope to help

Giuseppe

ROBERTO TACCON · ‎12-02-2008

Thanks again