Netflow Configuration

Unanswered Question
Feb 20th, 2007

Hi all,

I am trying to configure netflow on a 6509 w/ Sup 720 3BXL to send accounting data to a management server. I have netflow configured (see below) and "ip route-cache flow" set on all of my major interfaces. Netflow data is being sent to the management server, however I am only seeing about 100 Packets/s when I should be seeing in the 100,000s Packets/s range (additionally we are pushing over a gigabit of traffic and netflow reports barely anything).

What I am missing here?

Thanks in advance,

Max

ip flow-cache entries 131072

ip flow-cache timeout active 5

mls rp ip

mls ip multicast flow-stat-timer 9

mls aging fast time 30 threshold 64

mls aging long 900

mls aging normal 32

no mls netflow

mls flow ip interface-full

no mls flow ipv6

mls nde sender version 5

no mls acl tcam share-global

mls cef error action freeze

ip flow-export source Loopback0

ip flow-export version 5 origin-as

ip flow-export destination x.x.x.x 9995

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 1 (1 ratings)
Loading.
kyawzawhtut Tue, 02/20/2007 - 20:24

Hi

It is because of the CEF. Only the first few packets go to yr Sup Engine for routing and the following packets bypass and do switching.

That is the reason why you do not see the full picture. If you really really must see the full picture, you may like to disable CEF which is not recommended.

HTH.

Rate if help.

Cheers

maxclark Tue, 02/20/2007 - 20:27

Hi,

Disabling CEF is definately not a direction that I can take - is there seriously no way to get full accounting out of this device?

-Max

kyawzawhtut Tue, 02/20/2007 - 21:51

Hi Max

According to my experience, the answer is No.

Normally, I would monitor NetFlow traffic at Router instead.

Cheers

kurenyshev Wed, 02/21/2007 - 00:15

I have a Cat 6506 with Supervisor Engine 32. I have the same problem. My collector doesn't show full picture of trafic. Tell me which will consequences be if I turn off IP CEF?

And which option should I remove ?

Cat6505(config)#no ip cef ?

accounting Enable CEF accounting

distributed Distributed Cisco Express Forwarding

event-log CEF event log commands

interface CEF linecard commands

linecard CEF linecard commands

load-sharing Load sharing

nsf Set CEF non-stop forwarding (NSF) characteristics

table Set CEF forwarding table characteristics

traffic-statistics Enable collection of traffic statistics

kyawzawhtut Wed, 02/21/2007 - 17:12

Hi

You cannot disable CEF on C6500 series according to configuration guide. It is permanently enabled. But in other mid-range switches, you can enable/disable CEF.

I am not sure whether you can run both CEF and MLS togeter as both technology are similar in nature and yet CEF is better in term of performance.

Why don't you try to get NetFlow data from another distribution layer or WAN edge instead of core layer? Just my $0.02!!

Cheers.

Jan Nejman Wed, 02/21/2007 - 01:58

Hello,

don't disable CEF!!! Enable mls netflow option and set mls nde export.

mls flow ip interface-full

mls nde sender version 5

mls netflow

That's all. This enables sending of flows that are "switched" by supervisor. In your configuration is exported only the first packet of the flow! You can also enable exporting inter vlan netflow export (but be carefully when you enable it, bacause it can send a huge number of netflow exports...).

Have a nice day,

Jan Nejman

Caligare Co.

http://www.caligare.com

kurenyshev Wed, 02/21/2007 - 20:08

Device's config:

ip flow-cache entries 10000

ip flow-cache timeout active 1

ip flow ingress layer2-switched vlan 198

mls ip multicast flow-stat-timer 9

mls aging long 300

mls aging normal 120

mls flow ip interface-full

no mls flow ipv6

mls nde sender version 5

mls sampling time-based 64

no mls acl tcam share-global

mls cef error action freeze

..

system flowcontrol bus auto

..

ip flow-export source GigabitEthernet2/1

ip flow-export version 5

ip flow-export destination 10.0.2.2 9996

..

Also It has a lot of VLAN interfaces. Every such interface has a record: ip route-cache flow

In additional, Cat has a few interfaces with subinterfaces. For example: there are Giga2/1 and Giga2/2 with a lots of subinterfaces. Giga2/1 and Giga2/2 have a record:

ip route-cache flow

But I haven't seen a full statistics as before.

kurenyshev Wed, 02/21/2007 - 20:12

Also, I have interface Vlan15 with record ip route-cache flow, but I have never seen traffic of this interface, although, other VLAN interfaces present in statistics.

What is wrong ?

P.S. ip cef works

maxclark Thu, 02/22/2007 - 08:38

Jan,

This definately improved the situation - however it looks like I am still getting truncated data (i.e. packets shows in the 200-300k range and when I look at the interface I am doing 800k+ inbound & outbound). Does the Sup aggregate or truncate data in a way I should be aware of?

Thanks,

Max

Jan Nejman Thu, 02/22/2007 - 10:56

Hello,

can you send me your interface configuration? Which command do you using to get packet utilization on your interface and which software (or cisco command) do you using to collect netflow information?

Jan Nejman

Caligare Co.

http://www.caligare.com

maxclark Thu, 02/22/2007 - 11:04

Jan,

I am using "show interface x" to get the interface statistics (also when we compare to snmp via cacti the netflow numbers are low). I am running nfsen (so netflow v. 1.5.2) on a FreeBSD box as my collector. Nfcapd command below as well.

Thanks,

Max

/usr/local/bin/nfcapd -w -D -I router01 -p 9995 -u www -g www -B 200000 -l /usr/local/var/nfsen/profiles/live/router01 -P /usr/local/var/nfsen/run/router01.pid -x /usr/local/bin/nfprofile -q -p /usr/local/var/nfsen/profiles -s router01 -r %d/%f

#show interfaces po 2

Port-channel2 is up, line protocol is up (connected)

Hardware is EtherChannel, address is 0007.b355.5800 (bia 0007.b355.5800)

Internet address is x.x.x.x/30

MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,

reliability 255/255, txload 0/255, rxload 0/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s

input flow-control is off, output flow-control is on

Members in this channel: Gi7/7 Gi7/8

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:14, output 00:00:14, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 854510

Queueing strategy: fifo

Output queue: 0/40 (size/max)

1 minute input rate 1823532000 bits/sec, 969988 packets/sec

1 minute output rate 1714853000 bits/sec, 901492 packets/sec

L2 Switched: ucast: 41946 pkt, 2774528 bytes - mcast: 1996 pkt, 161608 bytes

L3 in Switched: ucast: 169986842127 pkt, 33679708457144 bytes - mcast: 0 pkt, 0 bytes mcast

L3 out Switched: ucast: 158253133760 pkt, 37291080296637 bytes mcast: 0 pkt, 0 bytes

169937081618 packets input, 33669816565275 bytes, 0 no buffer

Received 15232 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

158222255755 packets output, 37284177837380 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Jan Nejman Thu, 02/22/2007 - 11:57

Hello,

can nfsen program log how many netflow exports were received? If yes, compare 'show mls nde' counters with nfsen logs. Maybe nfsen lost some packets(flows). I think, that your configuration is OK at this time.

You will see always more packets via SNMP, because in the SNMP there are all layer2 packets. In the netflow there are "only" L3 packets... some L2 traffic can be lost (i.e. arp requests etc...), but this can be 5% at maximum.

Jan Nejman

Caligare Co.

http://www.caligare.com

Im running pretty much the same thing as maxclark...

- Supervisor Engine 720 (Active) WS-SUP720-3B

- cisco WS-C6509-E (R7000) processor

nfcapd/nfsen as the collcter box.

The one thing i cant figure out is that all the flows I am collecting are reporting with a ifindex pointing to a vlan interface. What I am looking for is per interface stats.

For example all traffic passing through GigabitEthernet2/27=Ifindex=51 but none of the flows are reporting any traffic from or to interface 51. I have tried multiple configs. anything special that needs to be done? Can the 6509-E give pre interface stats.

Jan Nejman Tue, 02/27/2007 - 07:50

Hello,

did you enable ip route-cache flow on all L3 interfaces?

Run command: "show ip interface brief | exclude unassigned "

and check if you enable netflow for it.

Have a nice day,

Jan Nejman

Caligare Co.

http://www.caligare.com

Thanks for the help..

Ip flow ingress is enabled on all vlan interfaces.

I am recieving flows from all vlan interfaces . When i run the command "show ip interface brief | exclude unassigned" The output is vlan interfaces only and yes they are netflow enabled.

The flows i want are flows from specific interfaces. IE. how much port 80 traffic from "interface GigabitEthernet1/7". How do i enable netflow on specific interfaces? Is it possible to recieve flows for these interfaces? I have tried different combinations of netflow versions and collecters, nothing seems to work.

Shane Gaumond

steve.busby Tue, 02/27/2007 - 13:20

Try

ip flow ingress layer2-switched vlan

ip flow export layer2-switched vlan

You don't mention (or I skimmed over it) what IOS Version you are running. With PFC3B or PFC3BXL running 12.2(18)SXE or higher you need these two commands to enable NDE for all traffic within the specified VLANs rather than just inter-VLAN traffic.

HTH

Steve

IOS version 12.2(18)SXF6 about to go to 12.2(18)SXF7..

I had the IP flow ingress layer2-switched commands in place and I was recieving all traffic within vlans as well as inter-vlan traffic.

The problem I have is that the info collected pertains only to VLAN interfaces. The ifindex #'s being sent to the collecter are those of the vlan interfaces. What i want is the ifindex's to be matched to the physical interfaces. for example Vlan 83 runs across 3 trunk ports setup on int 4/5 4/7 and 2/3. I am recieving vlan 83 data but the ifindex #'s of the flows dont match to these interfaces the ifindex being reported for all vlan 83 traffic is the ifindex of "int vlan 83"

I Think i found something. All of my physical interfaces are setup with the switchport command and the command ip flow ingress is not available. If i enter no switchport from an ifconfig the ip flow ingress command becomes available. It seems that the 2 cannot coexist. Perhaps there is some sort of global command or CEF command to enable all interfaces to send netflwo info.

How do i collect netflow stats on "int gi 2/3" without using the ip flow ingress command??

Thanks for all the help...

Shane Gaumond...

Clarification of my issue...

I have a server connected to port 3/4 with ip 192.168.56.9.

I am recieving flows with the IP 192.168.56.9 both as source and destination..lots of info.

The problem is that the flows have an ifindex matching to "int vlan 56". What im looking for is that the ifindex matches to "int gi 3/4"

We also run netflow on another 6006 chassis. Hybrid OS...Cat OS for the switching and IOS for the routing. Netflow reporting from this chasis is very good. The CatOS reports MLS Switched info with ifindex's matching to physical ports such as 2/3 or 4/5. The IOS only reports Routed info with ifindex's matching vlan interfaces.

I'm trying to duplicate the setup on the 6509 chasis. Is it possible??

Thanks for the imput....

Shane Gaumond

avmabe Thu, 03/22/2007 - 06:33

Your config is only capturing traffic that hits the CPU.

You need to turn on MLS Netflow to catch "hardware switched" traffic

Here is some output.

sh running-config full | include mls

mls ip multicast flow-stat-timer 9

mls aging long 64

mls aging normal 55

mls netflow usage notify 80 120

mls flow ip interface-full

no mls flow ipv6

mls nde sender

no mls acl tcam share-global

mls cef error action freeze

I dont know if the above helps but from the config prompt I have entered

mls netflow

I have trouble believing that the large amounts of flows and traffic I am recieving are only what hits the CPU. The Router is reporting all the flows even MLS/CEF I would expect it too but the flows are reporting Vlan interfaces not physical interfaces.

Actions

This Discussion