Solved: Cisco 1841 seems to "stop" sending netflow data

mitchen · ‎05-12-2009

I've configured Netflow on my Cisco 1841 router running 12.4(13r)T which is connected via IPSEC VPN to my head office.

My Netflow collector is in the head office and, at first, everything seemed to be ok and the collector was receiving netflow data.

However, then the collector just stopped receiving data for one particular device, my Cisco 1841. (I have other Cisco devices still sending netflow data and being received by the collector without issue - this is the only Cisco 1841 I have though)

However, every now and then, the collector will suddenly start seeing netflow traffic from the device again for a short while then stop. (Of course, I've rebooted the router but this has not helped)

At first I assumed it was a problem with my collector as the Cisco 1841 itself indicated it was still sending the netflow data to my collector:

sh ip flow export

Flow export v5 is enabled for main cache

Exporting flows to **** (****)

Exporting using source interface FastEthernet0/1.1

Version 5 flow records

421593 flows exported in 17461 udp datagrams

0 flows failed due to lack of export packet

6 export packets were sent up to process level

0 export packets were dropped due to no fib

2 export packets were dropped due to adjacency issues

0 export packets were dropped due to fragmentation failures

0 export packets were dropped due to encapsulation fixup failures

However, after setting up a packet sniffer on the collector itself and also using a packet sniffer on my firewall, I've established that the netflow data is NOT actually coming into the collector after all so it does seem to be a router issue.

Anyone had any similar problems with the Cisco 1841 or know how I can resolve this?

Richard Burts · ‎05-12-2009

I wonder if it could be a fragmentation issue with the VPN. If NetFlow builds a "full size" packet and it goes through the VPN, then adding the IPSec header might produce a packet that required fragmentation.

If you have other sites sending NetFlow through VPN tunnels could you check and see whether they handle fragmentation in a way that might be any different from what is done on the 1841?

HTH

Rick

HTH

Rick

View solution in original post

paolo bevilacqua · ‎05-12-2009

Can you update IOS and try again ?

Richard Burts · ‎05-12-2009

Neil

I wonder if the issue may have to do with sending the NetFlow through the VPN. Is it possible that there are periods of time when the VPN tunnel is not active that might account for the missing NetFlow records?

When the collector has been missing NetFlow from the 1841 and begins receiving again, does it get only intermittent NetFlow packets or does it get them steadily for a period of time? If steadily for a period of time, what is the amount of the period of time?

When you did you packet capture, did you notice whether the NetFlow packets might have the DoNotFragment bit turned on?

HTH

Rick

HTH

Rick

mitchen · ‎05-12-2009

Paolo - I will check if I can get another IOS to install but was hoping to leave that as pretty much a last resort (unless it could be established there was some sort of Netflow bug in the particular IOS I'm running)

Rick - yes, I'm convinced this is something to do with sending the Neflow through the VPN though I'm just unsure as to exactly what the problem is. (And it must be something to do with the 1841/this IOS as I have other devices also connected via IPSEC VPN but not exhibiting this problem).

The VPN tunnel is definitely active even when Netflow data is not being recieved as plenty other traffic is being sent over it (in fact, the periods when no netflow data is being received by my collector are far longer than when it does receive netflow data for this particular device)

The interesting thing is, on the few occassions my collector is receiving netflow data from this device, it does seem to be intermittent flow records that are being received.

I've not actually had the packet capture running when the Netflow data was being received (I just kept it running for a few hours to establish that it wasn't being received and my collector was therefore not at fault!)

But that is a good suggestion, I will keep the packet capture running for longer and try to capture some "successful" Netflow data from this device too.

Thanks for the help - any more advice/suggestions will be welcome too!

Richard Burts · ‎05-12-2009

I wonder if it could be a fragmentation issue with the VPN. If NetFlow builds a "full size" packet and it goes through the VPN, then adding the IPSec header might produce a packet that required fragmentation.

If you have other sites sending NetFlow through VPN tunnels could you check and see whether they handle fragmentation in a way that might be any different from what is done on the 1841?

HTH

Rick

HTH

Rick

mitchen · ‎05-18-2009

Rick - you are a genius! That's exactly what the problem was! For some reason, our 1841 had a larger MTU size set than our other routers. I changed the MTU on the 1841 to be the same as our other routers and, right away, I started to get all my netflow data from the router. Thanks very much indeed for your help on this - very much appreciated!

Paolo - yes, I have found similar in the past with IOS bugs. However, for a problem which is more of a "management" issue than service affecting, I'd rather try to get to the bottom of the problem first (unless there is a known bug) and leave the IOS upgrade as a last resort (particularly as there is every chance a new IOS could introduce a different bug which IS service affecting!) As it happens, this was the correct choice here as the problem wasn't IOS related after all. Thanks for your help though.

Richard Burts · ‎05-18-2009

Neil

I am glad that you got your problem resolved and that my suggestion correctly pointed to the issue. Thank you for using the rating system to indicate that your problem was resolved (and thanks for the rating). It makes the forum more useful when people can read about a problem and can know that they will read a response which pointed to the solution.

HTH

Rick

HTH

Rick

paolo bevilacqua · ‎05-13-2009

When working with cisco, never assume there is no bug just because it is not know yet. Many many issues are mysteriously resolved upgrading or downgrading.

It should not be like that, but it is.