Solved: Enabling and Verifying CEF load-balancing

SJ K · ‎05-16-2016

Hi all,

From the document and table below

http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-software-releases-120-mainline/47205-cef-whichpath.html

Incoming Interface	Outgoing Interface	Switching Method
CEF	Process	CEF
Process	CEF	Fast
Process	Fast Switching (IP route cache)	Fast Switching
CEF	Fast Switching	CEF

Let's say I have ip cef turn on, but have disabled them on all interfaces with "no ip route-cache cef" except for these 3 interfaces below.

port 1 , 2 and 3.

Port 1 is use for ingress traffic and Port 2 and Port 3 are connected to different routers and are use to loadbalance for egress traffic.

data --> port1 ---> port 2 or port 3.

If I want to use ip cef, according to the table above, I should turn on IP cef on the ingress which is port 1

=================

q1) If I wanted to use packet-base load sharing , should ip packet-load sharing be configured on the ingress (port1) or egress (port 2,and 3) ?

q2) How do I verify that my packets are CEF processed/switch ? Should I use

a) show ip int stats on the ingress (port 1) ?

or

b) show ip int stats on the egress (port 2 and 3) ?

if it is a), should I look at "Pkt In" or "Pkt Out" under "Route cache" ?

Been looking for an answer high and low with no avail, hope gurus here can shed some light.

Best Regards

Mark Malone · ‎05-25-2016

Hi

If cef is working correctly and I rarely have seen it not its actually good stable feature that seems to work without much interaction and when cef is enabled globally each interface should use it unless it cant be cef switched as specified in the document like destined for router or encapsulated traffic, depending on the traffic flow/path src and dst the router/switch will decide what's ingress/egress.

I wouldn't get to hung up on the actual show int stats as longs as you see them incrementing and not stalled completely in route-cache/processor or you may have an issue with cef in general

In the cef statistics the punting will increment but should not be by large amounts and that's normal you will see punts, what you need to look out for is there not being dropped continuously then that would be an issue.

The verification steps part 3 in that doc are good indicator of what to look for when cef is working correctly ,but usually once all your routes are in the show ip cef , the drop counters are generally clean in statistics and you can see increments in your sh int stats you shouldn't have an issue but if its something you have to make sure , setup the test verification conditions in a lab from the document that's probably the best way to prove it.

Its difficult to tell exactly on a production network what exactly is being cef switched and not as it doesn't let you get that granular in show commands and the paths are bidirectional and unfortunately wireshark doesn't show any flag or anything like that to indicate cef switched to distinguish per packet basis for troubleshooting that way

what I haven't tested is what does the debug ip cef packets gig x/x provide or the debug ip cef local , that may give you a bit more information. If I get a chance on my lab later il do it see what i get ,cant run debug in prod as a test :)

View solution in original post

Mark Malone · ‎05-16-2016

Hi CEF should be enabled on all interfaces unless for some reason you want to process switch which will increase cpu its just best practice to use it so I would avoid turning it off on ports unless you really need to in real prod network anyway

Personally I have only set the load balancing on egress interfaces because that's where you would have the multiple egress routes pointing outbound where the LB would occur , saying that I have never set cef to packet share on ingress I would need to test it to see what the actual results would be and I have read a lot of device will only support per destination LB on ingress , if there's only 1 ingress there's no load balancing anyway on that interface and yes that's where I check that I can see packets incrementing in cef I use the show int stats and look under route-cache packets in and out as they increment constantly , im not a 100% what the characters in an out use is for.I also use the show ip cef switching statistics to make sure im not punting everything to cpu , there will always be some packets go to cpu whether its unsupported encapsulations , qos or some other feature which may cause it , you just wnat to make sutre the majority are going through cef

SJ K · ‎05-16-2016

Hi Mark,

As usual, thanks again for sharing your thoughts and understanding, as well as the setup that you have.

saying that I have never set cef to packet share on ingress I would need to test it to see what the actual results

q1) do you mean you have ip cef on all interfaces (default) and setup ip load-sharing per packet only on the egress interfaces - right ?

The reason for my confusing is as the table/documentation from cisco as shown in the earlier thread above. It states that cef forwarding decision is done on ingress interface - but did not state further on where load-sharing should be.

and yes that's where I check that I can see packets incrementing in cef I use the show int stats and look under route-cache packets in and out as they increment constantly ,

q2) do you mean that you issue the "show int stats" on the ingress interface to see if packets are cef switched ?

a) for "Pkt Ins" under route cache, i would assume that incoming packets to the interface are CEF processed, but what about "Pkts Out" ?

or should we monitor for "Pkts In" under route-cache in ingress interface and

"Pkts Out" under route-cache for egress interface ?

Looking forward to hearing your thoughts.

Regards,
Noob

SJ K · ‎05-16-2016

Hi Mark,

Still there ?

Regards,

Noob

Mark Malone · ‎05-17-2016

Hey

1 Yes personally I would never disable cef on an interface as its a better method of switching traffic for the device , less resourceful than process switching , and if I was using load sharing yes I would setup on my WAN egress ports as if I had 2 default routes say and I wanted that type of LB that's where I would personally put it facing outbound to the ISP

2 Had to check this to be sure its been a while going that deep into cef

This is an int my network actually interconnect port , its an egress port packets came into it process switched,Packets going out route cache cef switched

GigabitEthernet1/0/1
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor     192658   26243342          0          0
             Route cache          0          0    1597482 115616976
                   Total     192658   26243342    1597482 115616976

But then when I check its ingress port of this device which would be the g1/0/11 you will see ingress cef switched and process switched but egress cef only , so its a bit of both

One thing to remember just because cef is on an you want it on as its useful not every feature uses it correctly , some don't support the encapsulation types they will be your unsupported cef counter and sometimes depending on qos setup you can negate cef and make traffic be process switched

GigabitEthernet1/0/11
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor     633823   66408679          0          0
             Route cache      13480     955534     884163   85974069
                   Total     647303   67364213     884163   85974069

Like below my switch has recievd most likely from riverbed and other switch behind it traffic it just cant put in cef and also a few unsupported packets

IPv4 CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP 0 0 8 0 1919001 5 0 0

SJ K · ‎05-24-2016

Hi Mark,

So sorry for the late reply and thanks for sharing your statistics.

1 thing i can't seem to fully grasp is that

a) cef forwarding is decided in the ingress interface (as stated in the cisco documentation link above).

b) majority of my traffic are 1 way- with packets going into ingress interface from 1 subnet and out of the egress interface to another subnet.

If i wanted to be assured that my outbound traffc are CEF switched,

q1) does that means that, for the ingress interface; during sh int stats, i would need to see a constant jump or higher statistics of "route-cache" switched packets then compared to "processor" under the "Pkts In" column ?

or should i just check CEF statistics and make sure that no packets (or not much of them) are being punted ?

Regards,
Noob

Mark Malone · ‎05-25-2016

Hi

If cef is working correctly and I rarely have seen it not its actually good stable feature that seems to work without much interaction and when cef is enabled globally each interface should use it unless it cant be cef switched as specified in the document like destined for router or encapsulated traffic, depending on the traffic flow/path src and dst the router/switch will decide what's ingress/egress.

I wouldn't get to hung up on the actual show int stats as longs as you see them incrementing and not stalled completely in route-cache/processor or you may have an issue with cef in general

In the cef statistics the punting will increment but should not be by large amounts and that's normal you will see punts, what you need to look out for is there not being dropped continuously then that would be an issue.

The verification steps part 3 in that doc are good indicator of what to look for when cef is working correctly ,but usually once all your routes are in the show ip cef , the drop counters are generally clean in statistics and you can see increments in your sh int stats you shouldn't have an issue but if its something you have to make sure , setup the test verification conditions in a lab from the document that's probably the best way to prove it.

Its difficult to tell exactly on a production network what exactly is being cef switched and not as it doesn't let you get that granular in show commands and the paths are bidirectional and unfortunately wireshark doesn't show any flag or anything like that to indicate cef switched to distinguish per packet basis for troubleshooting that way

what I haven't tested is what does the debug ip cef packets gig x/x provide or the debug ip cef local , that may give you a bit more information. If I get a chance on my lab later il do it see what i get ,cant run debug in prod as a test :)

SJ K · ‎05-28-2016

Hi Mark,

Thanks for your reply and sorry for the slow response.

The problem i am facing now is that my ingress interface is a VLAN interface.. and from show int stats, most of my "Pkts In" are "Processor" switched, there is little jump in the "Route Cache" portion.

CEF is turn on in the switch and all interfaces but i am still unable to verify whether packets ingressing that SVI are really CEF switched or not ;(

Regards,
Noob

Joseph W. Doherty · ‎05-17-2016

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

As Mark has noted, generally CEF should always be enable if supported. Some later IOS features depend on CEF being enabled.

If I wanted to use packet-base load sharing

Why might you want to do that?

It's true, packet-based load sharing will generally optimally use multiple links, but it can lead to packet re-ordering, and many, many applications are adversely impacted when that happens.

A possible alternative to improve load sharing, would be to use the PfR feature, if supported.

SJ K · ‎05-24-2016

Hi Joseph,

Thanks for replying and sorry for the late response.

I used to have 2 links ( still having them ), but i do not have alot of src-dest pair.

The bulk of my traffic is actually just between 1 src and destination.

Hence, i used to set it up as per-packet load sharing and is achieving higher throughout (despite the out of order packets).

Right now, using just per-destination sharing, i am achieving a much lower throughput (30%-40%) lesser.

Regards,

Noob