OSPF, CEF, and equal cost links

branfarm1 · ‎02-05-2009

Hi there.

I have two sites that are connected by two p2p DS3's. In each site I have a pair of 4500 switches, and each DS3 has a router at both ends. My setup looks like this:

Switch 1A -- Router1A -DS3- Router2A - Switch 2A

Switch 1B -- Router1B -DS3- Router2B -- Switch 2B

Each pair of switches run HSRP to act as the default gateway for all hosts on Vlan100. Right now I have my DS3's sitting in the same VLAN as the rest of the hosts. The DS3 routers also run HSRP. All devices are part of OSPF in the same area.

When I look at my routing tables I see that OSPF has installed two equal-cost routes to the network on the other side, however, utilization on the links is consistently 2-1. I haven't changed any of the default load-balancing settings, so it should still be using per-destination balancing. Also, since all of the routers are off the same Routed interface, I can't see any stats when I use 'show ip cef x.x.x.x internal'. Do I have a misconfiguration here? Should I be using different routed endpoints for the DS3's on each end?

Thanks,

Brandon

Elly Bornstein · ‎02-05-2009

4500s do not support per packet CEF load balancing in hardware, only per destination. So you could get 2:1 traffic ratio depending on the traffic flows.

Not to mention you have some polarization effect due to hsrp. Meaning you only use the per destination lookup on the HSRP active if the traffic is sourced from a LAN segment.

Lets say traffic is flowing from a host directly connected on sw1a to a host on sw2a. sw1a receives all traffic for this flow since it is the HSRP active (in this case). None of this traffic goes to sw1b. sw1a has 2 equal cost paths, but it only uses one for all packets of this flow since the destination IP for this flow never changes.

So in the end, it really depends on the traffic flows going across the 4500 since it is routing and only does per flow destination CEF load balancing on equal cost paths.

I believe the command to find out the exact path a flow would take on 4500 equal cost paths is something like...

'show mls cef lookup ...'

I know this is the command on the 6500, not sure of 4500.

branfarm1 · ‎02-05-2009

Thanks for the quick reply. I tried the 'show mls ... ' but that isn't a command on the 4500.

What would you recommend here? I don't have a lot of individual flows going across the links, which I'm assuming makes it more likely to see an uneven distribution. Can I change something to make the utilization more even though?

Thanks

Elly Bornstein · ‎02-05-2009

well you really just need to make it so that BOTH sw1a and sw1b are routing and even load of the traffic.

Either switch from HSRP to GLBP (caveat here is you have to have good vlan design for GLBP), see this doc:

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a00807d2520.shtml#dc1

This might be a good option for you if all your devices are directly connected to the 4500s and it might cause more utilization of the link between the 4500s.

Otherwise, you can keep HSRP and add more standby groups and make your DHCP server give out different default gateway addresses. One active on sw1a and one active on sw1b. That way, potentially, half the traffic going across the DS3s is being routed by sw1a and have by sw1b.

Sorry about the lookup command, I know there is an equivalent one somewhere. I will try to find it.

branfarm1 · ‎02-05-2009

Thanks again. Making sure Sw1a and sw1b and routing an even load will be difficult. Most of my servers are dual-homed to each switch (no end users to worry about), and I'm starting to build out blade systems with internal switches. So my sw1a and sw1b are playing a dual core/access role at this point. I'll read over that doc and see what it entails.

Thanks again