Solved: OSPF Not load balancing

Brandon Baker · ‎11-05-2013

We currently have 3 ISPs, the third just being added last week, When I set it up under OSPF in our Core switch it will not load balance, Our current defrault route in the core is going to ISP2 and when I add our new connection in as a default route it stelas all of the traffic from from ISP2 instead of load balancing. Can anyone please assist?

All 3 have same Costs as well

Richard Burts · ‎11-05-2013

Thank you for posting this output. Though I am afraid that it did not shed much light for me about the issue. So let me take a slightly different approach. Am I correct that before you added ip route 0.0.0.0 0.0.0.0 X.20.128.139 200 you had only this single static default route ip route 0.0.0.0 0.0.0.0 x.20.128.1 200 ? And that with this single static default route it was achieving load sharing/load balancing?

If that is the case then it suggests that the load sharing/load balancing was really being done on 128.1 and that perhaps we should be looking to this device to understand what causes the issue.

HTH

Rick

HTH

Rick

View solution in original post

Richard Burts · ‎11-05-2013

There is not enough information here for us to know what is the cause or to be able to give you much assistance. For one thing we would need to know how each of the ISP is connected to you. And we would need to see details of how you have configured OSPF. And we would benefit from seeing the output of show ip ospf database to see how each ISP is advertising routes to you. From your description it seems that the new ISP is advertising a route to you with a metric better than the existing ISP. And if the metric is better then OSPF chooses the single best route and does not use both when one is better than the other.

HTH

Rick

HTH

Rick

Brandon Baker · ‎11-05-2013

Each ISP is connected through Fiber and BGP. ISP1 and ISP3 do have the same router-id as well.

ISP1 into asr1006

gigabitethernet from asr to catalyst 6509 in Vlan10

ISP2 into 7200VXR

gigabitethernet from vxr to catalyst 6509 in Vlan10

ISP3 into asr1006 (new connection)

gigabitethernet from asr to catalyst 6509 in Vlan3 (new connection)

core-6509#show ip ospf database

OSPF Router with ID (X.X.128.22) (Process ID 1)

Router Link States (Area 0.0.0.1)

Link ID ADV Router Age Seq# Checksum Link count

x.x.129.2    x.x.129.2    19          0x80005CBE 0x0001A0 6 (CMTS)
X.X.128.1    X.X.128.1    1954        0x80005010 0x005CA6 1 (ISP2)
X.X.128.21   X.X.128.21   708         0x80001016 0x00E220 1 (BACKUP CORE)
X.X.128.22   X.X.128.22   1461        0x80007525 0x00D4F6 4 (CORE)
X.X.128.253 X.X.128.253 1457        0x80000F0A 0x003B11 2 (ISP1 and ISP3)

Net Link States (Area 0.0.0.1)

Link ID ADV Router Age Seq# Checksum
x.x.128.22 X.X.128.22 1461 0x800025AA 0x006F48

Type-5 AS External Link States

Link ID ADV Router Age Seq# Checksum Tag

0.0.0.0         X.X.128.1    1954        0x8000096F 0x007953 1 (ISP2)
0.0.0.0         X.X.128.22   697         0x80000104 0x00C0FB 1 (CORE)
0.0.0.0         X.X.128.253 445         0x8000071E 0x0034EE 1 (ISP1 and ISP3)

*other routes directing to core*

Neighbor ID Pri State Dead Time Address Interface

x.x.128.253 0 FULL/DROTHER 00:00:33 x.x.128.139 Vlan3 (ISP 3Orig.Pri 1 and BDR- I man. changed to Pri 0)

x.x.129.2 0 FULL/DROTHER 00:00:37 x.x.128.12 Vlan10 (CMTS)

x.x.128.1 1 FULL/DROTHER 00:00:31 x.x.128.1 Vlan10 (ISP2)

x.x.128.21 1 FULL/BDR 00:00:39 x.x.128.21 Vlan10 (BACKUP CORE)

x.x.128.253 1 FULL/DROTHER 00:00:32 x.x.128.2 Vlan10 (ISP1)

Brandon Baker · ‎11-05-2013

More info:

router ospf 1

log-adjacency-changes

redistribute connected

network x.x.128.0 0.0.0.127 area 0.0.0.1 (ISP1 and 2 fall under this category)

network x.x.128.136 0.0.0.7 area 0.0.0.1 (ISP3 falls under this)

maximum-paths 6 (just added this today but not help)

default-information originate always

Richard Burts · ‎11-05-2013

Thanks for the additional information. Can you also post the output of show ip ospf database external?

HTH

Rick

HTH

Rick

Brandon Baker · ‎11-05-2013

Sure thing:

core-6509#show ip ospf database external

OSPF Router with ID (x.x.128.22) (Process ID 1) (CORE)

Type-5 AS External Link States

LS age: 615
Options: (No TOS-capability, DC)
LS Type: AS External Link
Link State ID: 0.0.0.0 (External Network Number )
Advertising Router: X.X.128.1 (ISP2)
LS Seq Number: 80000971
Checksum: 0x7555
Length: 36
Network Mask: /0
        Metric Type: 2 (Larger than any link state path)
        TOS: 0
        Metric: 1
        Forward Address: 0.0.0.0
        External Route Tag: 1

LS age: 1348
Options: (No TOS-capability, DC)
LS Type: AS External Link
Link State ID: 0.0.0.0 (External Network Number )
Advertising Router: X.X.128.22
LS Seq Number: 80000105
Checksum: 0xBEFC
Length: 36
Network Mask: /0
        Metric Type: 2 (Larger than any link state path)
        TOS: 0
        Metric: 1
        Forward Address: X.X.128.1 (ISP2)
        External Route Tag: 1

LS age: 1118
Options: (No TOS-capability, DC)
LS Type: AS External Link
Link State ID: 0.0.0.0 (External Network Number )
Advertising Router: X.X.128.253 (ISP1 AND ISP3)
LS Seq Number: 8000071F
Checksum: 0x32EF
Length: 36
Network Mask: /0
        Metric Type: 2 (Larger than any link state path)
        TOS: 0
        Metric: 1
        Forward Address: 0.0.0.0
        External Route Tag: 1

*many more listed but assuming the default route is the only thing we are working with*

Richard Burts · ‎11-05-2013

Thanks for the additional information. Based on what I thought I understood about the symptoms you describe I thought that there would be some difference in the external routes being advertised as default route (and you are right that while there may be many more external route entries it is the default entries that I wanted to see). But it looks like metric type and metric value are the same. So I am puzzled. Or maybe I do not have a correct understanding of what the issue is.

So perhaps you can provide some clarification about the problem? And perhaps it might help if you would post the output from show ip route (especially the default route parts) and perhaps some information about the configuration of the ASR1006 and the 7200 for OSPF?

HTH

Rick

HTH

Rick

Brandon Baker · ‎11-05-2013

Basically I had 1 gigabit ethernet connection coming from our VXR to the catalyst, 1 connection cominf from our asr to the catalyst and they were load sharing. I have since tried adding another connection from the asr to the catalyst and whenver I set the defualt route with the same metric as the connection going to the vxr it steals all of the traffic from that so I do not have any traffic coming into the catalyst through the VXR and it is all going through both ports of the ASR. (The ISP1 does not have a static route set up in the catalyst so I am unsure if this may be helpful information or not)

Gateway of last resort is X.X.128.1 to network 0.0.0.0

S* 0.0.0.0/0 [200/0] via x.x.128.1

[200/0] via x.x.128.139

ASR:

router ospf 1

network X.X.128.0 0.0.0.127 area 0.0.0.1

network X.X.128.136 0.0.0.7 area 0.0.0.1

default-information originate

VXR:

router ospf 1

log-adjacency-changes

network X.X.128.0 0.0.0.127 area 0.0.0.1

default-information originate

Does the VXR need to have the network info for the new connection?

Richard Burts · ‎11-05-2013

In OSPF the network statement is used to determine which interfaces to include into the OSPF process. If the VXR does not have any interfaces in X.X.128.136 then there is no need to have that network statement.

Perhaps I am a little closer to understanding the issue. Am I correct in understanding that your catalyst has static default routes configured (and is using them rather than the advertised OSPF routes)? Would you post the static routes as they were configured with 2 and worked and how you configured the third route when you attempted to add it (and then all traffic went that way)?

HTH

Rick

HTH

Rick

Brandon Baker · ‎11-05-2013

B X207.0/24 [200/3] via X.67.17, 12:37:44

B X63.0/24 [200/5] via X.67.17, 5d21h

B X46.0/24 [200/5] via X.67.17, 5d21h

B X215.0/24 [200/5] via X.67.17, 5d21h

B X21.0/24 [200/5] via X67.17, 5d21h

X0.0/8 is variably subnetted, 11 subnets, 4 masks

B X46.0/24 [200/5] via X.67.17, 5d21h

B X.47.0/24 [200/5] via X.67.17, 5d21h

B X.41.0/24 [200/5] via X.67.17, 5d21h

B X58.0/24 [200/5] via X.67.17, 5d21h

B X.56.0/23 [200/5] via X.67.17, 5d21h

B X56.0/21 [200/5] via X.67.17, 5d21h

B X54.0/24 [200/6] via X67.17, 5d21h

B X11.0/24 [200/3] via X67.17, 5d21h

B X0.0/18 [200/2] via 66.X.17, 5d21h

B X.26.0/24 [200/5] via X.67.17, 5d21h

B X34.0/24 [200/3] via X.67.17, 05:18:09

B X182.0/24 [200/5] via X.67.17, 5d21h

B X.2.0/24 [200/3] via X67.17, 5d21h

C X15.0/24 is directly connected, Vlan50

X0.0/8 is variably subnetted, 11 subnets, 4 masks

B X228.0/22 [200/5] via X.67.17, 5d21h

B X224.0/19 [200/2] via X67.17, 5d21h

B X242.0/24 [200/7] via X67.17, 5d21h

C X248.0/24 is directly connected, Vlan1

B X0.0/23 [200/7] via X67.17, 5d21h

B X.214.0/24 [200/3] via X.67.17, 5d21h

B X109.0/24 [200/7] via X67.17, 5d21h

B X240.0/23 [200/5] via X.67.17, 5d21h

B X240.0/22 [200/5] via X.67.17, 5d21h

B X93.0/24 [200/3] via X.67.17, 5d21h

B X.43.0/24 [200/7] via X67.17, 5d21h

X0.0/32 is subnetted, 2 subnets

S X.132.194 is directly connected, Null0

S X132.206 is directly connected, Null0

X0.0/8 is variably subnetted, 4 subnets, 3 masks

B X52.0/22 [200/3] via X67.17, 5d21h

B X19.0/24 [200/3] via X67.17, 12:37:45

B X18.0/24 [200/3] via X67.17, 12:37:45

B X.76.0/23 [200/3] via X.67.17, 5d21h

B X51.0/24 [200/5] via X67.17, 5d21h

B X34.0/24 [200/5] via X67.17, 5d21h

B X57.0/24 [200/5] via X67.17, 5d21h

X0.0/8 is variably subnetted, 7 subnets, 4 masks

B X.134.0/23 [200/5] via X67.17, 5d21h

S X.67.105/32 [1/0] via X128.1

S X67.17/32 [1/0] via X.128.1

B X.94.0/24 [200/5] via X67.17, 5d21h

B X.64.0/24 [200/5] via X.67.105, 5d21h

B X64.0/19 [200/2] via X.67.105, 5d21h

B X.75.0/24 [200/5] via X67.17, 5d21h

C 192.168.130.0/24 is directly connected, Vlan400

B X94.0/24 [200/3] via X67.17, 5d21h

X0.0/16 is variably subnetted, 15 subnets, 4 masks

C X.140.0/24 is directly connected, Vlan420

C X141.0/24 is directly connected, Vlan410

C X142.0/24 is directly connected, Vlan36

C X.143.0/24 is directly connected, Vlan39

C X136.0/24 is directly connected, Vlan301

C X.137.0/24 is directly connected, Vlan302

C X.0/24 is directly connected, Vlan303

C X139.0/24 is directly connected, Vlan304

C X135.0/24 is directly connected, Vlan300

C X128.0/24 is directly connected, Vlan20

C X130.0/24 is directly connected, Vlan60

S X131.0/24 [1/0] via X123.40

O E2 X129.2/32 [110/1] via X128.12, 03:43:14, Vlan10

O E2 X.0.0/18 [110/1] via X128.12, 03:43:14, Vlan10

B X0.0/16 [200/0] via X.128.12, 5d21h

X0.0/20 is subnetted, 1 subnets

B X.128.0 [200/5] via X.67.17, 5d21h

X0.0/8 is variably subnetted, 3 subnets, 3 masks

B X.224.0/20 [200/3] via X.67.17, 5d21h

B X.238.0/24 [200/3] via X.67.17, 5d21h

B X.172.0/22 [200/3] via X67.17, 5d21h

C X.131.0/24 is directly connected, Vlan401

B X95.0/24 [200/3] via X.67.17, 5d21h

B X.62.0/24 [200/5] via X.67.17, 5d21h

C 192.168.21.0/24 is directly connected, Vlan30

X.0.0/8 is variably subnetted, 6 subnets, 3 masks

B X.172.0/23 [200/3] via X.67.17, 4d10h

O X.28.0/22 [110/2] via X.128.12, 03:43:16, Vlan10

O X.24.0/22 [110/2] via X.128.12, 03:43:16, Vlan10

O X.20.0/22 [110/2] via X128.12, 03:43:16, Vlan10

O X.16.0/22 [110/2] via X128.12, 03:43:16, Vlan10

B X.16.0/21 [200/0] via X128.1, 5d21h

C 192.168.20.0/24 is directly connected, Vlan25

B X.22.0/24 [200/3] via X.67.17, 5d21h

X.0.0/8 is variably subnetted, 6 subnets, 2 masks

O E2 X0.0/16 [110/1] via X128.12, 03:43:16, Vlan10

C X.0.0/24 is directly connected, Vlan101

C X0.0/16 is directly connected, Vlan102

C X0.0/16 is directly connected, Vlan100

C X.0.0/16 is directly connected, Vlan23

C X.0.0/16 is directly connected, Vlan40

C X6.0/24 is directly connected, Vlan2

X0.0/32 is subnetted, 1 subnets

S X.136.121 is directly connected, Null0

X135.0/26 is subnetted, 2 subnets

C X.135.64 is directly connected, Vlan37

C X.135.0 is directly connected, Vlan35

X.0.0/32 is subnetted, 1 subnets

S X.4.32 is directly connected, Null0

C X.129.0/24 is directly connected, Vlan7

B X.43.0/24 [200/5] via X.67.17, 5d21h

X.0.0/8 is variably subnetted, 15 subnets, 4 masks

O E2 X.102.0/24 [110/1] via X128.12, 03:43:17, Vlan10

S X212.153/32 [1/0] via X128.2

S X.51.97/32 [1/0] via X.128.2

O E2 X39.0/24 [110/1] via X128.12, 03:43:17, Vlan10

B X33.0/24 [200/0] via X128.12, 5d21h

C X.121.0/25 is directly connected, Vlan5

C X123.0/24 is directly connected, Vlan13

B X.197.0/24 [200/3] via X.67.17, 5d21h

B X.208.0/21 [200/3] via X.67.17, 5d21h

B X.211.0/24 [200/3] via X.67.17, 5d21h

B X.25.0/24 [200/3] via X67.17, 5d21h

B X39.128/25 [200/0] via X.128.12, 5d21h

O E2 X.33.128/25 [110/1] via X128.12, 03:43:17, Vlan10

B X112.0/24 [200/3] via X.67.17, 5d21h

B X225.0/24 [200/3] via X.67.17, 5d21h

B X.119.0/24 [200/3] via X.67.17, 5d21h

X0.0/8 is variably subnetted, 4 subnets, 2 masks

B X12.0/22 [200/3] via X.67.17, 5d21h

B X.214.0/23 [200/3] via X.67.17, 5d21h

B X188.0/23 [200/3] via X67.17, 5d21h

B X160.0/22 [200/3] via X67.17, 5d21h

X128.0/24 is variably subnetted, 2 subnets, 2 masks

C X128.0/25 is directly connected, Vlan10

C X.128.136/29 is directly connected, Vlan3

B X.42.0/24 [200/5] via X.67.17, 5d21h

X131.0/24 is variably subnetted, 2 subnets, 2 masks

C X.131.0/24 is directly connected, Vlan11

S X.131.215/32 [1/0] via X.128.1

C X130.0/24 is directly connected, Vlan6

B X18.0/24 [200/5] via X.67.17, 5d21h

S 0.0.0.0/0 [200/0] via X.128.1

O X.140.0/22 [110/2] via X.12, 03:43:17, Vlan10

B X.196.0/22 [200/5] via X.67.17, 05:18:08

B X.162.0/23 [200/5] via X.67.17, 5d21h

O E2 X.136.0/22 [110/1] via X.12, 03:43:17, Vlan10

B X.0.0/21 [200/5] via X.67.17, 5d21h

B X12.0/22 [200/5] via X.67.17, 05:18:08

B X.72.0/21 [200/5] via X.67.17, 05:18:08

B X.144.0/20 [200/3] via X.67.17, 5d21h

B X.60.0/23 [200/5] via X67.17, 5d21h

B X.44.0/23 [200/5] via X.67.17, 5d21h

B X128.0/20 [200/0] via X128.12, 5d21h

B X.184.0/21 [200/5] via X.67.17, 5d21h

This is how everyhting is set up currently and load balancing between the 2 connections, the only difference that I make when I try to use the 3rd connection is this:

S* 0.0.0.0/0 [200/0] via X.128.1

[200/0] via X.128.139

I hope this info helps

Richard Burts · ‎11-05-2013

There is a lot of information here. I had not understood that there was also BGP routing in your network but see a lot of what appear to be BGP learned routes. I do see some routes learned via OSPF. Would you do

show run | include ip route

on your catalyst and post the output?

And then give us the exact syntax that you used when you attempted to add the third default route?

HTH

Rick

HTH

Rick

Brandon Baker · ‎11-05-2013

core-6509#show run | include ip route

ip route 0.0.0.0 0.0.0.0 x.20.128.1 200

ip route x.90.212.153 255.255.255.255 x.20.128.2

ip route x.126.51.97 255.255.255.255 x.20.128.2

ip route x.118.67.17 255.255.255.255 x.20.128.1

ip route x.118.67.105 255.255.255.255 x.20.128.1

ip route x.47.136.121 255.255.255.255 Null0

ip route x.253.4.32 255.255.255.255 Null0

ip route x.16.131.0 255.255.255.0 x.20.123.40

ip route x.142.132.194 255.255.255.255 Null0

ip route x.142.132.206 255.255.255.255 Null0

ip route x.20.128.240 255.255.255.252 x.20.128.246

ip route x.20.128.248 255.255.255.252 x.20.128.254

ip route x.20.131.0 255.255.255.0 x.20.128.1

ip route x.20.131.215 255.255.255.255 x.20.128.1

I added this syntax to add the third route:

ip route 0.0.0.0 0.0.0.0 X.20.128.139 200

Richard Burts · ‎11-05-2013

Thank you for posting what I requested. It does help me understand some parts but still leaves me puzzled about what is going on. Perhaps if you post the output of show ip protocol from the catalyst it might help explain some aspects.

What I see in the output that you posted is a single static default route which is configured with an administrative distance which is usually done to create a floating static route. I do not understand how this single static default route could have achieved load sharing (or load balancing - though this term is perhaps less technically accurate). And it looks like your attempt to add the other static default route did work. And I do not understand how the result of this would have been that all traffic used that route. I would have expected that some traffic would have used the next hop of 128.1 and that some traffic would have used the next hop of 128.139.

HTH

Rick

HTH

Rick

Brandon Baker · ‎11-05-2013

core-6509#show ip protocol
*** IP Routing is NSF aware ***

Routing Protocol is "ospf 1"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID X.20.128.22
It is an autonomous system boundary router
Redistributing External Routes from,
    connected
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 6
Routing for Networks:
    X.20.128.0 0.0.0.127 area 0.0.0.1
    X.20.128.136 0.0.0.7 area 0.0.0.1
Routing Information Sources:
    Gateway         Distance      Last Update
    X.16.129.2         110      05:07:36
    X.20.128.1         110      6d00h
    X.20.128.2         110      13w3d
    X.20.128.253       110      4d07h
Distance: (default is 110)

Routing Protocol is "bgp 40098"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
IGP synchronization is disabled
Automatic route summarization is disabled
Neighbor(s):
    Address          FiltIn FiltOut DistIn DistOut Weight RouteMap
    X.20.128.1
    X.20.128.2
Maximum path: 1
Routing Information Sources:
    Gateway         Distance      Last Update
    X.20.128.1         200      05:07:43
    X.20.128.2         200      1d05h
Distance: external 20 internal 200 local 200

Richard Burts · ‎11-05-2013

Thank you for posting this output. Though I am afraid that it did not shed much light for me about the issue. So let me take a slightly different approach. Am I correct that before you added ip route 0.0.0.0 0.0.0.0 X.20.128.139 200 you had only this single static default route ip route 0.0.0.0 0.0.0.0 x.20.128.1 200 ? And that with this single static default route it was achieving load sharing/load balancing?

If that is the case then it suggests that the load sharing/load balancing was really being done on 128.1 and that perhaps we should be looking to this device to understand what causes the issue.

HTH

Rick

HTH

Rick