GLBP not working. Always showing state as UP

Answered Question
Aug 24th, 2009

Guys,

I have two 2611XM routers and I am using GLBP on the inside.

For the outside, I only have one single Public IP and I have this IP configured on both router;s outside interfaces. I am using the same mac-address on both routers outside interfaces and I get to have redundancy without requiring HSRP (as it needed 3 IPs and the ISP only provided with one... I could've installed a 3rd router and put it behind the ISP and then connect the two 2611xm behind the router so this way I could use HSRP on the outside interfaces using some other private Ips… but I didn't have a 3rd router)

This setup works perfect. The problem now is when I introduced GLBP on the inside interfaces. I did this so I could deploy only one single default gateway to all internal host so this way I could get redundancy as well as load balancing. During testing, after shutting down the outside interface of router1, no traffic was able to pass to or from the internal hosts.

The strange thing was that after some 5 to 10 minutes, the internal host will re-gain access to the internet. However, after another 5 to 10 minutes, the internal host will lose internet access and was not able to ping anything on the outside again.

When the internal host was unable to see the outside, I started doing some trace routes on the host to see what router it was using and I noticed that every time the host didn't have access to the internet (during testing that is where I manually shut down the outside interface of router1), the host was of cource trying to use the router that had the down internet interface (the outside interface).

My question is WHY is GLBP still trying to make router1 the default gateway for the internal hosts when it cannot route packets to the internet?? Also, when I do a “sh glbp” on router1, it shows that the router state was still active.

I still don't quite get the AVG, AVF, etc., but isnt the GLBP system supposed to place router1 as a “no good” router and not utilize it to repond to ARP requests from the inside hosts?

And also, WHY will my internal host have internet access some minutes, and then 5 to 10 minutes later, will lose it????

Thank you

I have this problem too.
0 votes
Correct Answer by Peter Paluch about 7 years 3 months ago

Hello Angel,

Regarding your unanswered question: Let's simulate it and see the results here.

I have two routers, R1 and R2, on a common LAN. The configuration:

R1:

track 1 interface FastEthernet0/1 line-protocol

!

interface FastEthernet0/0

ip address 10.0.0.1 255.255.255.0

duplex auto

speed auto

glbp 1 ip 10.0.0.254

glbp 1 priority 101

glbp 1 preempt

glbp 1 weighting 100 lower 91 upper 91

glbp 1 weighting track 1 decrement 10

R2:

track 1 interface FastEthernet0/1 line-protocol

!

interface FastEthernet0/0

ip address 10.0.0.2 255.255.255.0

duplex auto

speed auto

glbp 1 ip 10.0.0.254

glbp 1 preempt

glbp 1 weighting 100 lower 91 upper 91

glbp 1 weighting track 1 decrement 10

The configuration is very similar, only R1 has priority 101 while R2 has priority 100. AVG preemption is enabled. That means that the R1 should be elected as AVG, the R2 should be considered a standby AVG. This is confirmed in the output of "show glbp brief":

R1:

R1(config-if)#do show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 101 Active 10.0.0.254 local 10.0.0.2

Fa0/0 1 1 7 Active 0007.b400.0101 local -

Fa0/0 1 2 7 Listen 0007.b400.0102 10.0.0.2 -

R2:

R2(config-if)#do show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 100 Standby 10.0.0.254 10.0.0.1 local

Fa0/0 1 1 7 Listen 0007.b400.0101 10.0.0.1 -

Fa0/0 1 2 7 Active 0007.b400.0102 local -

When looking for AVG information in this output, always look at the line that has a minus sign in the Fwd column. As expected, in this line, R1 is Active and R2 is Standby. Remember that this output talks only about the AVG which is a control element in the GLBP group but does not necessarily forward data.

Then in each output, there are two forwarders (the two lower lines), one for each existing virtual forwarder. Note the forwarder number in the Fwd columns. The State columns describes your state with respect to that forwarder. If it is Active, you are the AVF with that number. If it is Listen, you are just watching if that forwarder dies, and if it does, you might retake its role.

Now when I shutdown the Fa0/1 interface on the R1, after a while, the state will change to:

R1:

R1(config-if)#do show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 101 Active 10.0.0.254 local 10.0.0.2

Fa0/0 1 1 7 Listen 0007.b400.0101 10.0.0.2 -

Fa0/0 1 2 7 Listen 0007.b400.0102 10.0.0.2 -

On R2:

R2#show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 100 Standby 10.0.0.254 10.0.0.1 local

Fa0/0 1 1 7 Active 0007.b400.0101 local -

Fa0/0 1 2 7 Active 0007.b400.0102 local -

The R1 still tells you that it is active but only in the line describing the AVG. Yes, of course, the R1 is still the AVG because its priority has not been changed - it is still 101 and thereby the highest on segment. However, its weight has changed as a result of the tracking, and it has fallen down below the configured threshold. Note that on R1, for all forwarders, the state is Listen which means that the R1 is actually not forwarding packets. Both virtual MAC addresses have been moved to R2 and if you look on R2 you will see that it is Active for both forwarders. It's about reading the show output properly.

I agree completely that of all gateway redundancy protocols, the GLBP is the most complex, and it is poorly documented. By the way, I think it's a pity that it is not implemented on 3560 as they are fairly common in the distribution layer and the GLBP would do nicely on them.

Best regards,

Peter

Correct Answer by Peter Paluch about 7 years 3 months ago

Hello Angel,

In your configuration I do not see any interface tracking for GLBP. This way, even if your outgoing interface goes down, the GLBP does not see it so it can not react.

Interface tracking in GLBP is a little bit more complicated because the GLBP does not directly reference an interface but merely a tracked object. A tracked object can be a SLA object, interface and its Layer1/Layer2 state, interface and its Layer3 state and so on. However, in the simplest configuration, the tracked object will simply be the interface with its L1/L2 state.

Let's suppose for now that your external interface is the Fa0/0 on both routers. Therefore, on both routers, add the following commands in the global configuration mode:

track object 1 interface Fa0/0 line-protocol

interface Fa0/1

glbp 1 weighting 100 lower 91 upper 91

glbp 1 weighting track 1 decrement 10

What this configuration does is that it first defines a track object 1 that includes the Fa0/0 interface and its L1/L2 state. Then, under Fa0/1, you define that

1.) Your current priority is 100. If it falls below 91, you will cease being an AVF. If it then rises above 91, you will reinstate yourself as an AVF.

2.) You are tracking the object 1 (in essence, the Fa0/0) and when the tracked object fails, your priority gets decremented by 10.

The commands "glbp 1 priority" and "glbp 1 preempt" may be left there but they do not influence the AVF selection - they relate only to the AVG elections. Are you familiar with the AVF and AVG functionality?

So give this a try and please let me know if this worked for you.

Best regards,

Peter

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Peter Paluch Tue, 08/25/2009 - 00:33

Hello Angel,

Having the same MAC address on the outside interfaces troubles me - my gut tells me that this is calling for trouble but I don't think for now that it is related to your problems with GLBP.

Can you please post the inside interface configuration of both your routers to see how is your GLBP set up? Also the output of show commands about the GLBP would be useful - when all is well and also when the outside interface is disconnected.

Best regards,

Peter

insccisco Tue, 08/25/2009 - 08:18

:) your gut is sort of right!! it's a long story... but like you said, the issue at hand right now isn't related to it.

My original GLBP config for the inside interfaces of the routers was as follow:

ROUTER-1

interface FastEthernet0/1

ip address 10.221.221.2 255.255.255.0

ip access-group 111 in

ip virtual-reassembly

ip tcp adjust-mss 1452

glbp 1 ip 10.221.221.1

glbp 1 priority 102

glbp 1 preempt delay minimum 15

ROUTER-2

interface FastEthernet0/1

ip address 10.221.221.3 255.255.255.0

ip access-group 111 in

ip virtual-reassembly

ip tcp adjust-mss 1452

glbp 1 ip 10.221.221.1

glbp 1 priority 100

glbp 1 preempt delay minimum 15

With this initial config, I have been having the problems I explained.

I was reading something about the "redirect" times but I am not too sure.

Can you help?

Correct Answer
Peter Paluch Tue, 08/25/2009 - 09:32

Hello Angel,

In your configuration I do not see any interface tracking for GLBP. This way, even if your outgoing interface goes down, the GLBP does not see it so it can not react.

Interface tracking in GLBP is a little bit more complicated because the GLBP does not directly reference an interface but merely a tracked object. A tracked object can be a SLA object, interface and its Layer1/Layer2 state, interface and its Layer3 state and so on. However, in the simplest configuration, the tracked object will simply be the interface with its L1/L2 state.

Let's suppose for now that your external interface is the Fa0/0 on both routers. Therefore, on both routers, add the following commands in the global configuration mode:

track object 1 interface Fa0/0 line-protocol

interface Fa0/1

glbp 1 weighting 100 lower 91 upper 91

glbp 1 weighting track 1 decrement 10

What this configuration does is that it first defines a track object 1 that includes the Fa0/0 interface and its L1/L2 state. Then, under Fa0/1, you define that

1.) Your current priority is 100. If it falls below 91, you will cease being an AVF. If it then rises above 91, you will reinstate yourself as an AVF.

2.) You are tracking the object 1 (in essence, the Fa0/0) and when the tracked object fails, your priority gets decremented by 10.

The commands "glbp 1 priority" and "glbp 1 preempt" may be left there but they do not influence the AVF selection - they relate only to the AVG elections. Are you familiar with the AVF and AVG functionality?

So give this a try and please let me know if this worked for you.

Best regards,

Peter

insccisco Tue, 08/25/2009 - 11:11

It works. I was actually already testing those things but your email is just a confirmation that I was on the right track.

However, this is not to say that I fully get this. I still don't understand why GLBP will not be smart enough to detect that its outside interface is down and thus should not use it and should keep itself away from responding to ARP requests...

I guess GLBP NEEDs the track command to do this?

I think I am confused because nowhere in the documentation it says anything about this. I guess you do really have to play with this to get it.

I am still having a problem though even though it works because when I take down the outside interface of router-1, all internal hosts do start sending their traffic out via the second router, but when I do a "sh glbp" on router-1, it still says that it is the Active one. This is confusing and I beleive is not expected.

I read read about the "redirect" timers and it said that it is a timer used so the router can still send traffic to the failed router.. .even though it has just failed. The default is 600 seconds, which equals 10 minutes and 10 minutes is the time my internal hosts were waiting to have internet connectivity.

Now, when I played with the redirect timer, I did make the traffic go out via the good router.. and I did this without using any tracking.

However, what I have noticed is that GLBP has a lot of paramenters and lots of terminology that one, it is confusing, and 2, it is not behaving the way it is expected (at least in my environment)

For example, what is the preempt used for? What if you use priority? etc

Peter Paluch Tue, 08/25/2009 - 13:19

Hi Angel,

Lots of thoughts and questions - let me see if I can answer some of those.

First of all, do not expect any gateway redundancy protocol like VRRP, HSRP or GLBP to automagically know what is your outside interface and to cease its operation just because the outside interface went away. Neither of these protocols is able to do this automatically. It would also be a bad thing to do because there are scenarios where you do not want to track anything, and on the other hand, there are scenarios when you need to track a series of interfaces. If you need this you have to explicitely configure it because only you know precisely what your requirements are.

I believe that the documentation talks about this - or more precisely, it talks about explicit configuration of object tracking. The documentation does not suggest that these things can be done automatically without tracking.

You have asked about the redirect timer. This is something different. Initially, when members of a GLBP group come online, they will be assigned their virtual MAC addresses by the Active Virtual Gateway (AVG). Each one of these routers that is assigned a virtual MAC address is called the Active Virtual Forwarder (AVF). The AVG will then subsequently respond to ARP requests for the virtual IP address by using all currently assigned virtual MAC addresses in round-robin fashion, thereby distributing the load from individual clients among all the AVFs. However, if a particular AVF disappears or stops functioning, we have a problem here: some stations are using its virtual MAC address as a gateway. You can't afford to simply let this virtual MAC address to disappear, otherwise the clients using this MAC as a gateway would be cut off. In this case, another existing AVF will assume the identity of this virtual MAC address in addition to its "own" virtual MAC (in essence, it will have two virtual MAC addresses) and the client connectivity will not be interrupted. The redirect timers defines the time during which the AVG will respond to ARP queries from stations using this "failed" virtual MAC address. Note that the clients will not be misled - some AVF will take care of serving them.

I think that your success with the redirect timer was merely a coincidence. On your PC, the ARP cache expired and on subsequent query, the AVG directed you to a different virtual MAC. Without an AVF going off, the redirect timeout does not apply, and the AVF will not go off without interface tracking.

A preemption is a permission to take over another device even if it appears functional. In VRRP and HSRP, the preemption allows a router to take over the role of the Active/Master router. The preemption in GLBP is more complicated because you can preempt the AVG and AVF routers independently. The AVG role is not preemptive by default. If you want to enable AVG preemption, you need to use the "glbp preempt" command. However, the AVF role is preemptive by default and it is controlled by the "glbp forwarder preempt" command.

I can imagine all this can be very confusing. I suggest that you identify a couple of things that confuse you the most in GLBP and ask about them. Then proceed to another. This way, we will eventually have the GLBP done ;)

Best regards,

Peter

Giuseppe Larosa Tue, 08/25/2009 - 22:36

Hello Angel,

>> I still don't understand why GLBP will not be smart enough to detect that its outside interface is down and thus should not use it and should keep itself away from responding to ARP requests...

This is the way GLBP works because it is the AVG that answers to ARP requests.

GLBP tracking has the objective to decide the association between AVF and devices in the GLBP group.

For this reason a change on a tracked object makes a router to takes the role of AVG that was of the other one and doesn't change the AVG role.

AVG role changes if there is a failure on the inside interface.

In other words you are looking at GLBP with HSRP in mind but GLBP is different.

Hope to help

Giuseppe

Peter Paluch Tue, 08/25/2009 - 23:03

Hi Giuseppe,

You are completely right here. I originally had the feeling that Angel expected the GLBP to do the tracking work automatically, on its own, automatically discovering the interfaces to track, without having the tracking actually configured.

Best regards,

Peter

insccisco Wed, 08/26/2009 - 02:37

yeah, it looks that I found the hard way. That is exactly what I was expecting...

so this is exactly why router-1, when down, was not passing traffic to the other one... I guess then that the internal hosts that were still trying to use router-1 as as their default gateway (dictated by GLBP) were getting their packets dropped ("destination host unreachable")

I do say then that GLBP is a bit tricky to understand, at first.

Also, although I had done lots of digging, and also with your help, I still had my original question unanswered. My router-1 had the higher priority and during my tests, although I was taking router-1 outside interface down, and although I had already tracking enabled, every time I did a "sh glbp" it was still showing router-1 as Active.

This thing was just itching. I did some more digging, and reached out for help to some of my cisco guys, and they informed me that router-1 was always going to show as the active one, regardless if it was passing traffic or not, because of its higher priority number. Plus, it also had to do with the inside interface of the router, which was never in a down state.

They proofed to me that when I manually shut down the inside interface of router-1, then I was seeing router-2 as the active one. This was indeed even trickier... they advised that the trick is on the forwarders and they showed me to look at this doing the "sh glbp br" command where it showed them as active active.

I perhaps might have understimated GLBP, but at the same time I did read all the documentation about it and I still think that the documentation is somewhat limited and confusing.... it is only after talking to you guys, and talking tomy other cisco friends that I have a better understanding of the protocol....

Correct Answer
Peter Paluch Wed, 08/26/2009 - 23:27

Hello Angel,

Regarding your unanswered question: Let's simulate it and see the results here.

I have two routers, R1 and R2, on a common LAN. The configuration:

R1:

track 1 interface FastEthernet0/1 line-protocol

!

interface FastEthernet0/0

ip address 10.0.0.1 255.255.255.0

duplex auto

speed auto

glbp 1 ip 10.0.0.254

glbp 1 priority 101

glbp 1 preempt

glbp 1 weighting 100 lower 91 upper 91

glbp 1 weighting track 1 decrement 10

R2:

track 1 interface FastEthernet0/1 line-protocol

!

interface FastEthernet0/0

ip address 10.0.0.2 255.255.255.0

duplex auto

speed auto

glbp 1 ip 10.0.0.254

glbp 1 preempt

glbp 1 weighting 100 lower 91 upper 91

glbp 1 weighting track 1 decrement 10

The configuration is very similar, only R1 has priority 101 while R2 has priority 100. AVG preemption is enabled. That means that the R1 should be elected as AVG, the R2 should be considered a standby AVG. This is confirmed in the output of "show glbp brief":

R1:

R1(config-if)#do show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 101 Active 10.0.0.254 local 10.0.0.2

Fa0/0 1 1 7 Active 0007.b400.0101 local -

Fa0/0 1 2 7 Listen 0007.b400.0102 10.0.0.2 -

R2:

R2(config-if)#do show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 100 Standby 10.0.0.254 10.0.0.1 local

Fa0/0 1 1 7 Listen 0007.b400.0101 10.0.0.1 -

Fa0/0 1 2 7 Active 0007.b400.0102 local -

When looking for AVG information in this output, always look at the line that has a minus sign in the Fwd column. As expected, in this line, R1 is Active and R2 is Standby. Remember that this output talks only about the AVG which is a control element in the GLBP group but does not necessarily forward data.

Then in each output, there are two forwarders (the two lower lines), one for each existing virtual forwarder. Note the forwarder number in the Fwd columns. The State columns describes your state with respect to that forwarder. If it is Active, you are the AVF with that number. If it is Listen, you are just watching if that forwarder dies, and if it does, you might retake its role.

Now when I shutdown the Fa0/1 interface on the R1, after a while, the state will change to:

R1:

R1(config-if)#do show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 101 Active 10.0.0.254 local 10.0.0.2

Fa0/0 1 1 7 Listen 0007.b400.0101 10.0.0.2 -

Fa0/0 1 2 7 Listen 0007.b400.0102 10.0.0.2 -

On R2:

R2#show glbp b

Interface Grp Fwd Pri State Address Active router Standby route

Fa0/0 1 - 100 Standby 10.0.0.254 10.0.0.1 local

Fa0/0 1 1 7 Active 0007.b400.0101 local -

Fa0/0 1 2 7 Active 0007.b400.0102 local -

The R1 still tells you that it is active but only in the line describing the AVG. Yes, of course, the R1 is still the AVG because its priority has not been changed - it is still 101 and thereby the highest on segment. However, its weight has changed as a result of the tracking, and it has fallen down below the configured threshold. Note that on R1, for all forwarders, the state is Listen which means that the R1 is actually not forwarding packets. Both virtual MAC addresses have been moved to R2 and if you look on R2 you will see that it is Active for both forwarders. It's about reading the show output properly.

I agree completely that of all gateway redundancy protocols, the GLBP is the most complex, and it is poorly documented. By the way, I think it's a pity that it is not implemented on 3560 as they are fairly common in the distribution layer and the GLBP would do nicely on them.

Best regards,

Peter

Actions

This Discussion