EIGRP Authentication FAIL :(

Answered Question
Mar 14th, 2012

A week ago, I changed our network's EIGRP Authentication statements. For the most part, everything worked...MOSTLY

I have a couple areas that are failing, and I'm trying to see about a work around, and also see what the heck is going on.

As I understand EIGRP Authentication, you can have several keys on a "key chain". Let's call our key chain LOSER (sorry, I need to self-punish myself )

DISCLAIMER: NO key-strings were changed..just times

So, the original setup for the key chain was pretty much this :

key chain LOSER

    

key 1

     key-string BLAH BLAH BLAH

          accept-lifetime 00:00:00 Mar 1 1993 00:15:00 Mar 7 2012

          send-lifetime 00:00:00 Mar 1 1993 00:00:00 Mar 7 2012

key 2

     key-string YADA YADA

          accept-lifetime 23:45:00 Mar 6 2012 infinite

          send-lifetime 00:00:00 Mar 7 2012 infinite

So, before the key expired, I changed it to this:

key chain LOSER

key 1

     key-string BLAH BLAH BLAH

          accept-lifetime 00:00:00 Mar 1 1993 00:00:00 Mar 5 2013

          send-lifetime 00:00:00 Mar 1 1993 00:00:00 Mar 5 2013

key 2

     key-string YADA YADA

          accept-lifetime 00:00:00 Mar 5 2012 infinite

          send-lifetime 00:00:00 Mar 5 2012 infinite

Again, key strings not changed.

FIrst, you're going to say "why the 1993 stuff?" Well, some of our sites are in power sensitive areas. We have NTP running. But if the power goes out, which it has, and the UPS fails..what happens? The device comes back up with default time of Mar 1 1993 00:00:00. We've run into that problem before. Changing the authentication time got it up and running again. Also, don't have the luxury of traveling to these sites; it's all or nothing.

So, in our scenario..I goofed up and forgot to change the authentication on a couple devices. So, the clock setting changed..and wham..loss of authentication, hence loss of connection.

So I'm trying to figure out a way to change it back to regain access remotely. Unfortunately, there is NO WAY I can get to these remote sites.

Now here is what I don't get. Let's say you got router A and router B. The interface between the two is trunked. But there's a routed VLAN that's got the new authentication on A, and the old authentication on B. Do the accept and send STATEMENTS have to match exactly? I thought as long as the date/time was correct, and the password matched..good to go. SO, if that's the case, B being the old config KEY 2 has from March 6 2345 to infinite to ACCEPT, and Marcy 7 0000 infinite to send. We are at March 15...the new config Key 2 has from March 5 0000 infinite to send, and March 5 0000 infinite to accept. But there is no routing occurring. EIGRP neighbors are not showing the Router B.

I've tried to just remove the authentication from VLAN 100. No change

I've tried to change it to the OLD config, thinking ok, they'll match on both ends..but no routing update. Of course they won't match Key 1 because it's past March 7, 2012. But key 2 SHOULD allow it! Shouldn't it???

Please help, because I feel like a total loser for this, and the self-flageration is starting to hurt

I have this problem too.
0 votes
Correct Answer by nkarpysh about 2 years 3 weeks ago

Hey

I think you did figured this out. A will always send same key again and again even if another side refusing it. The lowest active key is the only one to send - second key is usually for replacement process and kicking in once key1 is expired.

So possibly worth trying to expire key 1 on A manually (though make sure that other peers will accept key 2 as our will start sending that out based on interface config).

Nik

Correct Answer by nkarpysh about 2 years 3 weeks ago

Hey,

Router can have multiple keys for several reasons:

1. You Different routers to send you different keys, e.g. never to form neighborship between each other. E.G. you have A, B and C on local ethernet domain. If they all are on same subnet - so all will receive their EIGRP hellos. But if you want only A-b and A-c neigborship to work and prevent B-C neighborship - you can make those B and C to send different key number which other does not have in key chain.

E.G. A has key1, key2 and key3 - and sends key1 as it first valid in a  row. B sends key2 but also has key1 to match with A, C sends key3 and also has key1 to match with A. In this case A will form neighborship with B and C, But B and C will not match their case between themselves.

2. It is designed for seamless key rotation - to expire particular keys and start sending/accepting others which you already tried. Configured times should not exactly match - but they need to cover same intervals for router to start sending/accepting correct key at proper time. Eigrp router does not send the life-time specifications to neighbor - those are just locally significant.

It is strange "debug eigrp pac" did not show you a thing. It should show all hello packets between neighbors and also auth messages. Can you check if you have  "no logging monitor" enabled or "no loging console"? and see if those messages appear with "show log".

Here is just test output from this debug:

Here is the debug eigrp packet outputs:


Nov 30 22:59:55.459: EIGRP: received packet with MD5 authentication, key id = 1
Nov 30 22:59:55.463: EIGRP: Received HELLO on FastEthernet0/0 nbr 192.168.12.2
Nov 30 22:59:55.467:   AS 1, Flags 0x0, Seq 0/0 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
Nov 30 22:59:57.291: EIGRP: Sending HELLO on FastEthernet0/1
Nov 30 22:59:57.295:   AS 1, Flags 0x0, Seq 0/0 idbQ 0/0 iidbQ un/rely 0/0

Nik

Correct Answer by nkarpysh about 2 years 4 weeks ago

Hey,

I'm still thinking of SSH - so connection time-out from core and switch A means bad ip connectivity - so these devices either can't reach B or can't get reply back - possibly due to incorrect interface used a source of SSH. And connection refused when it is done from devices between A and B means that they able to reach B and get reply.

As I remember you can ping B from A in VLAN 100 - did you try to intitate SSH with sources interface of VLAN100 from A?

You can try configuring "ip ssh source-interface VLAN 100" and try SSH once again.

For EIGRP - looks like B does not send key at all - and that might be the reason for connection to be torn down. May be as an experiment you can try disabling EIGRP authentication to B and try form neighborship.

Nik

Correct Answer by nkarpysh about 2 years 4 weeks ago

Hello,

Is telnet blocked on B? Why not trying that as well. You can also clearly set source for telnet to make sure A is sending packets from VLAN 100 like

telnet B_IP_addr_IN-Vlan100  /source VLAN100

Regarding your last question. If you power-cycle A - coming up it may synch with NTP server and get current clocke before EIGRP will try to start. Strange that you don't see any EIGRP messages by debug (check if you have no logging monitor configured - that would block debugs sent to monitor session). Or B may just not initiating anything at all for some reason - e.g. config lost - though you still able to ping it so possibly not lost. May be we need some other eigrp debug on it - if possible you can try some others - debug ip eigrp ?

Nik

Correct Answer by nkarpysh about 2 years 1 month ago

AFAIk EIGRP authentication send only one key - the one which is on top. But the other side can math it to the all keys it has. So router with changed config sending key 1 BLAH BLAH BLAH. But other side accept life time for that key expired (if it regained the correct time) and negotiation will fail.

You can try to "debug eigrp pac" to see what you are getting from neighbor and possibly force same key to be sent.

The other thing I did not get. Even if EIGRP is down - other side is on the same VLAN vi trunk. SO there should be communication. Did you try telneting/SSH to the VLAN ip to get to the box? No routing is needed for that as that is connected subnet.

Nik

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (5 ratings)
Correct Answer
nkarpysh Wed, 03/14/2012 - 17:35

AFAIk EIGRP authentication send only one key - the one which is on top. But the other side can math it to the all keys it has. So router with changed config sending key 1 BLAH BLAH BLAH. But other side accept life time for that key expired (if it regained the correct time) and negotiation will fail.

You can try to "debug eigrp pac" to see what you are getting from neighbor and possibly force same key to be sent.

The other thing I did not get. Even if EIGRP is down - other side is on the same VLAN vi trunk. SO there should be communication. Did you try telneting/SSH to the VLAN ip to get to the box? No routing is needed for that as that is connected subnet.

Nik

NormMuelleman Wed, 03/14/2012 - 19:34

Nik;

Thanks for the reply...

THere are a couple devices in between. Let me clarify:

3750G "A"                            3750G (no routing on this, used as a media converter)    3560          3750G "B"

EIGRP                                   no routing                                                            no routing         EIGRP

Int g1/0/5  ----------------------------- g1/0/25---------------------g1/0/26--------------------------g1/0/1----g1/0/2--------g1/0/1

Switchport mode trunk      switchport mode trunk      switch mode trunk          switch mode trunk

Vlan 100                                                                                                                                      Vlan 100

IP 10.70.3.153/30                                                                                                               IP10.70.3.154/30 

"New" config                                                                                                                         "Old" config

Ok, brilliant idea with trying to SSH the VLAN. I tried from the 3560 immediately upstream from the "B" device. I did an SSH into it...it sat there, then nothing. Never even got the log-in banner. I tried just ssh, and the -l with both my login and the local login. Nada.

I also tried from the "A" device; nothing.

I tried EIGRP debug from the 3560...nothing seen there.

I then went to the "A" device. No packets seen there. I did a sho ip eigrp neighbors; the neighbor is not showing in the neighbor table

I shut the vlan 100 on the A device. I can see in term mon where the Dual Nbrchange msg shows that it went down..then came back up..but then that's it.

Funny thing. I could ping the B's VLAN 100 IP address from the A device, but not from anything in-between.

I tried to SSH from "A" to "B"'s vlan ip address. Kept getting "connection refused". Ok, I tried using -l with my user name, and the local user name. Nope...but I could ping it, so I know it's up.

Just as a test, from the directly attached 3560 to device "B", I ssh'ed back to device "A" with no problem.

I'm wondering this: If someone there on site were to power off "A"...it should lose it's time/date. When it came back on...it should get the Key 1 from A..it would be in it's Key 2 range..and should come up, should it not??

Correct Answer
nkarpysh Thu, 03/15/2012 - 07:49

Hello,

Is telnet blocked on B? Why not trying that as well. You can also clearly set source for telnet to make sure A is sending packets from VLAN 100 like

telnet B_IP_addr_IN-Vlan100  /source VLAN100

Regarding your last question. If you power-cycle A - coming up it may synch with NTP server and get current clocke before EIGRP will try to start. Strange that you don't see any EIGRP messages by debug (check if you have no logging monitor configured - that would block debugs sent to monitor session). Or B may just not initiating anything at all for some reason - e.g. config lost - though you still able to ping it so possibly not lost. May be we need some other eigrp debug on it - if possible you can try some others - debug ip eigrp ?

Nik

NormMuelleman Thu, 03/15/2012 - 10:42

We only allow ssh. I did try telnet; nope. I also tried ssh -l, using the local username, as well as my tacacs user name. It attempts to connect, but no response, then it times out. That's trying from both A, and from the network Core switch. When I try to connect from the devices between A and B, I keep getting "connection refused".

I did turn on EIGRP debugging on A. I can see VLAN 100 come up and form an adjacency. It lasts for a little bit, then drops. It says NBRCHANGE:EIGRP-IPv4(1): Neighbor 10.70.3.154 (Vlan100) is up:new adjacency. Then about 20 seconds later, it goes down, and says retry limit exceeded.

So, that shows me that it's TRYING to form the adjacency. It shows up in the neighbor table for a bit, then drops out (probably when trying to authenticate).

Any other suggestions? THese have been great. I'm guessing I'm stuck until we can physically get hands-on the device.

Norm Muelleman

Correct Answer
nkarpysh Fri, 03/16/2012 - 19:50

Hey,

I'm still thinking of SSH - so connection time-out from core and switch A means bad ip connectivity - so these devices either can't reach B or can't get reply back - possibly due to incorrect interface used a source of SSH. And connection refused when it is done from devices between A and B means that they able to reach B and get reply.

As I remember you can ping B from A in VLAN 100 - did you try to intitate SSH with sources interface of VLAN100 from A?

You can try configuring "ip ssh source-interface VLAN 100" and try SSH once again.

For EIGRP - looks like B does not send key at all - and that might be the reason for connection to be torn down. May be as an experiment you can try disabling EIGRP authentication to B and try form neighborship.

Nik

NormMuelleman Fri, 03/16/2012 - 22:01

Nik;

Thanks again for reply. We tackled this tonight again. Ping from A to B is successful. Cannot ping from inbetween A and B. The EIGRP debugging just shows neighbor change..it comes up..then goes right down.

I remembered that the ACL only allowed SSH from certain locations/devices. I tried to SSH with source address from my PC. Timed out. SSH from adjoining devices...connection refused. From A...times out. We did try Telnet (even though it's set to SSH only) to port 22. We did get a banner grab saying it was SSH v2. So there's connectivity.

I'll try that "ip ssh source-interface VLAN 100" and try it again.

I built a mini-lab tonight at work with the 4 devices. I copied the configs from KIWI Tools saved configs before the EIGRP change. I want to first experiement to see the result with the first change made. But, I'm thinking I try this:

If I can find someone locally at B (it's a long story...we can't move...certain people can ) to go to B physcially...I'll remove NTP statement from A. We'll power down both A and B. Power up..they'll both have 1 March 1993 00000 for date and time.

If you look at the EIGRP key chains...even though the TIME STATEMENTS are different, the Key 1, keystring, and actual times will be correct. Any idea if the accept/send statements have to be EXACT MATCHING? I've not seen anything about that. I'm understanding that the key number must match, the key string must match, and the time frame must be within the accept/send time ranges. Yeah, i'll take down several other legs of the network...but I'll be able to get into B..fix it...log out..replace NTP statement on A..reboot...and everything will be fine. I'm going to test it in the lab after my day off.

Anyone know the exact NATURE of how this thing works? I mean, what's the point of having 2 or 3 keys? If the sender ALWAYS uses key 1...and the receiver trys both key 1 and key 2..what sense does it make if you have different key-strings? They'll never match?? Or, does the sender try Key 1...if no good, then tries Key 2? And so on...?

Norm Muelleman

Correct Answer
nkarpysh Sat, 03/17/2012 - 18:55

Hey,

Router can have multiple keys for several reasons:

1. You Different routers to send you different keys, e.g. never to form neighborship between each other. E.G. you have A, B and C on local ethernet domain. If they all are on same subnet - so all will receive their EIGRP hellos. But if you want only A-b and A-c neigborship to work and prevent B-C neighborship - you can make those B and C to send different key number which other does not have in key chain.

E.G. A has key1, key2 and key3 - and sends key1 as it first valid in a  row. B sends key2 but also has key1 to match with A, C sends key3 and also has key1 to match with A. In this case A will form neighborship with B and C, But B and C will not match their case between themselves.

2. It is designed for seamless key rotation - to expire particular keys and start sending/accepting others which you already tried. Configured times should not exactly match - but they need to cover same intervals for router to start sending/accepting correct key at proper time. Eigrp router does not send the life-time specifications to neighbor - those are just locally significant.

It is strange "debug eigrp pac" did not show you a thing. It should show all hello packets between neighbors and also auth messages. Can you check if you have  "no logging monitor" enabled or "no loging console"? and see if those messages appear with "show log".

Here is just test output from this debug:

Here is the debug eigrp packet outputs:


Nov 30 22:59:55.459: EIGRP: received packet with MD5 authentication, key id = 1
Nov 30 22:59:55.463: EIGRP: Received HELLO on FastEthernet0/0 nbr 192.168.12.2
Nov 30 22:59:55.467:   AS 1, Flags 0x0, Seq 0/0 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
Nov 30 22:59:57.291: EIGRP: Sending HELLO on FastEthernet0/1
Nov 30 22:59:57.295:   AS 1, Flags 0x0, Seq 0/0 idbQ 0/0 iidbQ un/rely 0/0

Nik

NormMuelleman Sat, 03/17/2012 - 20:38

I had issued term mon several times..

It just kept saying VLAN 100 up...VLAN 100 down..no Hello packets..nada...yes, it's strange. I built a lab with just these four devices, but the actual deployed configs. when I did a debug eigrp..saw lots of msgs. I'm gonna see what happens if I pull A's eigrp authentication, reboot A and B without NTP, and see if they reform relationship.

Norm Muelleman

NormMuelleman Sun, 03/18/2012 - 16:27

Ok, so making some headway

In the lab tonight, I first tried issuing the ip ssh source-interface VLAN 100; still did not allow me to connect to the remote router. It is probably due to the ACL not recognizing the specific host.

I tried several combinations. It was funny; in the lab, I had it set up as close to RL as possible. A and B have the live configs. Granted, they are not connected to the network..no NTP...no anything else. I have the "old" EIGRP keychain on B.

I have the "new" eigrp keychain on "A". Guess what...no "hellos", other than a couple adjacency changes.

I tinkered around..and then I tried this:

I left old config on B. I put old config on A. Granted, the times were wrong because of the time. SO, I pulled the power to both A and B...then I plugged them in at same time. Guess what? They are routing! ANd, I'm seeing tons of EIGRP authentications...using key 1...looks like it's routing with authentication!

So, I have a tenative game plan to reset this nightmare. It will involve some coordination, but I "think" I can make this work and get back on track with little pain to some end users. This would be so much easier if I could just drive several blocks over to this site and fix this..but where I'm at..nope..I'd be arrested in the current situation sigh...

I'll post what happens when I get approval to push this out

NormMuelleman Mon, 03/19/2012 - 19:33

Nik;

Just a quick follow-up.

EIGRP authentication time statements..they are not sent across to the other router. I didn't see it send anything that looked like time statements. If I understand you correctly, and I think I'm getting the big picture now...

The only thing that matters is the key number, the key-string, and obviously the hash type. The times are only significant to each particular router, correct?

So, A and B are set up for EIGRP authentication. Both have key 1, with the key string of Blah

Router A's key 1send-lifetime is from Jan 1 2012 until Jan 1 2013.

               key 1accept-lifetime is from Jan 1 2012 until Jan 1 2013

Router B's key 1 send lifetime is from March 1 2012 until Mar 1 2013

               key 1 accept lifetime is from March 1 2012 until Mar 1 2013

Now, it doesnt matter per se to each router. Let's take for example it's Jan 1, 2012...

A will attempt to send key 1, because it's send lifetime for key 1 is from Jan 1 2012 until Jan 1 2013. It will always attempt key 1 because that's the time frame it's in, correct? Will it ever go to Key 2?...we'll get back to that..

B receives the EIGRP auth packet. It says "hey, it's the right key number..it's the right key string"..but wait, it won't accept key 1 because it's accept date isnt until March 1. So it will send a refusal....

Then, if I understand this right...if there is a Key 2...as long as key 2 is configured with the correct dates...Router A will attempt to send Key 2. Router B will see "ah, here's an attempt with Key 2"..it will check the accept date on that key..and if it's acceptable..we have interaction!

Does that sum it up correctly? Because if it does, I think I got it figured out now.

Awaiting your reply

Correct Answer
nkarpysh Tue, 03/20/2012 - 08:08

Hey

I think you did figured this out. A will always send same key again and again even if another side refusing it. The lowest active key is the only one to send - second key is usually for replacement process and kicking in once key1 is expired.

So possibly worth trying to expire key 1 on A manually (though make sure that other peers will accept key 2 as our will start sending that out based on interface config).

Nik

NormMuelleman Tue, 03/20/2012 - 11:37

THanks Nik; I did do this in a similar situation on another node area. It worked!

Thanks again for helping a brother out! I'll make sure to put Correct Answer so you get credit

Norm Muelleman

Actions

Login or Register to take actions

This Discussion

Posted March 14, 2012 at 4:56 PM
Stats:
Replies:13 Avg. Rating:5
Views:1533 Votes:0
Shares:0

Related Content

Discussions Leaderboard

Rank Username Points
1 14,997
2 8,150
3 7,720
4 7,078
5 6,723
Rank Username Points
175
80
60
59
55