A week ago, I changed our network's EIGRP Authentication statements. For the most part, everything worked...MOSTLY
I have a couple areas that are failing, and I'm trying to see about a work around, and also see what the heck is going on.
As I understand EIGRP Authentication, you can have several keys on a "key chain". Let's call our key chain LOSER (sorry, I need to self-punish myself )
DISCLAIMER: NO key-strings were changed..just times
So, the original setup for the key chain was pretty much this :
key chain LOSER
key-string BLAH BLAH BLAH
accept-lifetime 00:00:00 Mar 1 1993 00:15:00 Mar 7 2012
send-lifetime 00:00:00 Mar 1 1993 00:00:00 Mar 7 2012
key-string YADA YADA
accept-lifetime 23:45:00 Mar 6 2012 infinite
send-lifetime 00:00:00 Mar 7 2012 infinite
So, before the key expired, I changed it to this:
key chain LOSER
key-string BLAH BLAH BLAH
accept-lifetime 00:00:00 Mar 1 1993 00:00:00 Mar 5 2013
send-lifetime 00:00:00 Mar 1 1993 00:00:00 Mar 5 2013
key-string YADA YADA
accept-lifetime 00:00:00 Mar 5 2012 infinite
send-lifetime 00:00:00 Mar 5 2012 infinite
Again, key strings not changed.
FIrst, you're going to say "why the 1993 stuff?" Well, some of our sites are in power sensitive areas. We have NTP running. But if the power goes out, which it has, and the UPS fails..what happens? The device comes back up with default time of Mar 1 1993 00:00:00. We've run into that problem before. Changing the authentication time got it up and running again. Also, don't have the luxury of traveling to these sites; it's all or nothing.
So, in our scenario..I goofed up and forgot to change the authentication on a couple devices. So, the clock setting changed..and wham..loss of authentication, hence loss of connection.
So I'm trying to figure out a way to change it back to regain access remotely. Unfortunately, there is NO WAY I can get to these remote sites.
Now here is what I don't get. Let's say you got router A and router B. The interface between the two is trunked. But there's a routed VLAN that's got the new authentication on A, and the old authentication on B. Do the accept and send STATEMENTS have to match exactly? I thought as long as the date/time was correct, and the password matched..good to go. SO, if that's the case, B being the old config KEY 2 has from March 6 2345 to infinite to ACCEPT, and Marcy 7 0000 infinite to send. We are at March 15...the new config Key 2 has from March 5 0000 infinite to send, and March 5 0000 infinite to accept. But there is no routing occurring. EIGRP neighbors are not showing the Router B.
I've tried to just remove the authentication from VLAN 100. No change
I've tried to change it to the OLD config, thinking ok, they'll match on both ends..but no routing update. Of course they won't match Key 1 because it's past March 7, 2012. But key 2 SHOULD allow it! Shouldn't it???
Please help, because I feel like a total loser for this, and the self-flageration is starting to hurt
I think you did figured this out. A will always send same key again and again even if another side refusing it. The lowest active key is the only one to send - second key is usually for replacement process and kicking in once key1 is expired.
So possibly worth trying to expire key 1 on A manually (though make sure that other peers will accept key 2 as our will start sending that out based on interface config).
Router can have multiple keys for several reasons:
1. You Different routers to send you different keys, e.g. never to form neighborship between each other. E.G. you have A, B and C on local ethernet domain. If they all are on same subnet - so all will receive their EIGRP hellos. But if you want only A-b and A-c neigborship to work and prevent B-C neighborship - you can make those B and C to send different key number which other does not have in key chain.
E.G. A has key1, key2 and key3 - and sends key1 as it first valid in a row. B sends key2 but also has key1 to match with A, C sends key3 and also has key1 to match with A. In this case A will form neighborship with B and C, But B and C will not match their case between themselves.
2. It is designed for seamless key rotation - to expire particular keys and start sending/accepting others which you already tried. Configured times should not exactly match - but they need to cover same intervals for router to start sending/accepting correct key at proper time. Eigrp router does not send the life-time specifications to neighbor - those are just locally significant.
It is strange "debug eigrp pac" did not show you a thing. It should show all hello packets between neighbors and also auth messages. Can you check if you have "no logging monitor" enabled or "no loging console"? and see if those messages appear with "show log".
Here is just test output from this debug:
Here is the debug eigrp packet outputs:
Nov 30 22:59:55.459: EIGRP: received packet with MD5 authentication, key id = 1
Nov 30 22:59:55.463: EIGRP: Received HELLO on FastEthernet0/0 nbr 192.168.12.2
Nov 30 22:59:55.467: AS 1, Flags 0x0, Seq 0/0 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
Nov 30 22:59:57.291: EIGRP: Sending HELLO on FastEthernet0/1
Nov 30 22:59:57.295: AS 1, Flags 0x0, Seq 0/0 idbQ 0/0 iidbQ un/rely 0/0
I'm still thinking of SSH - so connection time-out from core and switch A means bad ip connectivity - so these devices either can't reach B or can't get reply back - possibly due to incorrect interface used a source of SSH. And connection refused when it is done from devices between A and B means that they able to reach B and get reply.
As I remember you can ping B from A in VLAN 100 - did you try to intitate SSH with sources interface of VLAN100 from A?
You can try configuring "ip ssh source-interface VLAN 100" and try SSH once again.
For EIGRP - looks like B does not send key at all - and that might be the reason for connection to be torn down. May be as an experiment you can try disabling EIGRP authentication to B and try form neighborship.
Is telnet blocked on B? Why not trying that as well. You can also clearly set source for telnet to make sure A is sending packets from VLAN 100 like
telnet B_IP_addr_IN-Vlan100 /source VLAN100
Regarding your last question. If you power-cycle A - coming up it may synch with NTP server and get current clocke before EIGRP will try to start. Strange that you don't see any EIGRP messages by debug (check if you have no logging monitor configured - that would block debugs sent to monitor session). Or B may just not initiating anything at all for some reason - e.g. config lost - though you still able to ping it so possibly not lost. May be we need some other eigrp debug on it - if possible you can try some others - debug ip eigrp ?
AFAIk EIGRP authentication send only one key - the one which is on top. But the other side can math it to the all keys it has. So router with changed config sending key 1 BLAH BLAH BLAH. But other side accept life time for that key expired (if it regained the correct time) and negotiation will fail.
You can try to "debug eigrp pac" to see what you are getting from neighbor and possibly force same key to be sent.
The other thing I did not get. Even if EIGRP is down - other side is on the same VLAN vi trunk. SO there should be communication. Did you try telneting/SSH to the VLAN ip to get to the box? No routing is needed for that as that is connected subnet.