SSH headache!

[NormMuelleman]

Hello all!

I am working in an enterprise LAN environment. We have about 100 switches, mostly 3560 and 3750's. Let me give you the setup, then ask for some great advice from this fantastic forum!

This is a typical Cisco network, yet it's flat. No routing on the access or distro layers. The core switch does do the routing.

We use an third party vendor network monitoring tool, and we use SecureCRT to remote into devices.

Here's the problem.

THere was a device we stumbled into that had not been put into our monitoring software. It has the same IOS as our other devices. All I can say is that it's the same version and type.

Each device has a management vlan. And each device has it's own management IP.

An ACL exists to prevent unauthorized SSH access into the devices, yet allows the management vlan scope to get in.

So, here's the problem...we can't SSH into our problem mystery device, let's call it Switch X. Switch X has an IP of 10.10.100.150. Now, I can be logged into it's uplink device, let's call it switch B. Switch B has an IP of 10.10.100.130. The ACL allows all devices from 10.10.100.0/24 to SSH. Our PC's at our desk are also in the same management VLAN. SSH version 2 is on the configs, and the domain names are the same on these two devices.

So, let's be clear. From my desktop, I can connect to any device on my network EXCEPT switch X. When I try to connect using SSH, port 22...it just sits there until it times out. I can do the same thing to any other switch, and connect just fine. We are using TACACS+ and RADIUS as well, and they are up and running just fine. THe configs on Switch X like I said are the same for switch B, except it's IP address of course.

While logged into switch B, I can do a CDP neighbor and see switch X connected via trunk link. Both sides are running dot1q encapsulation, and both are in trunk mode. I can ping switch X from switch B. When I try to SSH from B to X..I get timeout with no connection.

So, I hiked over to the building where switch X is located. I consoled into the switch. I confirmed that the ACL is the same as the ACL for switch B. It is set up to allow the management vlan inbound on the VTY 0 - 15. Yes, it's access-class (name) in on both vty 0 4 and 5 15. It also is set up for transport ssh in and transport ssh out.

I rechecked the domain name on Switch X; it was correct. I also did a crypto key zeroize and regenerated the crypto key. SSH v2 came up. Again, while in Switch X, I can do a CDP neighbor and see switch B. But I cannot SSH from switch X to Switch B, or any other devices that I tried.

Now, we did find a config error with VTP; the VTP domain name was different. But VTP has nothing to do with SSH. Just to placacte my co-workers, I went ahead and renamed the VTP domain name (it's running transparent mode). After I regenerated the crypto key, I saved everything of course. I then reloaded the switch. When all came back up, I still could not SSH

Any suggestions or thoughts? Oh, this is a 3560 switch, and it is trunked to a 3750.

[ebarticel]

Usualy if run in transparent mode the local VLANs are only local to that switch and not part of VTP domain. Saying that it could be a management VLAN issue. Is management VLAN the same in switch X as other switches?

Did you try to put it in client mode just to try SSH to it?

Hope this helps

Eugen

[NormMuelleman]

No, you're right about transparent mode. I didnt set this network up. I inherited it..and it's got issues. But..I should have not included the VTP issue, thought that might take away from the core topic.

But in answer to your question, yes, the management VLAN is..let's say...vlan 50. Vlan 50 on this switch is the same as the uplink switch which is Vlan 50.

As a side note..I started studying for the CCNP Switch..and first topic was VTP. Just to calm my nerves..if I change mode from transparent to client..and Switch X is revision 0, and the uplink is revision 126, as is the core switch which is the VTP Server and has revision 126...the lower number will seek out the higher number. So, VTP will disregard the vlan.dat file, and upload the latest revision 126 from the core switch, correct?

I really dislike VTP..yes, it has benefits..but one wrong move and your network is toast lol

[ebarticel]

Yes thats right, VTP client will update its database from core VTP server switch.

[NormMuelleman]

Ok, but that still doesnt explain the SSH issue.

The vlans are the same on both devices.

[ebarticel]

Could be a problem with the number of ssh sessions as well. Maybe you access server is limited to so many open sessions. Check the timeout setting for vty lines.

Create another ACL just for troubleshooting and specify the host addresses, not network addresses and port 22, and add it to the switch. If you can not remove temporarly the existing ACL make sure that you permit statements go to the top.

Maybe post a debug log output that we can look at it.

[NormMuelleman]

Ok, I just looked closely, and I think I found the problem...

So, again, we have the Core switch, ...Switch B...and Switch X.

The Core switch is the VTP server. It has multiple VLANS, but three stand out:

VLAN 50 \\SwMgmt// IP 10.10.100.66 /26

VLAN 80 \\Management// IP 10.10.100.162 /28

VLAN 90 \\SwMgmt// IP 10.10.100.34 /27

Now, switch B has the same list of VLANs, including the ones above. Doing a sho ip int bri, you can see switch B is assigned a managment ip out of VLAN 50, 10.10.100.72 ( a /26 would be network 10.10.100.64 - 127)

Switch X is showing it has an IP address 10.10.100.175, VLAN 99. This is outside the range that is showing for VLAN 50. While it is in the right range, I'm guessing that because it's not in the same VLAN as the other devices, that's why SSH isn't working.

Sound right? If so, pounding my head on desk lol

[Richard Burts]

Norm

That sounds logical to me. If switch X has an IP address in the right address range but in a different VLAN then it would explain the problem. The other switches are doing ARP requests in VLAN 50 for the address of switch X. But the ARP is a broadcast in VLAN 50. So switch X will not respond since its address is in VLAN 99. If you put the right address on switch X into the right VLAN then things should work.

HTH

Rick

[ebarticel]

Because switch X was on transparent mode, it has a different management vlan that other switches.

Change settigns on your PC with an IP from management subnet used on switch X and try to connect, if you succesful then you found the problem.

[NormMuelleman]

Well, I figured as much with the VLAN mismatch..

But I also noticed that the way this is subnetted, switch X has an IP address of the broadcast address for that particular subnet!! URGH! What a mess!

Not my doing...you should see this mess..we have like 30 VLANs, and at least 50 scopes in the DHCP.

I'm going to have to figure out the easiest way to redo this. I'm guessing to delete VLAN 90 (I typo'ed above..), put in VLAN 55 with the same name, and then give it a new IP address.

Head...pounding...glad I go to bed in a bit

Thanks all..I'll have to tackle this tonight after hours again. I'll post outcome. Unfortunately, due to my employer, cannot post any logs..and I had to try and make this sound right with ficticious IP addresses and such.

[ebarticel · Accepted solution]

It may work if you change the management VLAN to be the same across the domain.

Good luck

Eugen

[NormMuelleman]

Well, I thought I would give the final outcome of this mess:

First, I changed the VTP domain name. I made the switch a client, and it grabbed the correct VLAN information.

Then, as I suspected, the IP address it was assigned was INDEED that of the broadcast address. But it was in the wrong switch management VLAN anyway...there are like 5 or 6 management vlans in this network..it's a jumbled mess!

So, I moved the switch into the correct vlan, and gave it a new IP. I could now at least ping everywhere on this portion of the network, but could not ping the RADIUS server. I also could connect to the distro switch (the next adjoining uplink), and I got the warning banner and log on, but it would never take my credentials. So..I walk back to the office thinking "why won't it take it". As I get back to the office, I say outloud "it's acting like it's not coming back". Uh...did I change the default gateway? I look..NO! I changed the gateway...

Finally! SSH works. I can ping everywhere in the network. I can SSH into the device, and authenticate just fine.

Add that lesson to my experience

Thank you all for your help and insight!

[Richard Burts]

Norm

Congratulations on getting it working

Thank you for posting back to the thread and giving us the updated status. And thank you for using the rating system to indicate that the question was answered. It makes the forum more useful when people can read a thread and can know that a solution was found. Your marking has contributed to that process.

HTH

Rick

[Richard Burts · Accepted solution]

Norm

It depends a bit on the subnet mask. But the 10.10.100.175 would indeed be a broadcast address if the mask were /28 or /29 or /30. So it looks like the best alternative is to re-define the VLAN and to assign an IP address that fits better.

HTH

Rick