Solved: Strange problem - remote site.

rcoote5902_2 · ‎07-27-2009

I have a remote site with a 2811 router and several 2950 switches that started showing as unreachable in Network Assistant. All I could see in CNA was the router and the first switch. What is strange is if I telnet into the visible switch, I can see the 2 other switches connected to it via show CDP neighbors and they are both pingable. I can even telnet to them from the first switch.

There are no ACL's.

There is no firewall.

Links between the switches are trunks.

There have been no configuration changes made recently.

From the remote site, connectivity looks normal - LAN/WAN/Internet access is fine.

Here is a trace to the first (visible) switch:

1 <1 ms <1 ms <1 ms 172.16.128.1

2 <1 ms <1 ms <1 ms 172.16.255.163

3 * * * Request timed out.

4 * * * Request timed out.

5 2 ms 1 ms 1 ms 172.16.255.98

6 3 ms 2 ms 2 ms 172.16.52.10

And a trace to the second switch:

Tracing route to 172.16.52.11 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 172.16.128.1

2 <1 ms <1 ms <1 ms 172.16.255.163

3 * * * Request timed out.

4 * * * Request timed out.

5 2 ms 1 ms 1 ms 172.16.255.98

6 * * * Request timed out.

7 * * * Request timed out.

8 ^C

It doesn't appear to be an ICMP issue since I can ping from 52.10 to 52.11 and telnet between them.

It's got me pretty boggled. Any ideas?

jbrenesj · ‎07-27-2009

As you can see, none of your switches have a default-gateway but 52.10 is receiving redirects from 172.16.52.1 and that's why you can reach it.

You need

ip default-gateway 172.16.52.1 on all of your layer 2 switches

View solution in original post

jbrenesj · ‎07-27-2009

Please paste a sh ip route and sh ip redirects from the 52.10 and 52.10 switches

rcoote5902_2 · ‎07-27-2009

52.10:

CamSwMain-01#sho ip route

^

% Invalid input detected at '^' marker.

CamSwMain-01#show ip redir

CamSwMain-01#show ip redirects

Default gateway is not set

Host Gateway Last Use Total Uses Interface

172.16.131.23 172.16.52.1 0:39 4154 Vlan1

172.16.130.75 172.16.52.1 0:01 13009 Vlan1

64.235.218.180 172.16.52.1 0:01 5774 Vlan1

52.11:

CamSwEWing#show ip route

^

% Invalid input detected at '^' marker.

CamSwEWing#show ip red

CamSwEWing#show ip redirects

Default gateway is not set

Host Gateway Last Use Total Uses Interface

ICMP redirect cache is empty

jbrenesj · ‎07-27-2009

As you can see, none of your switches have a default-gateway but 52.10 is receiving redirects from 172.16.52.1 and that's why you can reach it.

You need

ip default-gateway 172.16.52.1 on all of your layer 2 switches

rcoote5902_2 · ‎07-27-2009

Hmmm. It's working fine in all my other remote sites, and they do not have that configured. Again, this site has been fine until recently and no changes have been made.

I added the default-gateway and the problem is still there.

Here is another site, same equipment, same setup:

104.10:

LsSw-01#show ip route

^

% Invalid input detected at '^' marker.

LsSw-01#show ip redire

LsSw-01#show ip redirects

Default gateway is not set

Host Gateway Last Use Total Uses Interface

ICMP redirect cache is empty

104.11:

LsSw-02#show ip route

^

% Invalid input detected at '^' marker.

LsSw-02#show ip red

LsSw-02#show ip redirects

Default gateway is not set

Host Gateway Last Use Total Uses Interface

ICMP redirect cache is empty

Yet I can ping, trace, and telnet to both devices.

jbrenesj · ‎07-27-2009

Weird, as you know, you can't reach a device from another subnet unless it has a DG. From the same subnet everything works (i.e. .10 reaching .11)

What if you try to ping and traceroute 172.16.255.98 from 172.16.52.11

Do the same from 52.10

Let me know..

rcoote5902_2 · ‎07-27-2009

255.98 is the outside interface of the router. Interesting results.

52.10:

CamSwMain-01#ping 172.16.255.98

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.255.98, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

52.11:

CamSwEWing#ping 172.16.255.98

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.255.98, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

Over at the "working" site (router outside interface is 172.16.255.34):

104.10:

LsSw-01#ping 172.16.255.34

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.255.34, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

104.11:

LsSw-02#ping 172.16.255.34

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.255.34, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms

rcoote5902_2 · ‎07-28-2009

Ok here's some more wierdness.

I can't ping the router's outside interface from the 52.10 switch, but I can ping the next-hop interface on our ISP's edge device.

An extended ping from the router's outside interface will not hit the switch. *boggled*

CamSwMain-01#ping 172.16.255.98

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.255.98, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

CamSwMain-01#ping 172.16.255.99

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.255.99, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms

And the Router:

CamRtr#ping

Protocol [ip]:

Target IP address: 172.16.52.10

Repeat count [5]:

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]: y

Source address or interface: 172.16.255.98

Type of service [0]:

Set DF bit in IP header? [no]:

Validate reply data? [no]:

Data pattern [0xABCD]:

Loose, Strict, Record, Timestamp, Verbose[none]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.52.10, timeout is 2 seconds:

Packet sent with a source address of 172.16.255.98

.....

Success rate is 0 percent (0/5)