Can't ping hosts in same subnet

[branfarm1]

Hi there. I have an odd problem and I need some troubleshooting assistance.

I have an office network configured as 192.168.0.0/16 and we have hosts configured in various ranges in that large subnet. Most server are 192.168.0.0/16 and others are 192.168.5.0/16 (treated kind of like they are in /24 subnets. I have a Microsoft DHCP server assigning addresses in the 192.168.3.0/16 space (again, treated like /24). All of my servers and hosts have a /16 subnet mask, and the default gateway is configured as 192.168.0.1.

Here's my problem: Whenever a host receives a DHCP address from my MS DHCP server, it can only ping certain servers and hosts in the 192.168.0.0/16 space. For instance, the host can ping anything in 192.168.5.0 and 192.168.3.0 but has troubles in the 192.168.0.0 range. I've double checked the DHCP server and I know it's giving out the proper subnet mask, and I've verified that with a packet sniffer on the host. Whenever the host tries to ping the gateway, or anything beyond the gateway, the pings timesout. Other hosts can ping the box, but the switch can't ping it.

When I look at the arp table on both the host and switch (Catalyst 4500) I can see that both are receiving the correct MAC addresses from ARP requests. And the most interesting thing, when I turn on ICMP debugging on the switch and start a continuous ping from the host, I can see ICMP echo replies the switch is sending back to the host, but nothing ever actually gets there.

I don't have DHCP snooping or dynamic arp inspection turned on. I can't think of anything else on the switch side of things to look at. Any ideas?

Thanks in advance,

--Brandon

[Yudong Wu]

Try to do a sniffer on both PC and port which is connected to PC. From your description, it looks like ICMP echo replies were lost between PC and switch.

[Edison Ortiz]

192.168.0.1 is the 4500 switch?

Can you post the switch config?

__

Edison.

[branfarm1]

I guess I should say 192.168.0.1 is the Vlan interface on the switch. I've attached the config.

Something else I noticed today -- when I ping from my computer to the host I'm testing on, I get one ping through then it times out. When I go to my computer or the switch to see what mac-address is associated with the IP, I get the MAC for the VLANs on my switch.

[Edison Ortiz]

Are both devices connected to the same switch? (The 4500 in question?).

If so, what switchports are they physically connected?

Can we see the IPCONFIG /ALL from both devices?

Can you ping 192.168.0.1 from Vlan 55, 85 and 501?

__

Edison.

[branfarm1]

The host I'm testing on is attached to Gig4/38. My computer is attached through my IP phone on port Fa3/39.

Here's the ipconfig from the box attached to Gig4/38:

Ethernet adapter Wireless Network Connection:

Media State . . . . . . . . . . . : Media disconnected

Description . . . . . . . . . . . : Broadcom 802.11b/g WLAN

Physical Address. . . . . . . . . : 00-90-4B-4C-62-DB

Ethernet adapter {6CE9868C-467B-45FB-89D2-98BE06E9AD7B}:

Connection-specific DNS Suffix . :

Description . . . . . . . . . . . : Nortel IPSECSHM Adapter - Packet Scheduler Miniport

Physical Address. . . . . . . . . : 44-45-53-54-42-00

Dhcp Enabled. . . . . . . . . . . : No

IP Address. . . . . . . . . . . . : 0.0.0.0

Subnet Mask . . . . . . . . . . . : 0.0.0.0

Default Gateway . . . . . . . . . :

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . : me.com

Description . . . . . . . . . . . : National Semiconductor DP83815-Based PCI Fast Ethernet Adapter

Physical Address. . . . . . . . . : 00-0E-7F-EB-5C-4E

Dhcp Enabled. . . . . . . . . . . : Yes

Autoconfiguration Enabled . . . . : Yes

IP Address. . . . . . . . . . . . : 192.168.3.37

Subnet Mask . . . . . . . . . . . : 255.255.0.0

Default Gateway . . . . . . . . . : 192.168.0.1

DHCP Server . . . . . . . . . . . : 192.168.0.21

DNS Servers . . . . . . . . . . . : 192.168.0.31

192.168.0.21

Primary WINS Server . . . . . . . : 192.168.0.31

Secondary WINS Server . . . . . . : 192.168.0.21

Lease Obtained. . . . . . . . . . : Wednesday, February 11, 2009 12:14:40 PM

Lease Expires . . . . . . . . . . : Thursday, February 19, 2009 12:14:40 PM

Here's my box:

Windows IP Configuration

Host Name . . . . . . . . . . . . : utb

Primary Dns Suffix . . . . . . . : me.com

Node Type . . . . . . . . . . . . : Hybrid

IP Routing Enabled. . . . . . . . : No

WINS Proxy Enabled. . . . . . . . : No

DNS Suffix Search List. . . . . . : me.com

Ethernet adapter Office Connection:

Connection-specific DNS Suffix . :

Description . . . . . . . . . . . : Linksys EG1032 v3 Instant Gigabit Desktop Network Adapter Driver

Physical Address. . . . . . . . . : 00-18-F8-0D-72-AF

DHCP Enabled. . . . . . . . . . . : No

Autoconfiguration Enabled . . . . : Yes

Link-local IPv6 Address . . . . . : fe80::908c:7d0b:b5de:e5f6%10(Preferred)

IPv4 Address. . . . . . . . . . . : 192.168.128.109(Preferred)

Subnet Mask . . . . . . . . . . . : 255.255.0.0

Default Gateway . . . . . . . . . : 192.168.0.1

DNS Servers . . . . . . . . . . . : 198.168.0.21

192.168.0.31

NetBIOS over Tcpip. . . . . . . . : Enabled

I will say that I'm seeing this problem on more computers than just the one I'm currently testing on. It seems to be random, and only affects DHCP assigned addresses. My box can ping the DHCP assigned address, but the testbox can't ping the gateway or viceversa. Hosts in other vlans can ping 192.168.0.1.

[Edison Ortiz]

Did you verify the testbox does not have FW enabled?

If you manually assign an IP to this testbox, are you able to duplicate the problem?

As for your computer, I'm assuming you are able to connect to other subnets via the 192.168.0.1 gateway, correct?

__

Edison.

[branfarm1]

I have verified the FW is disabled. It can ping other hosts, just not the gateway, and the gateway can't ping that box. When I assign a manual IP to the box I can ping the gateway and every other host I attempt.

My computer works fine (I have a static).

Do you think it could be something switch related? Are there any bugs in IOS that mess with DHCP?

[Edison Ortiz · Accepted solution]

[slap in my forehead]

You are blackholing that host in the routing table.

ip route 192.168.3.37 255.255.255.255 Null0

__

Edison.

[branfarm1]

Oh-my-gosh --- [Double forehead slap]

Wow. Thanks for seeing that. I guess if it were a snake it would've bit me. I had to ask around, but we added that line over a year ago to block someone who was abusing the internet.

Thanks a ton for your help!

[Edison Ortiz]

It was a fun troubleshooting exercise, NOT :)

__

Edison.