I am having an interesting/weird problem with one of my Catalyst 6509 switches.
The setup is as follows:
HostA - Switch1 - Switch2 - HostB
There is routing in place for HostA to reach HostB and vica versa.
I am testing this using ICMP. There is nothing blocking ICMP traffic on any of the devices.
Problem in question: I can only ping HostB from HostA (or Switch1) if HostB is found in the ARP table of Switch2. Pretty normal you might say since we need Switch2 to route/switch the packet to HostB and that will of course only happen if the ARP table is populated.
But what happens is the following:
I ping from HostA to HostB. It fails. I check Switch2 and I see that HostB is nowhere to be found in its ARP table.
Next I ping HostB from Switch2 (5 packets). First packet fails, last 4 I get a reply. I check and see that now HostB is to be found in the ARP table of Switch2.
Now I ping HostB from HostA and it works fine.
This behaviour is happening for all hosts in the subnet (VLAN) of HostB. I cannot reach them until I ping them from Switch2, after which they're in the ARP table and everything works.
Maybe it something different than the ARP table not being populated all together, but that is for now all I noticed. Does this problem ring a bell with anyone. And is there a fix?
It looks to me like Switch2 is malfunctioning, although I have no idea what is. Furthermore this apears to be working ok for a lot (read: all) of the other VLANS on Switch2.
Anyway. I'd love some help in troubleshooting this one.
I have a question on here as well that has a similar issue. For some reason on these 6500's the mac doesn't seem to be there on ports that are obviously connected, unless traffic is generated from that end host. I am awaiting a response to my query. If it helps I'll let you know.
This goes back to the way switches (or bridges) work. They have to see a frame from an end device to know its MAC address and once they see it they add an entry in their CAM (or MAC) table. The entry basically tells the switch that a specific MAC is reachable via a particular port and this is meant to prevent subsequent flooding of unicast frames. In short an end device needs to send at least one frame for the switch to 'see' it. Another point to remember is that each entry in the CAM table has an age associated with it and if a switch does not see subsequent frames from that host it will age out the entry and start flooding all frames destined to this MAC address until it is learnt again (and learning is done only when the owner of the MAC starts to communicate again).
Can you please explain your topology in a bit more detail? I am looking for information like what is the default gateway of each VLAN, how are the VLANs spread across the two switches, any HSRP configurations and if yes who is the active for what VLAN, routing between switches, etc. This will help us troubleshoot the problem in a more effective manner.
Thanks for your reply.
The default gateway of HostA is Switch1.
The default gateway of HostB is Switch2.
No HSRP configurations, no VRRP, no routing protocols.
HostA is on Vlan10. The switches are directly connected in a /30 network on Vlan20. HostB connects to Switch2 in Vlan30 (as do other hosts that are showing the same problem).
There are more VLANs on Switch1. All (read: all that I tested) hosts in these VLANs show the symptom while trying to connect to a host in Vlan30.
There are other VLANs on Switch2. These also show the same symptoms while trying to reach HostB (or another host in Vlan30).
When I traceroute (from Windows, so ICMP) to HostB from HostA, the last hop I get a reply from is the IP address of the Vlan20 interface on Switch2 (so the VLAN connecting the 2 switches). Pinging HostB from HostA and from Switch1 fails.
To reiterate, as soon as I ping HostB from Switch2, I can ping/traceroute/reach HostB from HostA and Switch1 (or anywhere else) just fine.
As u have mentioned that hostA is on vlan10 and u say that switch1 is the default gateway i think it should be interface vlan 10 on switch1 correct me if i am wrong.same on the switch2 i.e it should have a interface vlan30 as the default gateway.
For me it was obvious that by Switch1 I meant the respective Vlan interface on Switch1 (same on Switch2). My apologies for the confusion.
So the default gateway of HostA is interface Vlan10 on Switch1. The default gateway of HostB is interface Vlan30 on Switch2.
Thanks for the explanation and it does make things a lot more clear. One last question ... are you using static routes between the two switches? You must be but I just want to make sure.
The configuration looks simple enough. Try turning on some debugs on SwitchB and see if that tells you anything. I would also try to do a packet trace on the VLAN 30 to see if the SwitchB is generating arp requests or not.
I think it would be helpful if we could see the configurations of the VLAN interfaces.
Also it would be helpful to know the particular configuration parameters for hostB (ip address, mask, and default gateway).
I am wondering if there could possibly be a mismatch between the configuration of interfaces on the switch and configuration of the PC?