Solved: a question about arp tables...

linnea.wren · ‎04-03-2007

Hey...

A Windows server has 10 IP addresses.

Intermittently, all IP addresses except for the primary address become inaccessible.

This may well be a problem on the server itself, but in the course of looking everywhere, I noticed something on a core switch that I don't understand.

The switch is a 6509, sup720/msfc3, IOS Enterprise Services, v12.2(18)SXE5.

When all is working as it should, "sh arp" on the switch has entries for all 10 of the server's IP addresses.

However, as time passes, the Age in Minutes value for the server's primary IP address restarts, like this:

sh arp 1: Age = 5

sh arp 2: Age = 0

sh arp 3: Age = 3

sh arp 4: Age = 2

(Sh arp commands having been done at irregular intervals.)

For the other 9 IP addresses on the box, the Age value continues to increment until we have a crisis.

From this it would seem that the switch re-arps for the primary address, even though it has an entry for that IP in its table.

Why would it re-arp for an IP that it has in its table?

And, why re-arp for some addresses, but not for others?

(The age values for IP addresses on all our vlans are all over the map, anything from 0 up in to the 200s, so the phenomenon doesn't seem to be confined to this one device.)

Any thoughts/knowledge much appreciated...

Richard Burts · ‎04-04-2007

Linnea

That is an interesting behavior and there may be several things that could explain it. I think perhaps most likely is that the server periodically might be sending a gratuitous ARP (announcing itself). If the switch receives an ARP response from the switch (perhaps responding to some other device) it would refresh the entry in the ARP table.

I agree that the problem about the other addresses would seem to be a problem at the server. If you want to investigate either of these things further I would suggest that debug arp on the sup might be helpful.

HTH

Rick

HTH

Rick

View solution in original post

Richard Burts · ‎04-03-2007

Linnea

At least part of what you describe sounds like normal behavior. Especially the description of the Age value incrementing and then starting over - and covering a wide range of values. In IOS the default aging time for the ARP cache is 4 hours (240 minutes or 14400 seconds). And when IOS ages out an entry in the ARP table it sends an ARP request for the address and if it gets a response it inserts the address with time value of 0 and starts over again.

In your description of the issue I am not clear whether the switch is not sending a request for the addresses or whether the switch is requesting and the server is only answering for the first address. To investigate this further I would suggest running debug arp on the switch. This will show ARP activity including what the switch sends out and whether it receives responses to its requests.

HTH

Rick

HTH

Rick

IVAN PEPELNJAK · ‎04-04-2007

The ARP table is aged out (as pointed out by Rick) but is not repopulated unless you run CEF on the box (in which case the router re-issues an ARP request to maintain its CEF adjacency table).

linnea.wren · ‎04-04-2007

Thank you both.

I don't think I made the question sufficiently clear, though...

The entry for the 1st IP address of the server never ages beyond 5 or 6 minutes. We do have the aging timer set at the default of 4 hours, and the other entries reach that value. But the entry for the first IP address does not.

Therefore, it seems the entry for that IP address gets refreshed long before it has been the in the table long enough to time out. I was wondering what would cause that...

In the problem of losing access to the other IP addresses, what I see on the switch while the problem is occurring is entries for those IP addresses listed as "incomplete". And, on the server itself, when I do a packet capture, I see ARP requests coming in but no responses going out. That's why I'm thinking the problem is actually at the server level.

So, the question about the seemingly premature aging of that one address is more trying to fill in of a knowledge gap...

Thanks again...

Richard Burts · ‎04-04-2007

Linnea

That is an interesting behavior and there may be several things that could explain it. I think perhaps most likely is that the server periodically might be sending a gratuitous ARP (announcing itself). If the switch receives an ARP response from the switch (perhaps responding to some other device) it would refresh the entry in the ARP table.

I agree that the problem about the other addresses would seem to be a problem at the server. If you want to investigate either of these things further I would suggest that debug arp on the sup might be helpful.

HTH

Rick

HTH

Rick

nickjacobs · ‎04-04-2007

Most likely the server is using some kind of NIC or MS load balancing, and as mentioned above is using either gratuitous ARp , or the router is seeing different ARP response according to the NIC it wants a device to respond to at the time.

linnea.wren · ‎04-05-2007

Thanks Rick. Discovered there is Windowsy traffic from another machine on that subnet that accounts for the arp table refreshes on the switch...

Richard Burts · ‎04-05-2007

Linnea

I am glad that our comments were helpful for you to find the solution.

Thanks for using the rating system to indicate that your issue was resolved. (and thanks for the rating) It makes the forum more useful when people can read about an issue and can know that they will read a solution to the issue. I encourage you to continue your participation in the forum.

HTH

Rick

HTH

Rick