Broadband connectivity drops.

Unanswered Question
Nov 24th, 2008

sh ver | i IOS|(revision)

Cisco IOS Software, 2801 Software (C2801-ADVIPSERVICESK9-M), Version 12.4(3g), RELEASE SOFTWARE (fc2)

Cisco 2801 (revision 6.0) with 235520K/26624K bytes of memory.

Hello,

We've been seeing our WAN connection drop every 20 minutes or so. (Give or take 5-10 min)

This router is connected to a Time warner cable modem (Zyxel). We are using GRE over IPsec to tunnel the traffic.

Basically what we are seeing is we see the connectivity drop, and will not restore until the ARP table is cleared.

They've replaced the modem and done numerous config checks on it. and nothing comes back of intrest.

So one thing I did see that makes me think its an issue on TW side of things is when i do a debug arp I see alllll of TW's ARP traffic being forwarded from their upstream router. I don't see this traffic on any of our other routers with similar configurations. But this along with the fact that a "clear arp" litterly fixes the issue in seconds makes me think its the problem.

I honestly have never seen anything like this and would be intrested in any opinions.

I attached a quick log of a debug arp. I also X'd and Y'd out our real ip/mac.

Thanks.

Attachment: 
I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Giuseppe Larosa Mon, 11/24/2008 - 12:21

Hello Neil,

I agree that they have some problem: the head end is probably forwarding ARP requests on your part of the network.

What if you try to add a static ARP entry for the upstream router ?

Something like:

conf t

arp 176.20.60.3 22cd.00bb.0002 ARPA

note: use the correct values for your network.

Hope to help

Giuseppe

grambo1980 Mon, 11/24/2008 - 12:25

Thanks for the Reply Giuseppe,

When I had worked with TAC on this for a while we did attempt to do a static ARP entry - which did not resolve the issue.

Giuseppe Larosa Mon, 11/24/2008 - 12:31

Hello Neil,

it was the fist thing I thought of in a case like this.

Have you tried to submit these logs to the provider's support people ?

It could help to make them admit the problem is on their side

the timing of 20 minutes +/- 5 minutes could be the time to fill the ARP table on your router with these useless entries.

In addition to hardcode the upstream router mac you should find a way to filter incoming ARP frames but I'm afraid you cannot have an Access-list based on ethertype on your device.

Hope to help

Giuseppe

grambo1980 Mon, 11/24/2008 - 12:40

Boy it would be nice if they would admit that its an issue on their side :) they keep blaming the router.

Its been esclated 10 ways to sunday, and hopefully it finally reached someone who understands.

Giuseppe - that was also my thinking about the relevance to the how long it takes for the connection to go down.

The only other thing I found that has helped a little bit is to set a very low arp cache timeout on the ethernet interface. This way when the issue does occur it only takes a few seconds until the connection comes back.

Giuseppe Larosa Mon, 11/24/2008 - 12:53

Hello Neil,

>> The only other thing I found that has helped a little bit is to set a very low arp cache timeout on the ethernet interface. This way when the issue does occur it only takes a few seconds until the connection comes back.

The problem you are facing is an ARP rewrite of the upstream router's mac address: this explains the benefits provided by a very low arp cache timeout: the router is able to get rid of the wrong entry quickly instead of trying to use it for long time.

Hope to help

Giuseppe

tcordier Mon, 11/24/2008 - 12:57

I once had it that the provider had proxy-arp enabled on its router which created similar symptoms like the one you describe. You could check that with TW.

- Thomas

ed.mcandrew Thu, 12/18/2008 - 07:43

Hi guys...

Has there been any additional movement on this? I am having the exact same problems with Time Warner (Zyxel). The drops happen about once every 14 minutes for between 8 and 17 seconds. I'm trying the arp timeout at 1 second and watching the arp debug. I don't have a slew of wrong cable entries, just one every so often (trying to correlate it with the drops now)...

2008-12-18 09:00:02 Local7.Debug 10.0.0.254 399522: 992487: *Dec 18 09:04:27.071 PCTime: IP ARP req filtered src 192.168.1.1 0019.cb51.74a8, dst xx.xx.xx.xx 0000.0000.0000 wrong cable, interface FastEthernet0/0

grambo1980 Thu, 12/18/2008 - 08:03

Hi Ed,

I did actually get the issue resolved. However the root cause was never really identified. I had to have Time warner swap out the modem for a different brand modem. After this occured, we did not see the massive wrong interface ARP issues after that.

I think getting the occasional wrong interface message every so often is some what normal.

TW wouldnt give me access to this specific modem so i couldnt really look at it further. And they denied that their modem was misconfigured, or the root cause of the issue. So who knows.

I don't know if this will help you/them correlate the problem at all , but my ticket number to my specific issue was: [TWC ticket# 3210882]

ed.mcandrew Thu, 12/18/2008 - 08:10

They're trying to tell me that there is nothing wrong with the modem. I'm pinging my first hop past the modem through the router and I see the drops. They pinged it directly connected to the modem at the same time that I saw the drops, theirs did not drop, so they cite the router and have washed their hands of the modem...

Thanks for getting back so quickly. I guess I'll get them involved again.

ed.mcandrew Thu, 12/18/2008 - 08:17

Also, I forgot to mention that setting the interface speed to 10mbit seems to help the problem (takes the range from 14 minutes to as much as 2 hours).

grambo1980 Thu, 12/18/2008 - 09:57

Yeah I would prolly just ask or demand even that they replace the modem with a different brand of modem.

ed.mcandrew Thu, 12/18/2008 - 10:47

Well I'm confident that the drops have something to do with ARP requests from the modem... Every time (I mean EVERY TIME)I experience a drop, I see the following in the syslog... What's funny is that every now and again, there is an occassional "wrong cable", with no drop, but for each drop I see, there is an associated "wrong cable" message. But these are the ONLY messages like this that I'm seeing whatsoever. Not a kazillion like you had.

2008-12-18 12:15:59 Local7.Debug 10.0.0.254 415576: 1007097: Dec 18 12:15:55.455 PCTime: IP ARP req filtered src 192.168.1.1 0019.cb51.74a8, dst xx.xx.xx.185 0000.0000.0000 wrong cable, interface FastEthernet0/0

2008-12-18 12:15:59 Local7.Debug 10.0.0.254 415577: 1007098: Dec 18 12:15:57.475 PCTime: IP ARP req filtered src 192.168.1.1 0019.cb51.74a8, dst xx.xx.xx.190 0000.0000.0000 wrong cable, interface FastEthernet0/0

Douglas Fernand... Thu, 10/28/2010 - 19:11

Hello,

I can confirm that EVERY TIME that will happen a drop, I see a  ---- *Oct 28 14:11:21.992 NYC: IP ARP req filtered src 192.168.0.1 c417.feff.38e6, dst ??.??.??.?? 0000.0000.0000 wrong cable, interface FastEthernet0/0 ---- as said by ed.mcandrew.

And I can confirm also that the problem happens around 15 minutes, or 14 minutes and a few seconds as said by ed to.

I will try to force 10Mb to see if it goes to 2 hours...

And they (T.W.C.) also tried to convince us that "Everything is working OK with the modem", and all those automatic answers by-the-book.

What I did as a workaround to this was a kron to every minute "clear arp-cache".

It works, but causes some loss of packets between the start of drops and the next execution of "clear arp-cache".

I will fill myself with patience and try one more time, now with the logs and the experience of you on hand, a call to their support.

Lets see if we can solve this problem.

If I could solve this problem without depending of Time Warner Support, it would make me happy.

I was thinking in some thing around send and receiving gratuitous arp, broadcasts and etc...

Any sugestions?

Douglas Fernand... Fri, 11/05/2010 - 03:17

Just to inform

The modem that is been used in this connection is a ubee ddw3611com.

Some how, my colleague in NYC reached a good technician at Time Warner. Believing or not, they exists.

This guy upgraded the firmware of the modem, and everything was OK.

I removed the "clear arp" every minute kron, and all is good.

Actions

This Discussion