Pinging a WAN interface from a ISP monitoring platform

Unanswered Question

Hi:

We are going round and round with Cogent on this ping health check issue. Our ACL's have been in place for some time. Recently we had a link failure and when the T-1 returned to service, Cogent claimed they could not ping the WAN serial interface anymore. Infact, they stated they could not ping the other T-1 interface also.

I far as I can see within the incoming ACL for the fist T-1. we have this set of rules that should allow it:

ip access-list extended September-2010-199-2.29
permit tcp any 199.4.110.0 0.0.0.255 established
deny   ip 199.4.110.0 0.0.0.255 any log-input
deny   ip 10.0.0.0 0.255.255.255 any log-input
deny   ip 192.168.0.0 0.0.255.255 any log-input
deny   ip 172.16.0.0 0.15.255.255 any log-input
deny   ip 224.0.0.0 31.255.255.255 any log-input
deny   ip host 255.255.255.255 any log-input
permit icmp any any echo
permit icmp any any echo-reply
deny   icmp any any log-input

(the rest of the ACL was deleted for security reasons)

The outgoing ACL has this rule:

ip access-list extended NewInternet_v1.51
permit ip 199.4.110.0 0.0.0.255 any
permit ip host 140.174.40.90 any
permit ip 206.184.209.0 0.0.0.255 any
permit icmp any any echo
permit icmp any any echo-reply
deny icmp any any

From an external IP, you can not ping the WAN IP address. You can ping through the router to the LAN addresses.

I have tried turning off the outbound ACL and even I turned off the incoming ACL with no effect.

Running out of ideas here.

So...

It is either they can not send pings correctly (unlikely) or something is wrong in the actual ACL (I have no clue where).

Any suggestions would be great.

The Router is a Cisco 2621 with dual T-1's and two FE ports.

Thanks.

Kevin

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
gatlin007 Thu, 09/09/2010 - 13:51

Your ACL does look good.  If you can get them on the phone while they try to ping your interface it would be helpful.  Run a 'debug ip icmp' on your router.  Then from config mode execute a 'logging monitored debugging'; after this from privileged exec mode execute a 'term mon'.  Immediately execute a 'show proc cpu' and make sure your CPU isn't spiked.  I doubt it would be as this debug command is generally always safe unless someone is flooding you with ICMP.

At this point the ICMP echo and echo-reply messages your router receives and generates should stream across your telnet/ssh session.  I wouldn't think that it would be so verbose that you couldn't spot them trying to ping your WAN interface.  This will give you some idea as to if the service providers echo packet ever reaches you.  It will also make you aware if your router is sending the echo-reply.  At that point check your routing table for were your router is sending it.  Sometimes if you have redundant links you may be sending it out the other router.


After troubleshooting run a 'undebug all' from privileged mode to make sure you are not vulerable to an ICMP attack.

Chris

Richard Burts Sat, 09/11/2010 - 07:21

Kevin

first a couple of observations and then a couple of questions and possible issues:

- if they are trying to ping the router interface then the problem can not be the outbound access list. An outbound access list will not filter traffic that is generated by the router itself.

- you state that you removed the inbound access list and the problem continued. If correct then this was good troubleshooting and clearly demonstrates that the problem is not related to access list filtering (and the part of the access list that you posted certainly looks like it would not cause problems with ping).

= it is not clear from your post whether other traffic is working and the problem is only with ping to the interface or whether the problem is that no traffic works through the interface. Can you clarify whether the interface is working for other things?

= can you ping from your router to any remote resource, specifying the problematic interface as the source of the ping (using extended ping for this)?

= can you ping from your router to the connected interface of the ISP router, specifying the problematic interface as the source of the ping (using extended ping for this)?

= are you running CDP on the T1 interfaces? If so posting the output of show cdp neighbor detail might be helpful.

= you mention that there are 2 T1s and that there was an outage. Is it possible that the cables for the T1s got swapped in the process of troubleshooting and resolving the outage?

If you can provide this information we may be in a better position to help resolve your issue.

HTH

Rick

Hi:

Sorry I have not responded sooner. Been a bit busy this week.

The router (2621) is in operation and does pass traffic. The ACL's are on the right interface as otherwise, many things would not work.

From our remote office (a ADSL provided by ATT), I have an established VPN connection to the VPN server located behind the edge router (the 2621). In addition, the people who access the various web services have not complained.

Again from the remote office, I can ping any interface on the inside lan without a problem. But I can not ping the WAN IP of the serial interface. I can ping Cogent's end of the circuit however.

Feel free to ping the interface yourself to see if there is some weird.

Cogent end of the WAN interface: 140.174.40.89

My end of the WAN link: 140.174.40.90

I have tried as I mentioned even turning off BOTH ACL's (hated to do this) and tried pinging the WAN with no luck. By both, I mean the incoming and outgoing ACL's. The other T-1 does not have an outgoing ACL for some reason but it has been like this for several years.

You asked if I could ping the interface from the CLI. I did and it works.

cisco2621# ping 140.174.40.90

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/13/20 ms
cisco2621# ping 140.174.40.89

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.89, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/8 ms
cisco2621#

However if I try to ping Google as an example:

cisco2621#ping www.google.com
Translating "www.google.com"...domain server (199.4.110.11) [OK]

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 74.125.19.147, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
cisco2621#

It does not work on this router. However - This does work on my remote location router.

I am not running CDP on the WAN interfaces. Should I???

I am wanting to resolve this. Any ideas would be great.

Kevin

Richard Burts Sun, 09/12/2010 - 14:47

Kevin

First let me address the CDP question. Many people that I know do not run CDP on interfaces that face outside of their network and consider CDP on outward facing interfaces to be a security issue. I do not advocate running CDP on outward facing interfaces in normal circumstances. There are some circumstances where running CDP can provide very helpful troubleshooting information, and perhaps this might be a situation where CDP information could be useful. Of course if Cogent is not running CDP then you will not get anything useful. But it might be worth it to try turning on CDP for the T1 interfaces (I would do both), wait for a couple of minutes and try show cdp neighbor detail. And then turn off CDP.

When I attempt ping I get 100% packet loss to both addresses. And when I attempt traceroute I get into the Cogent network but not to anything that appears to be close to your subnet. So it looks to me like Cogent is doing some filtering, which may or may not relate to your particular issue. And that might explain why the ping to Google from the router fails. Do you know if ping from the router to Google worked before these problems started?

It is good news that traffic through the router works ok and that the VPN sessions establish without problem. It makes me wonder if the issue is something about the interfaces themselves. Would you post the output of traceroute on the router to the Cogent interface address and traceroute on the router to its own serial interface address?

HTH

Rick

Richard Burts Mon, 09/13/2010 - 20:54

Kevin

This is a very strange situation and I do not yet see a clear answer about what is going on.

First let me address an observation in one of my previous posts. I said that my attempts to ping either your interface or the Cogent interface results in 100% packet loss. That appears to be an issue with the firewall running on my laptop (with which I have been having some problems). I have changed setting in my laptop firewall and now I do get the results that you describe: ping to the Cogent address is successful but ping to your address fails.

My sense is that Cogent is blocking traffic. I base this on the combination of factors including:

- ping to their interface works while ping to your interface fails (and you now have proof that the ping to your interface is received on your router and that your router is attempting to respond).

- ping from your router to an address near your router is successful while ping from your router to a remote address fails.

- traceroute to the Cogent address and to your address gets part way into their network and then fails.

But I have not yet come up with a way to convince Cogent of this.

I will continue to think about this. Perhaps if you could provide some additional details about your environment, about the problem that impacted the T1, and about what was done to recover from that episode we might get some better understanding.;

HTH

Rick

[edit] let me acknowledge that my response shows up much earlier in the thread than I expected, which may make it difficult to follow the chronology of the responses.

And then let me agree with Chris that it is important to respond appropriately when our wives call. And I would suggest that in addition to the show ip route that he suggests, that it might be helpful to do a traceroute to that address and see if the traceroute shows what is the next hop, and how many hops it goes.

Hi:

I enabled CDP on the router and then enabled the CDP on both serial interfaces.

I then did:

cisco2621#show cdp neighbor detail

cisco2621#show cdp neighbor detail

cisco2621#show cdp neighbor detail

cisco2621#

And saw nothing (I waited about 5 minutes).

If I do a traceroute to either the local WAN port on either serial card, I see nothing. It comes with no results.

cisco2621# traceroute  140.174.40.90

Type escape sequence to abort.
Tracing the route to d1-2-1-3-17.a01.snfcca02.us.ce.verio.net (140.174.40.90)

  1  *  *  *
  2  *  *  *
  3  *  *  *
  4  *  *  *
  5  *  *  *
  6  *  *  *
  7  *  *  *
  8  *  *  *
  9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *

Same with the other serial port.

As a side note: the IOS is: c2600-i-mz.123-6.bin

It is has been working for a long time (years) without a problem.

I have done a hardware rest with no differences. I have done a software reset with no differences.

It is almost like there is some device in the middle of the Cogent WAN that is filtering traffic. Am I paranoid???

Cogent claims there is no filtering done on their routers.

Not sure I believe this anymore.

Kevin

Richard Burts Sun, 09/12/2010 - 19:03

Kevin

Thanks for trying CDP. I am not entirely surprised that it did not work but thought that it was worth a try. It probably means that Cogent has it turned off on their router (or that their router is not Cisco). So make sure it is turned off on your router.

I am surprised that the ping to the remote interface did work but that the traceroute did not. I have another experiment that I would ask you to try.

- do show interface for both T1s and note the input and output packet counters.

- do an extended ping to the 40.90 address and in the extended ping send 5000 packets

- do show interface for both T1s and note the input and output packet counters. are they both going up? are they going up on the same interface?

HTH

Rick

OK... Thanks for the email.

Here is the ping test. I hope it helps.

Kevin

cisco2621#show interfaces serial 0/1
Serial0/1 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 199 Internet connection (HDLC)
  Internet address is 140.174.40.90/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 12/255, rxload 9/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/19460/0 (size/max/drops/flushes); Total output drops: 2052
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/2038 (size/max total/threshold/drops)
     Conversations  0/40/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 57000 bits/sec, 77 packets/sec
  30 second output rate 76000 bits/sec, 91 packets/sec
     10517425 packets input, 2760598167 bytes, 0 no buffer
     Received 76575 broadcasts, 0 runts, 82 giants, 0 throttles
     162012 input errors, 44859 CRC, 88332 frame, 0 overrun, 0 ignored, 28820 abort
     7873081 packets output, 2062179523 bytes, 0 underruns
     0 output errors, 0 collisions, 34 interface resets
     0 output buffer failures, 0 output buffers swapped out
     15 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#show interfaces serial
% Incomplete command.

cisco2621#ping 140.174.40.90 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


(lines edited out to save space)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 12/13/36 ms
cisco2621#show interfaces serial 0/1
Serial0/1 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 199 Internet connection (HDLC)
  Internet address is 140.174.40.90/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 30/255, rxload 18/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:02, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/19460/0 (size/max/drops/flushes); Total output drops: 2052
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/2038 (size/max total/threshold/drops)
     Conversations  0/40/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 110000 bits/sec, 141 packets/sec
  30 second output rate 183000 bits/sec, 167 packets/sec
     10530583 packets input, 2761850861 bytes, 0 no buffer
     Received 76584 broadcasts, 0 runts, 82 giants, 0 throttles
     162012 input errors, 44859 CRC, 88332 frame, 0 overrun, 0 ignored, 28820 abort
     7888603 packets output, 2063978495 bytes, 0 underruns
     0 output errors, 0 collisions, 34 interface resets
     0 output buffer failures, 0 output buffers swapped out
     15 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#ping 140.174.40.89 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 140.174.40.89, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

(lines edited out to save space)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 4/7/220 ms
cisco2621#show interfaces serial 0/1
Serial0/1 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 199 Internet connection (HDLC)
  Internet address is 140.174.40.90/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 23/255, rxload 14/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/19460/0 (size/max/drops/flushes); Total output drops: 2052
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/2038 (size/max total/threshold/drops)
     Conversations  0/40/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 90000 bits/sec, 127 packets/sec
  30 second output rate 143000 bits/sec, 164 packets/sec
     10538523 packets input, 2762551800 bytes, 0 no buffer
     Received 76591 broadcasts, 0 runts, 82 giants, 0 throttles
     162012 input errors, 44859 CRC, 88332 frame, 0 overrun, 0 ignored, 28820 abort
     7898918 packets output, 2065075360 bytes, 0 underruns
     0 output errors, 0 collisions, 34 interface resets
     0 output buffer failures, 0 output buffers swapped out
     15 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

Richard Burts Sun, 09/12/2010 - 20:55

Kevin

Thanks for doing this testing for me. The results are interesting and a bit puzzling. When you ping your own address the packet counters should go up twice as much as when you ping the remote address. But the counters here do not do that. The packet counters go up more when you ping the remote than when you ping your own address. And when you ping your own address 5000 times there should be 10000 input packets and the input counters only went up 7940.

Could I get you to repeat the test and this time get the output of show interface for both of the T1 interfaces?

HTH

Rick

[edit] If there were NetFlow data available it might help. I am not sure whether a router running that version of code has support for NetFlow. Can you tell me whether that router supports NetFlow?

Hi:

I do not think it does Netflow. I think the 2621 XM does but the older 2621.

I will rerun the tests... This is the other serial interface.

I will leave for home, My wife just called...

Kevin

Serial 0/0

cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 12/255, rxload 96/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:04, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:02:10
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 584000 bits/sec, 128 packets/sec
  30 second output rate 76000 bits/sec, 108 packets/sec
     15144 packets input, 8555346 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     12706 packets output, 1161554 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#ping 131.103.116.65 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 131.103.116.65, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

(edited...)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 4/9/220 ms
cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 5/255, rxload 77/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:15, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:03:36
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 469000 bits/sec, 72 packets/sec
  30 second output rate 34000 bits/sec, 54 packets/sec
     23287 packets input, 13587805 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     19368 packets output, 1756526 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#

cisco2621#clear counters serial 0/0
Clear "show interface" counters on this interface [confirm]
cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 3/255, rxload 67/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:00:03
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 407000 bits/sec, 53 packets/sec
  30 second output rate 22000 bits/sec, 37 packets/sec
     164 packets input, 240902 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     85 packets output, 3801 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#ping 131.103.116.66 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 131.103.116.66, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(edited for ease of reading)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 12/15/40 ms
cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 15/255, rxload 77/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:06, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:01:37
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 469000 bits/sec, 142 packets/sec
  30 second output rate 96000 bits/sec, 125 packets/sec
     13073 packets input, 5435841 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     11634 packets output, 1113674 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#

Richard Burts Mon, 09/13/2010 - 12:10

Kevin

Thank you for running these tests that I asked for. They are not showing anything that points toward a problem. As I go through these results and as I re-read the posts in this thread I think that I perhaps did not understand the problem correctly. I assumed that it was some problem on the local subnet and that the ping attempt was coming primarily from the connected router. I now believe that those assumptions were faulty - my apologies.

If the problem has to do with remote sources of the ping then I go back to the suggestion made by Chris. debug ip icmp would be an excellent test. First make sure that the monitor logging level is set to debug (which is level 7) and then do terminal monitor (or if you are connected on the router console then make sure that logging console is set to debug). Then turn on debug ip icmp. If a ping request gets to your router it should then generate debug output.

HTH

Rick

Hi:

I will run the tests suggested by Chris.

The problem - to restate is:

Cogent monitors their customers by pinging the serial WAN interface on a periodic basis (every 5 minutes as I recall). If the ping fails after several ping tests in a row, they will call the customer within a few minutes to see if there is some sort of power failure or perhaps an equipment failure. Normally they are very good about this.

The problem now is that from an external location somewhere in the Gogent network, their monitoring platform can not ping the serial interface from the outside. It is possible to ping one of the hosts on the inside LAN. The public /24 and the smaller /27 block can be pinged from the outside. The issue is that the two WAN IP addresses - those that face the external world, can not be pinged. If you want to ping an internal host, try 199.4.110.11 (the name server). I should note that a traceroute to an internal host starts but never gets anywhere near the WAN. (See below). It can however be pinged:

hayward>ping 199.4.110.11

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 199.4.110.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms
hayward>

The router is in production use and seems to work fine otherwise. The two incoming ACL's are active and working (and on the right interface).

I will run the test next.

Thanks for all the help so far.

Kevin

hayward>traceroute 199.4.110.11

Type escape sequence to abort.
Tracing the route to ns1.cellmail.com (199.4.110.11)

  1 adsl-68-127-30-198.dsl.pltn13.pacbell.net (68.127.30.198) 4 msec 0 msec 0 msec
  2 192.0.2.100 12 msec 16 msec 12 msec
  3 64.164.107.130 12 msec 12 msec 12 msec
  4 151.164.93.229 12 msec 12 msec 16 msec
  5 151.164.101.122 44 msec 32 msec 52 msec
  6 po5-3.core01.sjc04.atlas.cogentco.com (154.54.13.97) 12 msec 16 msec 16 msec
  7 te3-2.mpd01.sjc04.atlas.cogentco.com (66.28.4.49) 16 msec *  *
  8 te0-0-0-6.mpd22.sfo01.atlas.cogentco.com (154.54.28.81) 16 msec
    te0-0-0-6.mpd21.sfo01.atlas.cogentco.com (154.54.2.165) 16 msec
    te0-0-0-6.mpd22.sfo01.atlas.cogentco.com (154.54.28.81) 12 msec
  9 te7-3.mpd01.sfo01.atlas.cogentco.com (154.54.3.222) 16 msec 12 msec 12 msec
10 gi0-1.ca01.sfo01.atlas.cogentco.com (154.54.5.90) 12 msec 16 msec 12 msec
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *

Hello:

From the debug information, I got the following information.

I was pinging the 2nd T-1 from my remote office (the .193 address).

ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193
Sep 14 01:59:26: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 30 packets
ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193
ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193
ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193

hayward>ping 140.174.40.90

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
hayward>

Then in the problem router...

84.72(123) (Serial0/1 ) -> 199.4.110.40(123), 1 packet
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193

So it looks like the reply is being sent OK...

Ideas??? Somewhere the packets are dropped.

Thanks.

Kevin

gatlin007 Mon, 09/13/2010 - 20:51

Kevin,

First off you are a good man.  We should all pay attention when our wives call.

If you execute a 'show ip route 68.127.30.193' from the troublesome router is the next hop over the interface you expect it to be?  Is the gateway for this route the optimum path?

From the ACL's you posted it doesn't look like this traffic would be dropped.  Could it be that this traffic is crossing an interface that may have security policy that conflicts with it?  Or perhaps it's matching a null route?


Chris

A quick response shows:

cisco2621#show ip route 68.127.30.193
% Network not in table
cisco2621#

I then go to the other router and do:

hayward>ping 199.4.110.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 199.4.110.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
hayward>

Then I look again in the "troubled" router:

cisco2621#show ip route 68.127.30.193
% Network not in table
cisco2621#

I then from the remote site, did a SSH session to a host internally and got the same message.

There is a route in the "troubled" router that is:

route-map oldlink permit 10
match ip address 110
set ip default next-hop 140.174.40.89
!
route-map newlink permit 10
match ip address 120
set ip default next-hop 131.103.116.65

And the ACL for this is:

access-list 110 permit ip 199.4.110.0 0.0.0.255 any
access-list 120 permit ip 206.184.209.96 0.0.0.31 any

Is it that the WAN IP's are not part of the ACL????

Thanks.

Kevin

gatlin007 Tue, 09/14/2010 - 05:15

Kevin,

What is your default route pointed at?  For example 'ip route 0.0.0.0 0.0.0.0 X.X.X.X'.  Is this next hop the proper one for the ping reply?

Those route-maps appear to have been created for policy based routing (PBR).  Are they currently applied to an active interface?  You should see something like the following in the configuration:

interface x/x
ip policy route-map oldlink


interface y/y
ip policy route-map newlink



Chris

Hello Chris:

Here is the information you requested:

route-map oldlink permit 10
match ip address 110
set ip default next-hop 140.174.40.89
!
route-map newlink permit 10
match ip address 120
set ip default next-hop 131.103.116.65

interface FastEthernet0/0
description 199 services connection
ip address 199.4.110.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip policy route-map oldlink
load-interval 30
speed 100
full-duplex
no cdp enable
!
interface Serial0/0
description 206 Internet connection (PPP)
bandwidth 1544
ip address 131.103.116.66 255.255.255.252
ip access-group September-2010-206-2.29 in
no ip redirects
no ip unreachables
no ip proxy-arp
encapsulation ppp
no ip mroute-cache
load-interval 30
fair-queue
service-module t1 timeslots 1-24
!
interface FastEthernet0/1
description 206 services connection
ip address 206.184.209.97 255.255.255.224
no ip redirects
no ip unreachables
no ip proxy-arp
ip policy route-map newlink
load-interval 30
speed 100
full-duplex
no cdp enable
!
interface Serial0/1
description 199 Internet connection (HDLC)
bandwidth 1544
ip address 140.174.40.90 255.255.255.252
ip access-group September-2010-199-2.29 in
ip access-group NewInternet_v1.51 out
no ip redirects
no ip unreachables
no ip proxy-arp
load-interval 30
fair-queue
service-module t1 timeslots 1-24

The idea (which been working fine until the last failure by the ATT local loop problem) is that I have two IP blocks. One is a /27 and the second is a /24 block.

On the LAN behind the edge router (aka - the 'problem' router), hosts either have a IP address from one block or the other. As an example, the Usenet collector uses the 2nd T-1 only and the primary T-1 (the line that failed and with the 140.174.40.90 WAN IP) is used for web applications. The LAN is a flat LAN and all public devices are on the LAN. All of the corporate stuff is behind a firewall that is attached to the first LAN layer.

So traffic flows out on specific interfaces based on the source IP on the LAN. It has been like this since about 2001 or so. The router was replaced in 2004-2005 when the 2514 ran out of memory to handle the ACL.

I hope this is useful for you. Any help would be great.

Thanks.

Kevin

Chris:

I realized that I did not answer your exact question.

There is not a default route (ip route) as the PBR should do it (assuming I read the book correctly and I also believe what my Cisco experts told me in the past).

Perhaps the PBR takes care of all the LAN traffic correctly but the ICMP for the WAN requires a default route???

Thanks.

Kevin

gatlin007 Tue, 09/14/2010 - 11:00

Kevin,


You are absolutely on the right track.  The reason the echo-reply never gets to the monitoring server is because the router has no route for it.  Back in 2001 PBR may have been a good solution for this; these days VRF-lite may be a better solution.



Installing a default route may upset the delicate balance of your network.  It would be safer to put a specific route for the monitoring server; or anything you want to be able to ping the WAN interface.  Something like this:


ip route n.n.n.n s.s.s.s 140.174.40.89



Chris

gatlin007 Tue, 09/14/2010 - 12:27

Kevin,


Two static default routes can be installed in the routing table.  In your case it may be troublesome as it would per flow load share out the two WAN circuits with CEF enalbed.  Without CEF it would per packet load share.

I recommend redesigning your internet routing before installing a default route.  I think it would be safe to install the specific management server route for the specific WAN interface you want monitored.


Assume the managment server is 4.4.4.4 and your WAN interface IP you want to monitor is 140.174.40.90.  A safe change would be:


ip route 4.4.4.4 255.255.255.0 140.174.40.89



Chris

Hi:

Thanks. I will try this. The Cogent people have a series of netblocks they use for monitoring.

I will create a static route for each one.

Is there a way to force a packet back using the same incoming WAN interface?

Thus, if Cogent pings the serial WAN #1, the ICMP packet will go back on that interface.

If they ping the second WAN interface, the packet is returned on that interface...

As to a redesign, always interested. Need work?

Kevin

gatlin007 Tue, 09/14/2010 - 13:32

Kevin,

There are complex NAT schemes that can cause a router to create 'state' in regard to an interface.  In this case I think that approach would be overcomplicated.  Since you just need this functionally for specific monitoring servers static routes will serve the purpose and be manageable.

Does the same service provider management server monitor both WAN interfaces?  ICMP is always troublesome in regard to monitoring dual connected sites.  In a dynamic routing environment the ability to ping doesn't always mean a circuit is healthy.  Enabling a routing protocol and monitoring the state of the neighbor adjacencies would be more effective.

I always have time for interesting network projects.


Chris

Hi:

I did create a static route as shown below:

ip route 68.127.30.192 255.255.255.248 140.174.40.89

And I was able to ping the WAN port from the outside now.

hayward>ping 140.174.40.89

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.89, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/20/24 ms
hayward>ping 140.174.40.90

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms
hayward>

I need to do some further testing to make sure I have not broken anything else however.

Kevin

I just got this email from Cogent about 30 seconds ago...

Dear Cogent Customer,

Cogent proactively monitors your Layer 3 service (Order ID: 1-11544167) utilizing ICMP. 
Attempts to enter your circuit into our monitoring systems have been unsuccessful. Cogent requests your assistance in resolving this issue by allowing ICMP traffic to the following IP Address 140.174.40.90 from the Cogent monitoring IP address blocks of 66.28.3.0/24, 66.250.250.0/24 and 130.117.254.0/24. The IP range for IPv6 monitoring is 2001:978:1:300::/56.

For more information on the Cogent monitoring system please refer to the Customers Service Users Guide sections V and XI.

Link: http://www.cogentco.com/Guide/NA%20User%20Guide.pdf

Cogent remains committed to providing you with a high quality connection and a superior customer service experience. For 24x7 assistance, please contact Cogent Customer Support at [email protected] or by phone (877) 726-4368 option 2.

Thank you again for choosing Cogent Communications as your Internet Service provider.


Regards,

Cogent Communications

Hello:

I have added this to the configuration file.

ip route 66.28.3.0 255.255.255.0 140.174.40.89
ip route 66.250.250.0 255.255.255.0 140.174.40.89
ip route 68.127.30.192 255.255.255.248 140.174.40.89
ip route 130.117.254.0 255.255.255.0 140.174.40.89

From my remote location, it seems to work.

Now trying to get Cogent to test it.

The question is if I add the other WAN port IP address, where does it route from (which serial port)?

Thanks for all the help !!!

Lets see if Cogent can use it now.

Kevin

OK...

It seems that Cogent is happy on the primary T-1 but they did not indicate if the second one work.

Will advise more when I hear it.

Thanks again for all the great help!

Kevin

Cogent Help Desk wrote:
> Dear Cogent Customer,
>
> I am now able to ping you from the monitoring blocks and have added the circuit back into the monitoring system. If you have further questions please feel free to contact us by e-mail at [email protected] or by phone at 877-7COGENT (877-726-4368) option 2.
>
> Thank You,

gatlin007 Tue, 09/14/2010 - 17:08

The whole community is behind you Kevin.  We admire your perseverance.  Everyone appreciates knowing if their advice leads to a solution.



Chris

Well it seems that Cogent is NOW finally happy...

Dear Cogent Customer,

This message confirms your Layer 3 service (Order ID: 1-11544167) has been successfully entered into the Cogent monitoring system. For more information on the Cogent monitoring system please refer to the Customers Service Users Guide sections V and XI.

And the second mail:


Cogent Help Desk wrote:
> Dear Cogent Customer,
>
> I have place both T1's back into monitoring. They both are pingable from the monitoring blocks. If you have further questions please feel free to contact us by e-mail at [email protected] or by phone at 877-7COGENT (877-726-4368) option 2.
>
> Thank You,

Thanks to all of you!!!!!!

Can not thank you enough.

Kevin

Actions

This Discussion

Related Content