Re: Pinging a WAN interface from a ISP monitoring platform - Page 2

kgreene · ‎09-09-2010

Hi:

We are going round and round with Cogent on this ping health check issue. Our ACL's have been in place for some time. Recently we had a link failure and when the T-1 returned to service, Cogent claimed they could not ping the WAN serial interface anymore. Infact, they stated they could not ping the other T-1 interface also.

I far as I can see within the incoming ACL for the fist T-1. we have this set of rules that should allow it:

ip access-list extended September-2010-199-2.29
permit tcp any 199.4.110.0 0.0.0.255 established
deny   ip 199.4.110.0 0.0.0.255 any log-input
deny   ip 10.0.0.0 0.255.255.255 any log-input
deny   ip 192.168.0.0 0.0.255.255 any log-input
deny   ip 172.16.0.0 0.15.255.255 any log-input
deny   ip 224.0.0.0 31.255.255.255 any log-input
deny   ip host 255.255.255.255 any log-input
permit icmp any any echo
permit icmp any any echo-reply
deny   icmp any any log-input

(the rest of the ACL was deleted for security reasons)

The outgoing ACL has this rule:

ip access-list extended NewInternet_v1.51
permit ip 199.4.110.0 0.0.0.255 any
permit ip host 140.174.40.90 any
permit ip 206.184.209.0 0.0.0.255 any
permit icmp any any echo
permit icmp any any echo-reply
deny icmp any any

From an external IP, you can not ping the WAN IP address. You can ping through the router to the LAN addresses.

I have tried turning off the outbound ACL and even I turned off the incoming ACL with no effect.

Running out of ideas here.

So...

It is either they can not send pings correctly (unlikely) or something is wrong in the actual ACL (I have no clue where).

Any suggestions would be great.

The Router is a Cisco 2621 with dual T-1's and two FE ports.

Thanks.

Kevin

kgreene · ‎09-13-2010

A quick response shows:

cisco2621#show ip route 68.127.30.193
% Network not in table
cisco2621#

I then go to the other router and do:

hayward>ping 199.4.110.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 199.4.110.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
hayward>

Then I look again in the "troubled" router:

cisco2621#show ip route 68.127.30.193
% Network not in table
cisco2621#

I then from the remote site, did a SSH session to a host internally and got the same message.

There is a route in the "troubled" router that is:

route-map oldlink permit 10
match ip address 110
set ip default next-hop 140.174.40.89
!
route-map newlink permit 10
match ip address 120
set ip default next-hop 131.103.116.65

And the ACL for this is:

access-list 110 permit ip 199.4.110.0 0.0.0.255 any
access-list 120 permit ip 206.184.209.96 0.0.0.31 any

Is it that the WAN IP's are not part of the ACL????

Thanks.

Kevin

gatlin007 · ‎09-14-2010

Kevin,

What is your default route pointed at? For example 'ip route 0.0.0.0 0.0.0.0 X.X.X.X'. Is this next hop the proper one for the ping reply?

Those route-maps appear to have been created for policy based routing (PBR). Are they currently applied to an active interface? You should see something like the following in the configuration:

interface x/x
ip policy route-map oldlink

interface y/y
ip policy route-map newlink

Chris

kgreene · ‎09-14-2010

Hello Chris:

Here is the information you requested:

route-map oldlink permit 10
match ip address 110
set ip default next-hop 140.174.40.89
!
route-map newlink permit 10
match ip address 120
set ip default next-hop 131.103.116.65

interface FastEthernet0/0
description 199 services connection
ip address 199.4.110.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip policy route-map oldlink
load-interval 30
speed 100
full-duplex
no cdp enable
!
interface Serial0/0
description 206 Internet connection (PPP)
bandwidth 1544
ip address 131.103.116.66 255.255.255.252
ip access-group September-2010-206-2.29 in
no ip redirects
no ip unreachables
no ip proxy-arp
encapsulation ppp
no ip mroute-cache
load-interval 30
fair-queue
service-module t1 timeslots 1-24
!
interface FastEthernet0/1
description 206 services connection
ip address 206.184.209.97 255.255.255.224
no ip redirects
no ip unreachables
no ip proxy-arp
ip policy route-map newlink
load-interval 30
speed 100
full-duplex
no cdp enable
!
interface Serial0/1
description 199 Internet connection (HDLC)
bandwidth 1544
ip address 140.174.40.90 255.255.255.252
ip access-group September-2010-199-2.29 in
ip access-group NewInternet_v1.51 out
no ip redirects
no ip unreachables
no ip proxy-arp
load-interval 30
fair-queue
service-module t1 timeslots 1-24

The idea (which been working fine until the last failure by the ATT local loop problem) is that I have two IP blocks. One is a /27 and the second is a /24 block.

On the LAN behind the edge router (aka - the 'problem' router), hosts either have a IP address from one block or the other. As an example, the Usenet collector uses the 2nd T-1 only and the primary T-1 (the line that failed and with the 140.174.40.90 WAN IP) is used for web applications. The LAN is a flat LAN and all public devices are on the LAN. All of the corporate stuff is behind a firewall that is attached to the first LAN layer.

So traffic flows out on specific interfaces based on the source IP on the LAN. It has been like this since about 2001 or so. The router was replaced in 2004-2005 when the 2514 ran out of memory to handle the ACL.

I hope this is useful for you. Any help would be great.

Thanks.

Kevin

kgreene · ‎09-14-2010

Chris:

I realized that I did not answer your exact question.

There is not a default route (ip route) as the PBR should do it (assuming I read the book correctly and I also believe what my Cisco experts told me in the past).

Perhaps the PBR takes care of all the LAN traffic correctly but the ICMP for the WAN requires a default route???

Thanks.

Kevin

gatlin007 · ‎09-14-2010

Kevin,

You are absolutely on the right track. The reason the echo-reply never gets to the monitoring server is because the router has no route for it. Back in 2001 PBR may have been a good solution for this; these days VRF-lite may be a better solution.

Installing a default route may upset the delicate balance of your network. It would be safer to put a specific route for the monitoring server; or anything you want to be able to ping the WAN interface. Something like this:

ip route n.n.n.n s.s.s.s 140.174.40.89

Chris

kgreene · ‎09-14-2010

Hi:

I assume I can have several routes such as:

ip route 0.0.0.0 0.0.0.0.0 140.174.40.89 (the far end of the first WAN link)

ip route 0.0.0.0 0.0.0.0.0 131.103.116.65 (the far end of the second WAN link)

Is this correct?

Does not look right...

Kevin

gatlin007 · ‎09-14-2010

Kevin,

Two static default routes can be installed in the routing table. In your case it may be troublesome as it would per flow load share out the two WAN circuits with CEF enalbed. Without CEF it would per packet load share.

I recommend redesigning your internet routing before installing a default route. I think it would be safe to install the specific management server route for the specific WAN interface you want monitored.

Assume the managment server is 4.4.4.4 and your WAN interface IP you want to monitor is 140.174.40.90. A safe change would be:

ip route 4.4.4.4 255.255.255.0 140.174.40.89

Chris

kgreene · ‎09-14-2010

Hi:

Thanks. I will try this. The Cogent people have a series of netblocks they use for monitoring.

I will create a static route for each one.

Is there a way to force a packet back using the same incoming WAN interface?

Thus, if Cogent pings the serial WAN #1, the ICMP packet will go back on that interface.

If they ping the second WAN interface, the packet is returned on that interface...

As to a redesign, always interested. Need work?

Kevin

gatlin007 · ‎09-14-2010

Kevin,

There are complex NAT schemes that can cause a router to create 'state' in regard to an interface. In this case I think that approach would be overcomplicated. Since you just need this functionally for specific monitoring servers static routes will serve the purpose and be manageable.

Does the same service provider management server monitor both WAN interfaces? ICMP is always troublesome in regard to monitoring dual connected sites. In a dynamic routing environment the ability to ping doesn't always mean a circuit is healthy. Enabling a routing protocol and monitoring the state of the neighbor adjacencies would be more effective.

I always have time for interesting network projects.

Chris

kgreene · ‎09-14-2010

Hi:

I did create a static route as shown below:

ip route 68.127.30.192 255.255.255.248 140.174.40.89

And I was able to ping the WAN port from the outside now.

hayward>ping 140.174.40.89

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.89, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/20/24 ms
hayward>ping 140.174.40.90

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms
hayward>

I need to do some further testing to make sure I have not broken anything else however.

Kevin

I just got this email from Cogent about 30 seconds ago...

Dear Cogent Customer,

Cogent proactively monitors your Layer 3 service (Order ID: 1-11544167) utilizing ICMP. 
Attempts to enter your circuit into our monitoring systems have been unsuccessful. Cogent requests your assistance in resolving this issue by allowing ICMP traffic to the following IP Address 140.174.40.90 from the Cogent monitoring IP address blocks of 66.28.3.0/24, 66.250.250.0/24 and 130.117.254.0/24. The IP range for IPv6 monitoring is 2001:978:1:300::/56.

For more information on the Cogent monitoring system please refer to the Customers Service Users Guide sections V and XI.

Link: http://www.cogentco.com/Guide/NA%20User%20Guide.pdf

Cogent remains committed to providing you with a high quality connection and a superior customer service experience. For 24x7 assistance, please contact Cogent Customer Support at support@cogentco.com or by phone (877) 726-4368 option 2.

Thank you again for choosing Cogent Communications as your Internet Service provider.


Regards,

Cogent Communications

kgreene · ‎09-14-2010

Hello:

I have added this to the configuration file.

ip route 66.28.3.0 255.255.255.0 140.174.40.89
ip route 66.250.250.0 255.255.255.0 140.174.40.89
ip route 68.127.30.192 255.255.255.248 140.174.40.89
ip route 130.117.254.0 255.255.255.0 140.174.40.89

From my remote location, it seems to work.

Now trying to get Cogent to test it.

The question is if I add the other WAN port IP address, where does it route from (which serial port)?

Thanks for all the help !!!

Lets see if Cogent can use it now.

Kevin

kgreene · ‎09-14-2010

OK...

It seems that Cogent is happy on the primary T-1 but they did not indicate if the second one work.

Will advise more when I hear it.

Thanks again for all the great help!

Kevin

Cogent Help Desk wrote:
> Dear Cogent Customer,
>
> I am now able to ping you from the monitoring blocks and have added the circuit back into the monitoring system. If you have further questions please feel free to contact us by e-mail at support@cogentco.com or by phone at 877-7COGENT (877-726-4368) option 2.
>
> Thank You,

gatlin007 · ‎09-14-2010

The whole community is behind you Kevin. We admire your perseverance. Everyone appreciates knowing if their advice leads to a solution.

Chris

kgreene · ‎09-14-2010

Well it seems that Cogent is NOW finally happy...

Dear Cogent Customer,

This message confirms your Layer 3 service (Order ID: 1-11544167) has been successfully entered into the Cogent monitoring system. For more information on the Cogent monitoring system please refer to the Customers Service Users Guide sections V and XI.

And the second mail:

Cogent Help Desk wrote:
> Dear Cogent Customer,
>
> I have place both T1's back into monitoring. They both are pingable from the monitoring blocks. If you have further questions please feel free to contact us by e-mail at support@cogentco.com or by phone at 877-7COGENT (877-726-4368) option 2.
>
> Thank You,

Thanks to all of you!!!!!!

Can not thank you enough.

Kevin