We have a communications link with a customer. There is some sort of routing issue going on. I attached a diagram of everything to help out.
As you can see, we cannot communicate with 10.90.5.1 (or anything else on the 10.90.5.x network for that matter), which is the destination firewall. All access lists and security has been turned off for troubleshooting.
The problem looks like the "Customer WatchGuard Firewall" route statement. I believe the gateway needs to be changed from 10.10.5.155 to whatever the next hop IP address is.
Is that correct?
So the route statement on the Customer WatchGuard Firewall should be...
ip route 10.90.5.0 255.255.255.0 ???.???.???.???
What is the gateway IP address going to be?
I was told the link between the Customer WatchGuard Firewall and the Destination WatchGuard Firewall is a VPN, but I do not have any specifics.
Your assumption is correct. You have a routing loop between the 'Customer Watchguard' and the 'Customer Gateway Router'.
Is 10.90.5.0/24 pingable from the 'Customer Watchguard' ? If not, you need to find out what device is able to ping that network and send the route there. This device will be your default gateway. Of course, this device must be pingable from the 'Customer Watchguard' device.
I should have provided that info.
We can ping 10.90.5.0/24 from ALL of the devices on the 10.10.5.0/24 network.
In other words...
The "Customer WatchGuard" can ping 10.90.5.x
The "Customer Gateway Router" can ping 10.90.5.x
The ethernet interface on "Our Gateway Router" can ping 10.90.5.x
It is my understanding that there are no devices between the customer's WatchGuard firewalls. They are both connected to each other via VPN.
I still dont know what specific ip address the gateway needs to be on the "Customer WatchGuard Firewall" route statement. The traffic obviously needs to be sent through their VPN to reach its destination.
My apologies, I should've read the diagram more carefully, the route statements look fine and there isn't a loop.
If you can ping the destination network from all devices then from which network/device are you getting the destination unreachable ?
Can you provide a traceroute from that device ?
What is S-192.168.10.42? I am assuming that's the source address from which you are trying to get to 10.90.5.1.
If that's the case then it looks like one of the devices, either the watchguard firewall or the customer router, doesn't have a route back to the 192.168.10.x network. The problem appears to be the return traffic. Check the routing table on those devices and make sure they have a route back to you.
The problem is we can not communicate with 10.90.5.x from any of our own networks. The only network that can ping 10.90.5.1 is 10.10.5.x. Our networks are coming in through the serial interface of "Our Gateway Router".
The 192.168.10.42 interface is a frame relay link back to our FR hub router. I do not think it is relevant with what we are discussing.
Here is where it gets cute...
We are NATTING the 10.90.5.x network(ethernet side) to the 192.168.251.x network(serial side) on "Our Gateway Router".
192.168.251.x is being routed to 192.168.250.x.
Our devices on 192.168.250.x (which are coming in through the serial interface of "Our Gateway Router") can not communicate with 10.90.5.x (which appears to them as 192.168.251.x).
Did I lose anyone? I apologize for the confusion, as this diagram is actually a small snippet of a couple hundred routers that are currently in production.
Sundar, I believe you are correct. I am going to have the customer check his routing tabels to see if he has a route back to us. I do not believe he does.
He should have on all his devices...
ip route 192.168.250.0 255.255.255.0 x.x.x.x
(where x.x.x.x is the next hop back to us)
You are absolutely correct. Since you are able to source traffic from the 10.10.5.x network and get to to 10.90.5.x network and it only fails when traffic is originated from 192.168.250.0/24 NET.
The return traffic is your problem. Check all the routers in the transit path to make sure they have a route back to the 192.168.250.0/24 NET
Just want to offer a workaround in the interim while you get the concerned party to fix the routing problem. You could configure PAT (NAT) on your gateway router to source the E0 IP of 10.10.5.12.
Thanks Sundar, but we have other natted networks traveling through S0 and E0 on that router...so I wouldnt want to do that. The customer will be contacting me shortly anyway.
That is part of the reason why I didnt catch such an easy mistake - I have been working with PAT for the past five years and I am still not used to the NAT methods we have on this network.
I appreciate everyone's help.