Hi, Giuseppe:

Im not sure you understood what I was asking.

The spoke router has a tunnel interface, a vlan (users LAN) interface, and the public address interface to the SP.

The hub router has the public address interface to the SP, the tunnel interface, etc.

From the spoke, I do NOT source the LAN subnet when I run the PING. I just PING the outside interface of the hub (the tunnel endpoint), and I go over the Internet. I do not use the tunnel, as the trace shows. On the other hand, if I trace to something that sits behind the outside interface of the hub, it will use the tunnel, even though I am still not sourcing the LAN. Im just running a straight trace.

COBOU27THBASELINE_RL01#trace 138.69.21.250 <----sits way behind the hub on the internal LAN of the data center.

Type escape sequence to abort.

Tracing the route to 138.69.21.250

1 10.40.16.1 84 msec 80 msec 84 msec <----tunnel on hub, then it goes to a FW, so we lose the trace.

2 * * *

3 * * *

4 * * *

5

COBOU27THBASELINE_RL01#ping 138.69.21.250

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 138.69.21.250, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 84/86/92 ms

COBOU27THBASELINE_RL01#

My first question is why this behavior?

Thanks

Hello Joe,

I think that to understand what is happening you should attach

the configuration of the HUB and of the spokes (the parts of interest)

sh ip route

sh ip nhrp

on both ends

Are you using a dynamic routing protocol over the mGRE tunnel ?

Hope to help

Giuseppe

Giuseppe:

I think I'm starting to figure it out on my own. :-)

My first question was why the tunnel is not used when I trace from the spoke router to the outside, tunnel destination interface on the hub, without sourcing the user-LAN interface. I think the answer is actually clear. Dont know why I didnt see it before....

The route in the routing table of the spoke for the outside hub interface is a static route whose next hop points to the provider. Ex: ip route 138.60.X.X 255.255.255.255 .

So, given that the LAN interface is not being sourced, which means that the tunnel is not being used, the trace packet to the outside interface of the hub will adopt a source address of the exit interface and get routed like any other normal iP datagram. We are therefore able to see all the intervening hops. What I mean by the "tunnel is not being used" is that the packet does not go through any GRE encapsulation at the spoke -- and probably not even any IPSec (ESP) enapsulation either. This latter point I am not sure of, actually.

The second point I am not sure of is why when we use a tunnel for transport that we do not see the intervening hops when we run a trace. My "guess" is that the GRE and IPSec encapsulation cause the TTL counter to not get decremented as normally occurs during a trace. Typically, the TTL in the IP header is sent with a value of 1, and when it is returned by the first hop it is resent with a TTL of 2, and so on and so forth, until the final destination router responds "positively." This last point is something I would like to be able to read about but have not been able to find at all on the Internet.

Do you know of a white paper that details the step by step procedure for GRE encapsulation when it is run over IPSec? I mean a real detailed thing...?

What are your thoughts on what I have written so far?

Thanks

Hello Joe,

I think you have found the answer by yourself.

by examining the IP routing table if you see a route like 138.60.x.x using as next hop the ISP provider this is correct:

the only requirement to have stable tunnels is that the ip addresses used as IPSec endpoints (or GRE endpoints if not using tunnel protection) the public ip addresses are not advertised over the tunnel itself.

So this static route is needed to be able to setup the IPSec SA that carries the mGRE tunnel inside.

Failing to do so causes recursive errors and tunnel flaps up/down.

So starting from this info when you trace without specifying a source the nearest interface to the destination is used for the source IP address.

In your case when you don't see the intermediate hops traffic is going inside ipsec/GRE/ip.

When you see the intermediate hops there is no GRE and no ipsec and packets are sent with no additional encapsulation.

I attach for you a presentation from networkers 2k3 that has the diagrams of encapsulation.

It misses some new features but it is very clean explanation of DMVPN

Hope to help

Giuseppe

Thanks, Giuseppe:

"In your case when you don't see the intermediate hops traffic is going inside ipsec/GRE/ip.

When you see the intermediate hops there is no GRE and no ipsec and packets are sent with no additional encapsulation."

I have established this. I was wondering why this is the case. It must have something to do with the TTL counter and how it's decremented when in an IPSec tunnel.

I havent read the document you sent me yet, but it seems OK. I would have liked to find something that has a narrative/dialogue with it. This is a slide presentation that is designed to have someone instructing to fill in the blanks. But it's better than nothing and I thank you for sending it.

Thanks

Hello Joe,

about the TTL with encapsulated packets:

the TTL parameter is not copied in the external IPv4 header so when doing traceroute with TTL=1 the inner packet is examined at the router at the other end of the GRE tunnel for this reason you don't see the intermediate hops.

Hope to help

Giuseppe

Giuseppe:

This is precisely what I suspected. So, the INNER IPv4 header -- the original one that has been encapsulated by GRE -- does not export its TTL counter to the IPSec outer header (ESP header). Correct?

Where did you read this? I haven't been able to find it anywhere...

Thanks

Giuseppe:

NOW you're talkin', my man....

That does make sense.

Thanks for sticking with me on this one.

Have you noticed an interesting pattern?

I ask a question but resolve it myself, with you filling in some of the blanks. And no one else joins in our conversations. Hmmmm

Maybe we got a system you and I. :-)

Grazie.