Solved: Re: GRE issue

boban-petrovic · ‎12-28-2011

The probelm is following: there is IP connectivity between public IP addresses, x.y.148.202 and x.y.132.202, but not between 10.0.10.1 and 10.0.10.2. Relevant configurations are following:

Router A:

interface Tunnel0

ip address 10.0.10.1 255.255.255.0

tunnel source Vlan1

tunnel destination x.y.132.202

interface Vlan1

ip address x.y.148.202 255.255.255.252

#sh ip int tu0

Tunnel0 is up, line protocol is up

Internet address is 10.0.10.1/24

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 1476 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.5

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is enabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is disabled

IP CEF switching is enabled

IP CEF Feature Fast switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Policy routing is disabled

Network address translation is disabled

BGP Policy Mapping is disabled

#sh int tu0

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 10.0.10.1/24

MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source x.y.148.202 (Vlan1), destination x.y.132.202

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

Router B:

interface Tunnel0

ip address 10.0.10.2 255.255.255.0

tunnel source GigabitEthernet0/0

tunnel destination x.y.148.202

interface GigabitEthernet0/0

description outside

ip address x.y.132.202 255.255.255.252

ip nat outside

ip virtual-reassembly in

duplex auto

speed auto

#sh ip int tu0

Tunnel0 is up, line protocol is up

Internet address is 10.0.10.2/24

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 1476 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is enabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is disabled

IP CEF switching is enabled

IP CEF switching turbo vector

IP Null turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Policy routing is disabled

Network address translation is disabled

BGP Policy Mapping is disabled

Input features: MCI Check

WCCP Redirect outbound is disabled

WCCP Redirect inbound is disabled

WCCP Redirect exclude is disabled

#sh int tu0

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 10.0.10.2/24

MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source x.y.132.202 (GigabitEthernet0/0), destination x.y.148.202

Tunnel Subblocks:

src-track:

Tunnel0 source tracking subblock associated with GigabitEthernet0/0

Set of tunnels with source GigabitEthernet0/0, 1 member (includes iterators), on interface <OK>

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255, Fast tunneling enabled

Tunnel transport MTU 1476 bytes

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

I believe that there is some problem with MTUs, but I'm not so sure. How to correct this configuration?

Dan Frey · ‎12-29-2011

Hi Boban,

The OSPF config on Router A could be an issue. The redistribute connected command also includes the WAN interface IP. Would recommend adding a routemap filter so the WAN IP is not included in the OSPF advertisement.

Is the redistribute static command needed on this router, I only see a default static route?

- Dan

View solution in original post

Peter Paluch · ‎12-28-2011

Boban,

Even if you ping the 10.0.10.2 from the router having the IP address 10.0.10.1 or vice versa, is the ping unsuccessful? Such ping is deep below the MTU. If the ping is unsuccessful, your problem is most probably caused by something different than MTU.

Is there any firewall en route between these two routers that could potentially drop GRE traffic? Are you using any ACLs or firewall mechanisms on these routers?

Best regards,

Peter

boban-petrovic · ‎12-28-2011

Peter,

Thanks for the answer.

The ping is unsuccessful.

There are a couple of ACLs, but for other purposes (OSPF, NAT,..) and neither of them is applied to vlan1 or gi0/0 explicitly.

Both routers are connected to WAN via vlan1 and gi0/0 interfaces to the same ISP, I don't believe that ISPdo not allow gre.

On router B, there is NAT configured. Can that make some problems in this case?

Peter Paluch · ‎12-28-2011

Hello Boban,

Without knowing your entire configuration, it is difficult to say. Would it be possible to post the entire configuration of both your routers, with the sensitive information removed?

Best regards,

Peter

boban-petrovic · ‎12-28-2011

Router A:

#sh run

version 12.4

logging buffered 8192 debugging

no logging console

no logging monitor

!

resource policy

!

mmi polling-interval 60

no mmi auto-configure

no mmi pvc

mmi snmp-timeout 180

ip subnet-zero

ip cef

!

interface Tunnel0

ip address 10.0.10.1 255.255.255.0

ip ospf cost 200

tunnel source Vlan1

tunnel destination x.y.132.202

!

interface FastEthernet0/0

ip address 192.168.0.17 255.255.255.252

ip ospf network non-broadcast

duplex auto

speed auto

!

interface FastEthernet0/1

ip address 192.168.0.3 255.255.255.240

duplex auto

speed auto

!

interface FastEthernet0/1/0

!

interface FastEthernet0/1/1

!

interface FastEthernet0/1/2

!

interface FastEthernet0/1/3

!

interface Vlan1

ip address x.y.148.202 255.255.255.252

!

router ospf 10

log-adjacency-changes

redistribute connected metric-type 1 subnets

redistribute static metric-type 1 subnets

network 10.0.10.0 0.0.0.255 area 0

network 192.168.0.0 0.0.255.255 area 0

!

ip classless

ip route 0.0.0.0 0.0.0.0 x.y.148.201

!

no ip http server

!

access-list 1 permit 192.168.0.0 0.0.255.255

!

control-plane

!

line con 0

line aux 0

line vty 0 4

access-class 1 in

exec-timeout 120 0

login local

transport input all

transport output all

!

end

Router B:

version 15.1

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname routerb

!

boot-start-marker

boot-end-marker

!

logging buffered 51200 warnings

!

no ipv6 cef

ip source-route

ip cef

!

login block-for 100 attempts 10 within 60

!

redundancy

!

ip ssh version 2

!

interface Loopback0

no ip address

!

interface Tunnel0

ip address 10.0.10.2 255.255.255.0

ip ospf cost 200

tunnel source GigabitEthernet0/0

tunnel destination x.y.148.202

!

interface Embedded-Service-Engine0/0

no ip address

shutdown

!

interface GigabitEthernet0/0

description outside

ip address x.x.132.202 255.255.255.252

ip nat outside

ip virtual-reassembly in

duplex auto

speed auto

!

interface GigabitEthernet0/1

no ip address

duplex auto

speed auto

no mop enabled

!

interface GigabitEthernet0/1.2

description Management

encapsulation dot1Q 2

ip address 192.168.2.1 255.255.255.0

!

interface GigabitEthernet0/1.3

description LAN

encapsulation dot1Q 3

ip address 192.168.3.1 255.255.255.0

ip nat inside

ip virtual-reassembly in

!

router ospf 1

router-id 192.168.2.1

redistribute connected subnets route-map control-ospf

network 192.168.0.0 0.0.255.255 area 0.0.0.0

network 10.0.10.0 0.0.0.255 area 0.0.0.0

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

ip nat pool nat-pool x.y.132.202 x.y.132.202 prefix-length 30

ip nat inside source list nat-allowed pool nat-pool overload

ip route 0.0.0.0 0.0.0.0 x.y.132.201

!

ip access-list standard nat-allowed

permit 192.168.3.0 0.0.0.255

ip access-list standard ospf-allowed

permit 192.168.0.0 0.0.255.255

permit 10.0.10.0 0.0.0.255

!

access-list 23 permit 192.168.0.0 0.0.0.255

!

route-map control-ospf permit 10

match ip address ospf-allowed

!

snmp-server community secret RO 23

!

control-plane

!

line con 0

login local

line aux 0

line 2

no activation-character

no exec

transport preferred none

transport input all

transport output pad telnet rlogin lapb-ta mop udptn v120 ssh

stopbits 1

line vty 0 4

access-class 23 in

login local

transport input ssh

transport output telnet ssh

!

scheduler allocate 20000 1000

end

Peter Paluch · ‎12-28-2011

Hi Boban,

I do not see any obvious configuration error here. The NAT should not be a problem but still, for testing purposes, you should temporarily deactivate it, remove all NAT entries from the NAT table using the clear ip nat translation command and try the ping again.

It would also be helpful to try to turn on tunnel debugging using the command debug tunnel and seeing if the tunneled packets are being properly sent from either device - and whether they are received at all on the other device. This output may provide us with some clues as to what is happening.

Best regards,

Peter

boban-petrovic · ‎12-28-2011

Peter,

I turne on debug tunnel on router A, and then tried to ping 10.0.10.2. Here is part of output of sh logging:

*Dec 28 23:19:20.320: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=124)

*Dec 28 23:19:22.320: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=124)

*Dec 28 23:19:24.320: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=124)

*Dec 28 23:19:26.320: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=124)

*Dec 28 23:19:30.112: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=100)

*Dec 28 23:19:40.112: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=100)

*Dec 28 23:19:50.112: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=100)

*Dec 28 23:20:00.112: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=100)

*Dec 28 23:20:10.112: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=100)

*Dec 28 23:20:20.112: Tunnel0: GRE/IP encapsulated x.y.148.202->x.y.132.202 (linktype=7, len=100)

I can see ping with 2 seconds delays, and keepalive packets probably with 10 secs delays after that, but I haven't configured keepalive, and thats strange.

When I do the same on Router B, I can see similiar output.

Dec 28 23:15:17.331: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=100 ttl=250 tos=0x0)

Dec 28 23:15:17.331: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=100 ttl=250)

Dec 28 23:15:17.331: Tunnel0: GRE decapsulated IP packet (linktype=7, len=76)

Dec 28 23:15:27.331: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=100 ttl=250 tos=0x0)

Dec 28 23:15:27.331: Tunnel0: GRE/IP (PS) to decaps x.254.148.202->x.y.132.202 (tbl=0,"default" len=100 ttl=250)

Dec 28 23:15:27.331: Tunnel0: GRE decapsulated IP packet (linktype=7, len=76)

Dec 28 23:15:37.331: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=100 ttl=250 tos=0x0)

Dec 28 23:15:37.331: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=100 ttl=250)

Dec 28 23:15:37.331: Tunnel0: GRE decapsulated IP packet (linktype=7, len=76)

Dec 28 23:15:47.327: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=100 ttl=250 tos=0x0)

Dec 28 23:15:47.327: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=100 ttl=250)

Dec 28 23:15:47.327: Tunnel0: GRE decapsulated IP packet (linktype=7, len=76)

And this is when I try to ping 10.0.10.1:

Dec 28 23:19:20.783: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:19:20.783: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:19:22.783: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:19:22.783: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:19:24.783: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:19:24.783: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:19:26.783: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:19:26.783: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:19:28.783: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:19:28.783: Tunnel0 count tx, adding 0 encap bytes

adding 0 encap bytes looks strange, hmmmm...

I can't deactivate NAT because this routers are in production environment.

boban-petrovic · ‎12-28-2011

And this is the output on router B when I ping 10.0.10.2 from router A:

Dec 28 23:30:29.647: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=124 ttl=250 tos=0x0)

Dec 28 23:30:29.647: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=124 ttl=250)

Dec 28 23:30:29.647: Tunnel0: GRE decapsulated IP packet (linktype=7, len=100)

Dec 28 23:30:29.647: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:30:29.647: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:30:31.643: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=124 ttl=250 tos=0x0)

Dec 28 23:30:31.643: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=124 ttl=250)

Dec 28 23:30:31.643: Tunnel0: GRE decapsulated IP packet (linktype=7, len=100)

Dec 28 23:30:31.643: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:30:31.643: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:30:33.643: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=124 ttl=250 tos=0x0)

Dec 28 23:30:33.643: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=124 ttl=250)

Dec 28 23:30:33.643: Tunnel0: GRE decapsulated IP packet (linktype=7, len=100)

Dec 28 23:30:33.643: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:30:33.643: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:30:35.643: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=124 ttl=250 tos=0x0)

Dec 28 23:30:35.643: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=124 ttl=250)

Dec 28 23:30:35.643: Tunnel0: GRE decapsulated IP packet (linktype=7, len=100)

Dec 28 23:30:35.643: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:30:35.643: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:30:37.643: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=124 ttl=250 tos=0x0)

Dec 28 23:30:37.643: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=124 ttl=250)

Dec 28 23:30:37.643: Tunnel0: GRE decapsulated IP packet (linktype=7, len=100)

Dec 28 23:30:37.643: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:30:37.643: Tunnel0 count tx, adding 0 encap bytes

So we can see that router B receives pings via tunnel and decapsulate them. After that, Router B encapsulate ping replies, but add 0 encap bytes. Now, I will check what is happening with ping replies on router A.

Peter Paluch · ‎12-28-2011

Hi Boban,

The GRE packets sent in 10-second intervals are most probably OSPF Hello packets, as you have OSPF running over the tunnel. Most probably, these are not keepalives.

The information about adding 0 encapsulation bytes is strange indeed.

Let's make another experiment. On Router B, activate these debugs:

debug tunnel

debug ip icmp

Then, from Router A, ping Router B and capture the debug output.

Best regards,

Peter

boban-petrovic · ‎12-28-2011

Hi,

this is the output:

Dec 28 23:45:39.303: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=124 ttl=250 tos=0x0)

Dec 28 23:45:39.303: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=124 ttl=250)

Dec 28 23:45:39.303: Tunnel0: GRE decapsulated IP packet (linktype=7, len=100)

Dec 28 23:45:39.303: ICMP: echo reply sent, src 10.0.10.2, dst 10.0.10.1, topology BASE, dscp 0 topoid 0

Dec 28 23:45:39.303: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:45:39.303: Tunnel0 count tx, adding 0 encap bytes

Dec 28 23:45:41.303: Tunnel0: GRE/IP to classify x.y.148.202->x.y.132.202 (tbl=0,"Default" len=124 ttl=250 tos=0x0)

Dec 28 23:45:41.303: Tunnel0: GRE/IP (PS) to decaps x.y.148.202->x.y.132.202 (tbl=0,"default" len=124 ttl=250)

Dec 28 23:45:41.303: Tunnel0: GRE decapsulated IP packet (linktype=7, len=100)

Dec 28 23:45:41.303: ICMP: echo reply sent, src 10.0.10.2, dst 10.0.10.1, topology BASE, dscp 0 topoid 0

Dec 28 23:45:41.303: Tunnel0: GRE/IP encapsulated x.y.132.202->x.y.148.202 (linktype=7, len=124)

Dec 28 23:45:41.303: Tunnel0 count tx, adding 0 encap bytes

Router B successfuly receives ping reply and tries to answer it. It encapsulates ping response, but Router A never receives that response. I think that adding 0 encap bytes is the key problem.

boban-petrovic · ‎12-28-2011

I've added network 10.0.10.0 0.0.0.255 area 0 to OSPF proces on router B, and router A become the OSPF neighbour to router B, but in INIT state, which shows that tunnel is functionig good in A->B direction, but not otherwise.

Peter Paluch · ‎12-28-2011

Boban,

Thank you for all your investigation. I agree with your assessment of the situation: that the B->A direction is not working. The "0 encap bytes" seems to be rather suspicious to me. However, I see no direct way of solving this problem. This seems to me to be an IOS bug.

One possibility to try is to delete and recreate the Tunnel interface on Router B, possibly using a different tunnel interface number (i.e. not Tun0 but, e.g., Tun1).

Another possibility - potentially disruptive and dangerous - is to temporarily disable and reenable CEF. The CEF is heavily involved in the process of encapsulating the GRE-tunelled packets, and this procedure will erase and repopulate all CEF structures, supposing there may be a problematic entry. Alternatively, you may want to try to erase the CEF structures manually using two commands: clear cef table ipv4 and clear adjacency

Considering the fact that this behavior is obviously erroneous, I would also suggest reloading the device and/or changing the IOS version (by the way, what is the exact IOS version on Router A and B?). Obviously, this needs to be done in a maintenance window.

Best regards,

Peter

boban-petrovic · ‎12-29-2011

Peter,

Thanks for your answers.

I've tried to delete Tunnel 0 and then create Tunnel 1, but problem persisted.

I'll try to reload the router B, when working conditions allow it.

Router B:

#sh ver
Cisco IOS Software, C1900 Software (C1900-UNIVERSALK9-M), Version 15.1(4)M2, RELEASE SOFTWARE (fc1)

Router A:

#sh ver

Cisco IOS Software, 1841 Software (C1841-IPBASE-M), Version 12.4(3i), RELEASE SOFTWARE (fc2)

Message was edited by: Boban Petrovic; Router B is 15.1, and router A is 12.4

Peter Paluch · ‎12-29-2011

Boban,

Thank you for the info. Regarding the IOS version on the router B, it is strongly outdated and I would personally strongly vouch for upgrading it, along with the reload.

Best regards,

Peter

EDIT: The 12.4(3i) is outdated The 15.1 IOS is one of the current IOSes but obviously buggy. I would also consider upgrading it to a newer 15.1M release if possible.

Dan Frey · ‎12-29-2011

Hi Boban,

The OSPF config on Router A could be an issue. The redistribute connected command also includes the WAN interface IP. Would recommend adding a routemap filter so the WAN IP is not included in the OSPF advertisement.

Is the redistribute static command needed on this router, I only see a default static route?

- Dan