Re: Problems with Zone based Firewall and mtr (mytraceroute)

Stefan Giera · ‎01-26-2012

We are using ZFW on an ASR1001 and have experienced a problem: when I try to use mtr (mytraceroute, see

http://en.wikipedia.org/wiki/MTR_%28software%29), I am getting packetloss on all hops between the source and the destination. e.g.:

<code>

Packets Pings

Host Loss% Snt Last Avg Best Wrst StDev

1. Stuttgart-I28-1.belwue.de 100.0 8 0.0 0.0 0.0 0.0 0.0

2. Stuttgart-AL30-1-gi0-0-0-3.belwue.net 100.0 7 0.0 0.0 0.0 0.0 0.0

3. Karlsruhe-RZ-1-10GE-0-1-0-1.belwue.net 100.0 7 0.0 0.0 0.0 0.0 0.0

4. Karlsruhe1-10GE-4-0-0.belwue.net 100.0 7 0.0 0.0 0.0 0.0 0.0

5. Mannheim1-10GE-3-0-0.belwue.net 100.0 7 0.0 0.0 0.0 0.0 0.0

6. Frankfurt-DECIX-1-10GE-0-0-0-0.belwue.net 100.0 7 0.0 0.0 0.0 0.0 0.0

7. de-cix20.net.google.com 100.0 7 0.0 0.0 0.0 0.0 0.0

8. 72.14.238.230 100.0 7 0.0 0.0 0.0 0.0 0.0

9. 72.14.239.62 100.0 7 0.0 0.0 0.0 0.0 0.0

10. 209.85.242.187 100.0 7 0.0 0.0 0.0 0.0 0.0

11. ???

12. ???

13. ???

14. bk-in-f94.1e100.net 0.0% 7 20.0 20.6 20.0 21.2 0.4

</code>

So it seems that the Firewall on my asr1001 is throwing away all packets with ttl-exceeded coming back from hops in between, they have another destination address.

At the moment I am inspecting all kind of traffic from my network outgoing:

!

ip access-list extended 101

permit ip any any

!

class-map type inspect match-all cmap1

match access-group name 101

!

policy-map type inspect pmap1

class type inspect cmap1

inspect

!

etc... (zones, zone-pair in-out with policies applied)

So I tried to let pass all icmp-traffic from the outside to my network:

!

class-map type inspect match-all cmap_icmp

match protocol icmp

!

policy-map type inspect pmap2

class type inspect cmap_icmp

pass

!

etc... (zones, zone-pair out-in with policies applied)

So this has no effect, but I tested and I could figure out, that when I pass all icmp-traffic from my network to the outside, THEN mtr does work.

BUT then normal ping does not work anymore, because it will not be inspected any more.

But I want to have a secure Firewall with inspecting echo-replys and working mtr anyway.

Has anyone the same problem or can even solve this issue?

Thanks in advance,

Stefan

andrew.prince · ‎01-26-2012

In the URL you provided I noticed

"MTR also has a UDP mode (invoked with "-u" on the command line or pressing the "u" key in the curses interface) that sends UDP packets, each with an increasing destination port, toward the destination host. When the UDP mode is used, MTR relies on ICMP port unreachable packets (type 3, code 3) when the destination is reached"

have you tried this?

Stefan Giera · ‎01-26-2012

Yes I tried this and it worked fine, but our border-routers are filtering udp-packets, so this isn't favorable for our NOC...

Sent from Cisco Technical Support iPhone App

andrew.prince · ‎01-26-2012

Have you tried to "inspect" ICMP outbound and "pass" ICMP time-exceeded inbound.

Stefan Giera · ‎01-27-2012

Hi Andrew, thanks for Your answer...

So I have now:

class-map type inspect match-any cmap_icmp

match access-group name icmp_types

ip access-list extended icmp_types

permit icmp any any ttl-exceeded

PMAP IN--> OUT

(don't be confused, my "vlanxxx_pmap_in" is the pmap FROM my network TO the outside...)

policy-map type inspect vlan664_pmap_in

class type inspect vlan664_cmap_in (this is an extended ACL "permit ip x.x.x.x any")

inspect

class type inspect ipsec_cmap_in (this is because I have problems with VPN when inspected, another problem...)

pass log

class class-default

drop log

PMAP OUT-->IN

policy-map type inspect vlan664_pmap_out

class type inspect cmap_icmp (here comes the "ttl-exceeded"-ACL)

pass log

class type inspect vlan664_cmap_out (some open ports for some clients)

inspect

class type inspect ipsec_cmap_out (same problem with VPN when inspected)

pass log

class class-default

drop log

But unfortunately, the same problem occurs. Curiously, the first two packets seem to go "through" the firewall, but with 3rd packet the packetloss comes up:

Packets Pings

Host Loss% Snt Last Avg Best Wrst StDev

1. Stuttgart-I28-1.belwue.de 50.0% 3 0.3 0.3 0.3 0.3 0.0

2. Stuttgart-AL30-1-gi0-0-0-3.belwue.net 50.0% 3 0.9 0.9 0.9 0.9 0.0

3. Karlsruhe-RZ-1-10GE-0-1-0-1.belwue.net 0.0% 2 2.7 2.7 2.7 2.7 0.0

4. Karlsruhe1-10GE-4-0-0.belwue.net 0.0% 2 1.5 1.5 1.5 1.5 0.0

5. Mannheim1-10GE-3-0-0.belwue.net 0.0% 2 2.5 2.5 2.5 2.5 0.0

6. Frankfurt-DECIX-1-10GE-0-0-0-0.belwue.net 0.0% 2 4.1 4.1 4.1 4.1 0.0

7. de-cix20.net.google.com 0.0% 2 5.0 5.0 5.0 5.0 0.0

8. 72.14.238.44 0.0% 2 39.2 39.2 39.2 39.2 0.0

9. 72.14.236.68 0.0% 2 5.4 5.4 5.4 5.4 0.0

10. 209.85.254.118 0.0% 2 5.4 5.4 5.4 5.4 0.0

11. ???

12. google-public-dns-a.google.com 0.0% 2 5.5 5.3 5.2 5.5 0.2

Packets Pings

Host Loss% Snt Last Avg Best Wrst StDev

1. Stuttgart-I28-1.belwue.de 66.7% 4 0.3 0.3 0.3 0.3 0.0

2. Stuttgart-AL30-1-gi0-0-0-3.belwue.net 66.7% 4 0.8 0.8 0.8 0.8 0.0

3. Karlsruhe-RZ-1-10GE-0-1-0-1.belwue.net 66.7% 4 2.1 2.1 2.1 2.1 0.0

4. Karlsruhe1-10GE-4-0-0.belwue.net 66.7% 4 1.5 1.5 1.5 1.5 0.0

5. Mannheim1-10GE-3-0-0.belwue.net 66.7% 4 2.6 2.6 2.6 2.6 0.0

6. Frankfurt-DECIX-1-10GE-0-0-0-0.belwue.net 66.7% 4 4.2 4.2 4.2 4.2 0.0

7. de-cix20.net.google.com 66.7% 4 5.3 5.3 5.3 5.3 0.0

8. 72.14.238.44 66.7% 4 70.3 70.3 70.3 70.3 0.0

9. 72.14.239.60 66.7% 4 5.8 5.8 5.8 5.8 0.0

10. 209.85.254.116 66.7% 4 5.8 5.8 5.8 5.8 0.0

11. ???

12. google-public-dns-a.google.com 0.0% 4 6.3 5.7 5.2 6.3 0.5

In the sessions on the routers, I see only this entry:

Session 206F66C (129.143.6.89:8)=>(8.8.8.8:0) icmp SIS_OPEN

Any other suggestions?

andrew.prince · ‎01-27-2012

What are the results with normal trace route? I use "wintrace" as an example

Stefan Giera · ‎01-27-2012

Normal traceroute looks good:

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets

1 Stuttgart-I28-1.belwue.de (129.143.6.65) 0 ms 0 ms 0 ms

2 Stuttgart-AL30-1-gi0-0-0-3.belwue.net (129.143.56.225) 1 ms 1 ms 1 ms

3 Karlsruhe-RZ-1-10GE-0-1-0-1.belwue.net (129.143.57.46) 2 ms 2 ms 2 ms

4 Karlsruhe1-10GE-4-0-0.belwue.net (129.143.57.50) 2 ms 1 ms 1 ms

5 Mannheim1-10GE-3-0-0.belwue.net (129.143.1.174) 3 ms 3 ms 3 ms

6 Frankfurt-DECIX-1-10GE-0-0-0-0.belwue.net (129.143.1.22) 12 ms 12 ms 9 ms

7 de-cix20.net.google.com (80.81.193.108) 5 ms 5 ms 5 ms

8 72.14.238.230 (72.14.238.230) 5 ms (TOS=128!) 72.14.238.44 (72.14.238.44) 59 ms 72.14.238.46 (72.14.238.46) 5 ms

9 72.14.239.62 (72.14.239.62) [MPLS: Label 684488 Exp 4] 28 ms 72.14.236.68 (72.14.236.68) [MPLS: Label 474319 Exp 4] 5 ms 72.14.239.62 (72.14.239.62) [MPLS: Label 669096 Exp 4] 6 ms

10 209.85.254.114 (209.85.254.114) 5 ms 209.85.254.118 (209.85.254.118) 6 ms 209.85.254.112 (209.85.254.112) 5 ms

11 * * *

12 google-public-dns-a.google.com (8.8.8.8) 5 ms 5 ms 5 ms

andrew.prince · ‎01-27-2012

Stefan Giera · ‎01-30-2012

But what can I do now?

Normal traceroute is good, but mtr is throwing away all reply ttl-exceeded packets after the 2nd reply-packet.

I also tried to set high the icmp idle and ageout-timers with a parameter-map, but this does not help either :-(

andrew.prince · ‎01-30-2012

Well you could just use normal traceroute - it does the same thing. You have other options to look at, like wintrace or pingplotter - they all do the same thing.

JMTPW.