Firewall installed after Internet border Cisco with stateful firewall enabled is blocking some packets. Why?

Answered Question
Jun 28th, 2010

Hi,

I have an old issue I can not explain.

I have the following Internet connection topology:

LAN - core Cisco 1811 - Wingate Proxy/Firewall - Internet border Cisco 871 - ISP.

On core Cisco 1811 I have stateful firewall enabled and filter blocking everything on its interface looking at Wingate:

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman";}

ip inspect name internet tcp

ip inspect name internet udp

ip inspect name internet icmp

+

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman";}

interface Vlan8

description To_Internet

ip address 192.168.200.10 255.255.255.224

ip access-group vlan8_in in

no ip redirects

no ip unreachables

no ip proxy-arp

ip inspect internet out

ip virtual-reassembly

+

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman";}

ip access-list extended vlan8_in

deny   ip any any

On Wingate I have http Proxy enabled plus NAT service enabled (clients mainly use Proxy connection method, only 3-4 - NAT).

On the border Cisco 871 I have NAT, stateful firewall enabled and filter blocking  everything on its interface looking at Internet (I attach the whole config just in case, it is not very large. Add. info: we use 2 ISPs, switching is done by means of IP SLA and pinging 2 Internet Root DNS servers):

Now, with initially setup border router (after applying "no ip unreachables") the Firewall section of Wingate, where blocked packets are shown, was totally clean!

After some changes, I mentioned hundreds of packets are blocked by Wingate and almost all are from well known web servers ports 80 and 443. Obviously, these packets are responses for our clients requests for some reason blocked by Wingate (IMHO, Wingate just can't see they are responses for the made requests).

Please help me to find out the reason. This is important, because we have to make some procedures everyday to be sure security is ok, plus we get quite large log files of Wingate, etc.

After long thinking I can suppose the following reasons:

1. double NAT (for some clients NAT is done on Wingate and then on the border Cisco).

2. NAT timeouts on border Cisco (I changed some settings:

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman";}

ip nat translation timeout 14400

ip nat translation tcp-timeout 14400

ip nat translation icmp-timeout 1800)

3. CEF enabled on the border Cisco (I read somewhere that some conflicts happens with stateful firewall)

4. "No ip unreachables" together with MTU issue.

5. Anything else.

All the possible reasons are just my fantasies, I am not sure. And I don't want to blindly change everything.

Please help to find the reason.

Connection topology schema added.

I have this problem too.
0 votes
Correct Answer by duffman55 about 3 years 9 months ago

I don't see anything obvious, but since you have a few hypotheses as to what the problem might be, we may as well start testing them:

- try turning CEF off on the internet-facing router.  You'll probably see a little performance hit, but it's better than dropping packets.  Honestly, I don't think this is it, though; I've heard of CEF giving problems with load-sharing on multiple paths, but I've never seen it be a problem with a firewall (limited experience, though).  If this doesn't fix it, turn CEF back on.

- try disabling all IP inspection on the router (temporarily!).  This really shouldn't be the problem, as inspection doesn't rewrite any packets AFAIK; it just keeps a session table so the firewall knows what traffic to let back inside.

- you could also try just using the Wingate as a proxy server only, and leaving the firewalling up to the border router.  If you're worried about receiving alerts and stuff, you can turn alerts on on the router (I think they're on by default), and syslogging them to a server somewhere.

- I think the NAT timeouts on the router are more than long enough, and I don't think that's the problem anyway.

Good luck!

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (1 ratings)
duffman55 Wed, 07/07/2010 - 18:39

If the problem manifests itself on the Wingate, that would be where I would look first.  Gotta wonder why you're doing NAT twice, though.  Seems like there should be a simpler solution than that.

What, specifically, were "some changes" that you made prior to the Wingate dropping packets?

Also, I have a very similar dual-ISP setup - 2 IP SLA's with tracking - but I ping our ISP's DNS resolvers instead.  The root nameservers get enough traffic as it is; I would feel wrong pinging those for an IP SLA like that.

alen.danielyan@... Thu, 07/08/2010 - 03:44

Dear duffman55,

Thank you for your reply.

duffman55 wrote:

If the problem manifests itself on the Wingate, that would be where I would look first.  Gotta wonder why you're doing NAT twice, though.  Seems like there should be a simpler solution than that.


It may be on Wingate side, but I think border Cisco somehow corrupts or modifies (or may be changes fragmented packets delivery order, or something else), which makes Wingate to block and drop the packets.

We get ~ 100 - 200 drops each day and this is obviously less than 0.1% of the whole traffic (besides, no one ever says that he's getting errors while trying to open any app or site), so it is not critical for the Internet connection functioning. But we have some procedures to do, when firewall is blocking packets to find out the reason, initiator, etc. and it is becoming annoying...

About double NAT. Wingate is good as we have GUI plus logging for each user activity. Actually Winagte is working as www, ftp, pop3 and smtp proxy, so for the main part of users NAT is made only once (on the border Cisco for Wingate). But a couple of users need NAT. There are 2 ways: either send the NAT traffic round Wingate (which is not convenient and is wrong, IMHO), or give access for the users to Wingate NAT service. And in the latter case we get double NAT.

But, research showed that bloked packets are not necessarily Wingate NATed, the most part are of proxy servers.

duffman55 wrote:

What, specifically, were "some changes" that you made prior to the Wingate dropping packets?

I am not sure, but I "undid" all changes and it did not help.

I was thinking mainly about removed "ip inspect http", but after setting it back nothing changed. Wingate is still blocking packets, most part of which are from TCP 80 and 443 - obviously web traffic in response of our clients requests.

duffman55 wrote:

Also, I have a very similar dual-ISP setup - 2 IP SLA's with tracking - but I ping our ISP's DNS resolvers instead.  The root nameservers get enough traffic as it is; I would feel wrong pinging those for an IP SLA like that.

We set it purposely: in our places it is more usual when you have connectivity to ISP, but the latter has not connection to Internet itself, than situations when you have no connection to ISP.

Do you see anything (else) unusual in the config, which can be the reason of the issue?

Correct Answer
duffman55 Thu, 07/08/2010 - 05:08

I don't see anything obvious, but since you have a few hypotheses as to what the problem might be, we may as well start testing them:

- try turning CEF off on the internet-facing router.  You'll probably see a little performance hit, but it's better than dropping packets.  Honestly, I don't think this is it, though; I've heard of CEF giving problems with load-sharing on multiple paths, but I've never seen it be a problem with a firewall (limited experience, though).  If this doesn't fix it, turn CEF back on.

- try disabling all IP inspection on the router (temporarily!).  This really shouldn't be the problem, as inspection doesn't rewrite any packets AFAIK; it just keeps a session table so the firewall knows what traffic to let back inside.

- you could also try just using the Wingate as a proxy server only, and leaving the firewalling up to the border router.  If you're worried about receiving alerts and stuff, you can turn alerts on on the router (I think they're on by default), and syslogging them to a server somewhere.

- I think the NAT timeouts on the router are more than long enough, and I don't think that's the problem anyway.

Good luck!

alen.danielyan@... Thu, 07/08/2010 - 06:36

Dear duffman55

Thank you very much for your versions.

I forgot about CEF myself. I'll check everything and let you know tomorrow.

After my first post I read a lot about NAT timeouts and I don't think it is the reason either.

duffman55 wrote:

- try disabling all IP inspection on the router (temporarily!).  This really shouldn't be the problem, as inspection doesn't rewrite any packets AFAIK; it just keeps a session table so the firewall knows what traffic to let back inside.

Wait a moment, if I disable inspection I will not have Internet connection at all. Or you want me to check if droped packets were wrongly passed through the Cisco's filter (and they are not responses on our requests)?

duffman55 wrote:

- you could also try just using the Wingate as a proxy server only, and leaving the firewalling up to the border router.  If you're worried about receiving alerts and stuff, you can turn alerts on on the router (I think they're on by default), and syslogging them to a server somewhere.

I would like to keep it as a second frontier of defense. IMHO, it's good to have 2 totally different firewalls.

duffman55 Thu, 07/08/2010 - 07:09
Wait a moment, if I disable inspection I will not have Internet connection at all. Or you want me to check if droped packets were wrongly passed through the Cisco's filter (and they are not responses on our requests)?

Correct - you'll have to take the ACL off, too.  Again, this would be strictly temporary, to see whether the Cisco firewall is what's causing the traffic to get dropped.  You need to nail down the exact problem.

I doubt that the drops are malicious packets, as they shouldn't have gotten through the router's ACL.  The only things you have allowed through the outside interfaces are NTP and ICMP, so you wouldn't be seeing HTTP and HTTPS traffic coming in - unless it was allowed by the firewall.

duffman55 wrote:

- you could also try just using the Wingate as a proxy server only, and leaving the firewalling up to the border router.  If you're worried about receiving alerts and stuff, you can turn alerts on on the router (I think they're on by default), and syslogging them to a server somewhere.

I would like to keep it as a second frontier of defense. IMHO, it's good to have 2 totally different firewalls.

Agreed.  But it's one more thing to try, if we can't figure out how to resolve this!

alen.danielyan@... Tue, 07/13/2010 - 01:42

Report:

- "no ip cef" did not change anything.

- Disabling stateful firewall prevents Internet connection, no packets were blocked by Wingate, but I could wait for only 5 minutes, which is not enough (I couldn't wait more because I am doing tests during business hours to reproduce the issue). Anyway, I am almost sure Cisco's ACLs on external interfaces work fine and do not pass any unauthorized packet.

- I just canceled my NAT timeouts, will see if anything changes.

In the last 2 days I found out 2 interesting facts:

1. After disabling statefull firewall ("no ip inspect Internet out") and thus closing Internet connection at all and then enabling it, Wingate did not drop any packets during ~ 20 minutes and then started again. Of course, that may happen occasionally, I'll recheck. But that is very interesting and showes, IMHO, the issue has something to do with timeouts (I don't know which ones).

2. Yesterday one of my admins installed Skype on an additional PC and I started to receive a huge amount of incoming packets blocked by Wingate. After some research I found out that, Skype is making transmissions on behalf of other skype users (this is default behaviour and is called "Supernode mode"). Now, the interesting part is, nevertheless Skype was set up to work via web proxy it was trying to make outgoing NAT connections, which were blocked by Wingate NAT service module as not allowed for that user, plus Wingate Firewall module was also  blocking quite big amount of incoming connections! (Obviously those were "independent" incoming connections)

Why did not the border Cisco block those incoming connections? And why did Wingate? (I think in this case the correct behaviour was the one of Wingate).

P.S.

Using Skype behind a NAT - note for home users of Skype behind a router
"A firewall that blocks inbound traffic or uses NAT (Network Address Translation) also won't stop Skype. When a Skype client starts,
it opens a session with a supernode in the Skype network. If the client cannot be contacted from the Internet, the supernode will notify the client when a call comes in via the open connection. If the recipient cannot directly contact the sender, the supernode or a relay agent can then act as a proxy between the two callers. "

BTW, a week ago I had enabled http, https and some other used 7-th layer protos  inspections on my border Cisco. So the Cisco router was able to make http / https inspections either...

It seems to me, our issue could began not after some changes in the router config, but after installation of some programs! Some P2P clients or something like that!

Now, the question is, in case this is true, how to make Cisco block such packets?

duffman55 Tue, 07/13/2010 - 04:42

I don't know what version of IOS you have on your internet-facing router, but NBAR can identify Skype traffic on more recent releases with a simple "match protocol skype" in your class-map.  If your NBAR doesn't have that capability, you should be able to load a PDLM to enable that.  As for other types of P2P traffic, like kazaa or gnutella or something, NBAR can identify those, too.  Same process.  Good luck!

alen.danielyan@... Tue, 07/13/2010 - 05:52

duffman55 wrote:

I don't know what version of IOS you have on your internet-facing router, but NBAR can identify Skype traffic on more recent releases with a simple "match protocol skype" in your class-map.  If your NBAR doesn't have that capability, you should be able to load a PDLM to enable that.  As for other types of P2P traffic, like kazaa or gnutella or something, NBAR can identify those, too.  Same process.  Good luck!

That was just my suggestion, I am not sure the problem started after some apps installation. Besides, today I checked some blocked ips again - they belong to usual web servers, so it is not a fact the reason is p2p apps.

P.S. Anyway, I cann't understand, why Wingate blocked Skype incoming traffic and Cisco did not (Skype was functioning normally, so Wingate blocked really "unuseful" traffic.)

alen.danielyan@... Wed, 07/14/2010 - 06:17

Today I got Qbic answer:

check the TCP flags for these blocked packets. The usual scenario for this is when a connection is closed but a late RST packet is received, which then shows as blocked.

I checked and indeed, almost all packets have "R" flag, which I believe means RST. So the question is: how to make border Cisco to drop that timed out RST packets (not all reset packets, just timed out)? And is it possible at all?

alen.danielyan@... Wed, 07/21/2010 - 00:36

I found out that almost all of the blocked packets have R and a few RA flags. I asked Qbic (creators of Wingate) about the issue, here is the answer:

Check the TCP flags for these blocked packets. The usual scenario for this is when a connection is closed but a late RST packet is received, which then shows as blocked.

An RA packet is sent when an R packet is received. You will see an RA packet being blocked for the same reason as an R packet. The session in the firewall has already timed out or been closed by the firewall, so the packet isn't associated with any open sessions and is dropped.


I know that juniper routers have an option called "TCP RST Invalidate Session Immediately" which causes sessions to be closed immediately when an RST packet is received. It sounds like something similar might be enabled on the path of these sessions (probably on the remote side if WinGate is getting the warnings). If your local Cisco has an option like this, try toggling it, which may cause it to deal with extraneous R / RA packets itself.

Is there a way to make my border Cisco 871 to detect and block the timed out (sic!) Reset and Reset Acknowledgement packets?

P.S. I wonder, how should Cisco know the firewall has closed some sessions...

duffman55 Wed, 07/21/2010 - 04:43

I just saw something related to this yesterday, actually. Cisco routers time out TCP NAT entries after 30 or 60 seconds (don't remember which) when they see a RST or FIN flag in a stream. They don't time those out immediately by default.

I'm not sure, but I would imagine that if you make those time out immediately - or set it to maybe 1 or 2 seconds - then the router's NAT will drop extraneous RST segments. If that's, indeed, what you're getting.

The command is:

ip nat translation finrst-timeout

That may solve your issue; give it a shot!

alen.danielyan@... Wed, 07/21/2010 - 06:07

Thank you, I'll try it and report.

P.S. Unfortunately it is not possible to set finrst-timeout equal to immediately.

alen.danielyan@... Wed, 07/21/2010 - 06:40

No success.

If it even helped, it helped a little bit. Still tens of blocked packets in 30 minutes.

May be it could help if I could set the timeout value to immediately, don't know.

It is also possible that "ip nat translation  finrst-timeout" has no influence in this case, as it concerns to Cisco router's nat timeouts, while blocked packets are blocked because they are timed out for Wingate (I still cann't understand how Cisco could know the session is timed out for Wingate!?)...

duffman55 Wed, 07/21/2010 - 07:32

You can't set it to "immediately", but you can set it to "0".  It wants a value in seconds.

ip nat translation finrst-timeout 0


I would think that if the NAT translations get timed out, then the router will drop extraneous packets simply because it wouldn't know where the packets are supposed to go.

If that doesn't solve the problem, I'm out of ideas!

alen.danielyan@... Wed, 07/21/2010 - 07:43

duffman55 wrote:

You can't set it to "immediately", but you can set it to "0".  It wants a value in seconds.

ip nat translation finrst-timeout 0


I would think that if the NAT translations get timed out, then the router will drop extraneous packets simply because it wouldn't know where the packets are supposed to go.

If that doesn't solve the problem, I'm out of ideas!

"ip nat translation  finrst-timeout 0" effectively equals to "never"!

duffman55 Wed, 07/21/2010 - 08:14

Ah, darn. Yeah, I don't really know where to go from here, then - sorry!

alen.danielyan@... Sun, 07/25/2010 - 05:27

Dear duffman55,

As we defined the problem is not in Cisco router (which still does not mean it is impossible to solve the problem on the Cisco side), I mark your answer about possible variants as correct.

Meanwhile, I got an advice to increase TCP timeout from inside Wingate (there is such setting in the advanced settings, I never looked there). Increasing it up to 10 minutes did not change anything. I'll keep trying and report if anything changes.

Thank you for your help.

Actions

Login or Register to take actions

This Discussion

Posted June 28, 2010 at 4:25 AM
Stats:
Replies:17 Avg. Rating:5
Views:1172 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard

Rank Username Points
1 15,007
2 8,155
3 7,745
4 7,083
5 6,742
Rank Username Points
140
80
78
64
40