RVS4000 disabling IPS kills DNS

Answered Question
Jul 29th, 2009

Background: I recently bought a couple of RVS4000 routers set up a VPN between my home and office. Initial setup went great and I got the VPN working with Dynamic DNS at both ends quite easily. The only problem was that the performance sucked badly; my ISP gives me 45Mb/s and a frequently see this in both directions from both speed test sites and SFTP to other servers, and they are planning to upgrade our building to 100Mb/s next month. With the RVS4000 in place I typically get 12Mb/s download and 7Mb/s down. No problem, I thought. I've read that most of the performance problems come from the Intrusion Prevention System having to look at every packet. So I switched the IPS off...

Problem: The problem I am having is that if I switch the IPS off then DNS breaks on every computer on the network :-( Look-ups go so slowly that my web browsers frequently time out. If I access somewhere by IP address then everything is fine and the throughput of the router is acceptable (I've see 28Mb/s when I've actually managed to get resolve the address of a speed test server). Unfortuantely this doesn't do me much good if I have no DNS. I restored the factory configuration for one router and then ran tests after each step of the configuration. Everything was fine until I switch the IPS off. I then restored to factory state again, confirmed everything was fine and changed nothing else except the IPS; when I disabled IPS the DNS died again!

Question: Is there any work-around for this? I pay for 100M/s wiring between my home and office and with the IPS enabled I'm loosing 90% of the performance. If I disable the IPS the device is essentially useless. HELP! Any suggestions welcome. If I can't fix this soon the routers are going to have to go back for a refund.

I have this problem too.
0 votes
Correct Answer by streaves about 4 years 7 months ago

Firmware v 1.3.0.5 is now available for download here: http://www.cisco.com/en/US/products/ps9928/index.html.

Share your thoughts.

Thanks!

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (1 ratings)
David Carr Thu, 07/30/2009 - 06:33

NIck, What firmware version are you running on the rvs4000 router?

nickovs123 Thu, 07/30/2009 - 08:11

The routers came with 1.2.11, which seems to be the latest I can find.

streaves Thu, 07/30/2009 - 08:16

Hi Nick --

Can you please provide your RVS4000 config file and network topology? This will help us troubleshoot.

You can email it to me if you prefer: streaves at cisco dot com.

Thanks,

--Stephanie

nickovs123 Thu, 07/30/2009 - 08:52

Stephanie , I'll download the config file and mail it to you shortly. That said, I can reliably reproduce the problem with a single router attached directly the ISP by restoring factory configuration and then switching off the IPS. No other configuration change is necessary.

FWIW, all the client machines I've tried on which I can carry out any diagnostics are Apple Macs, since that is all we have here. It would appear that the same problems are also occurring on some random devices which contain embedded Linux systems too, but I've not tried Windows.

David Carr Thu, 07/30/2009 - 08:16

Where are you pointing your dns?  Are you pointing it internally or are you pointing it to your isp.  You might try changing the dns servers to possibly 4.2.2.1 or 4.2.2.2 and see if that resolves the issue.

nickovs123 Thu, 07/30/2009 - 08:45

The router is being given DNS settings as part of its DHCP configuration (as it happens the servers are 204.14.152.2 and ...5) so the router DNS is pointing to the ISP. The client machines are all getting DHCP from the router and this comes with DNS settings too. Typically they are receiving these same ISP addresses, although it seems that the sometimes get given the address of the router instead. I will try manually forcing them to 4.2.2.1 and ...2 once I've had another cup of coffee.

gorciscoliksys Tue, 08/04/2009 - 22:36

I have the same problem!

I’m connected via PPPoe to a local internet provider; the speed is limited to 50Mbps.

Till now, I used with succes, a Linksys WRT54GL, withou any problem! But the WAN speed it’s hardware limitted at ~28Mbps.

I bought a new Linksys gateway-router RVS4000 to increase this WAN speed.

Everythins is fine, but the speed is very low with IPS (Intrusion Prevention System) – around 15Mbps. I tryed to disable it, but the DNS is not working enymore!

If I try to reach directly the IP, the speed is very good!

I tryed different firmwares (v1.1.14, 1.2.10, 1.2.11) with restore factory default before and after flashing, but without any success.

On static IP or DHCP on WAN, the speed is very good without IPS.

If I try to acces the web pages directry in web administration page at Diagnostics > Trace Route, the DNS seems to be OK.

But the problem is not present on all platforms...

I have a desktop and a laptop with Windows XP OS and IPS kill DNS.

On my LAN I have a satellite receiver with Linux OS and the DNS is fine with even without IPS.

Also my XBOX 360, reach everything in internet with IPS disabled, but my iPhone is not working.

For the moment, on computers and iPhone I put a manual DNS and everything is fine! For me, this one is ok: 209.244.0.4; 4.2.2.2 is not working sometimes.

nickovs123 Mon, 08/31/2009 - 15:12

Well, after some more investigation I have made a little progress on this, although it is still not actually resolved.

Having attached packet capture tools on both sides of the router and watched the DNS packets going through it seems that there is a bug in the RVS4000 firmware. With the IPS enabled, DNS packets pass through the device with just the source IP address changed (as you would expect due to NAT being enabled) and the checksum values in the IP and UDP headers updated (because the source IP address changed). In this situation everything works fine but the throughput is throttled by the IPS software.  If the IPS is disabled, the router not only changes the source IP address but also changes the source UDP port. Worse, it replaces the randomly chosen, non-privileged port number picked by the client with a sequentially chosen port number, starting at port zero. Thus with IPS disabled the outgoing DNS packets look exactly like the packets that would be used to bounce a DDoS attack off someone else's DNS server and my ISP is deciding that they look bogus and rejecting them.

It's also worth noting that replacing the randomly chosen, non-privileged port number with a sequentially chosen port number, starting at port zero, introduces a serious new security vulnerability. If an exploit is found for a bug in the DNS client software on PCs then the only thing stopping this being as dangerous as the recent bugs in the BIND DNS servers is that it's hard for attackers to know which port to attack. Unfortunately if the client machine is behind an RVS4000 then the attacker doesn't need to know which port is being used because he can just send exploit packets with the source address set the the victim's ISP's DNS server and low, sequentially numbered destination ports and the RVS will conveniently pass the exploit on to the correct port on the victim machine. Thus for this type of exploit, having an RVS with IPS disabled is actually less secure than not having the router there at all.

Re-mapping the source UDP port is unnecessary unless there is a port number collision, and the router manages just fine without re-mapping if IPS is enabled. If the port must be re-mapped it should be replaced with something that is randomly chosen and not a privileged port number, just like a good client would have picked. The current state on the RVS4000 1.2.11 firmware is broken and should be fixed.

David Carr Tue, 09/01/2009 - 07:23

Thank you nick, for your input on the rvs4000.  It is noted and they are working towards releasing a new firmware for fix this issue.

nickovs123 Fri, 09/11/2009 - 08:16

Thanks for the pointer.

FWIW, I had looked there but neither the new firmware version number not the release date appear anywhere on that page. Perhaps it would be worth considering a web site policy about ensuing that links to firmware specify either version number, release date or both.

Anyway, I'll try the new firmware and see if it cures the problem for me.

nickovs123 Fri, 09/11/2009 - 17:17

I can confirm that the V1.3.0.5 firmware cures this problem!

Thanks for the fix, Cisco.

nickovs123 Fri, 09/11/2009 - 17:38

FWIW, with the IPS switched off I just clocked 35Mb/s download and 23Mb/s upload on my 45Mb/s line and there seems to be other traffic on the line. It looks like the bottlenecks are now in the connection to the ISP and not in the router.

gianluca.colangelo Mon, 11/23/2009 - 02:28

Hi all,

I'm running the latest firmware 1.3.0.5 and happened to me already a couple of times that DNS queries don't get any answer. It seems they're just dropped. IPS is disabled. Initially I thought it was my ISP but then bypassing the router everything started to work perfectly. I had to reboot the router to get it going again. So it seems that 1.3.0.5 does not fix completely the issue with DNS. Did anybody experience the same?

Best regards,

Gianluca

David Carr Mon, 11/23/2009 - 07:59

Did you do a factory reset of the router after upgrading the firmware or did you purchase it with that version.  If you did the firmware and did not do the factory reset after the upgrade, do that and reload the configurations in it and see if that resolves it.

gianluca.colangelo Sat, 11/28/2009 - 09:32

Hi all,

I tried the recommendation of doing a factory reset after loading firmware 1.3.0.5 but few days later I had another case of dead DNS. It seems that something goes wrong. What kind of logs should I take in order to let you troubleshoot the problem?

Best regards,

Gianluca

wwwsssxxx Thu, 12/17/2009 - 20:10

I am posting in this thread, because I think the problem is related:

Disabling IPS kills h323 connections in the new firmware: connections from outside using Polycom PVX client to Radvision Click To Meet Conference Server are impossible if IPS is disabled. If IPS is enabled, everything is back to normal.

My config: on firewall port 1720/TCP and 3230-3330/TCP-UDP (only UDP are nessesary AFAIK) are forwarded to the server, port 8080/TCP is forwarded to the same server and it works with or without IPS.

Actions

Login or Register to take actions

This Discussion

Posted July 29, 2009 at 7:15 PM
Stats:
Replies:20 Avg. Rating:5
Views:9366 Votes:0
Shares:0
Tags: dns, ips, rvs4000
+

Related Content

Discussions Leaderboard