Solved: RVS4000 disabling IPS kills DNS

nickovs123 · ‎07-29-2009

Background: I recently bought a couple of RVS4000 routers set up a VPN between my home and office. Initial setup went great and I got the VPN working with Dynamic DNS at both ends quite easily. The only problem was that the performance sucked badly; my ISP gives me 45Mb/s and a frequently see this in both directions from both speed test sites and SFTP to other servers, and they are planning to upgrade our building to 100Mb/s next month. With the RVS4000 in place I typically get 12Mb/s download and 7Mb/s down. No problem, I thought. I've read that most of the performance problems come from the Intrusion Prevention System having to look at every packet. So I switched the IPS off...

Problem: The problem I am having is that if I switch the IPS off then DNS breaks on every computer on the network :-( Look-ups go so slowly that my web browsers frequently time out. If I access somewhere by IP address then everything is fine and the throughput of the router is acceptable (I've see 28Mb/s when I've actually managed to get resolve the address of a speed test server). Unfortuantely this doesn't do me much good if I have no DNS. I restored the factory configuration for one router and then ran tests after each step of the configuration. Everything was fine until I switch the IPS off. I then restored to factory state again, confirmed everything was fine and changed nothing else except the IPS; when I disabled IPS the DNS died again!

Question: Is there any work-around for this? I pay for 100M/s wiring between my home and office and with the IPS enabled I'm loosing 90% of the performance. If I disable the IPS the device is essentially useless. HELP! Any suggestions welcome. If I can't fix this soon the routers are going to have to go back for a refund.

streaves · ‎09-11-2009

Firmware v 1.3.0.5 is now available for download here: http://www.cisco.com/en/US/products/ps9928/index.html.

Share your thoughts.

Thanks!

View solution in original post

David Carr · ‎07-30-2009

NIck, What firmware version are you running on the rvs4000 router?

nickovs123 · ‎07-30-2009

The routers came with 1.2.11, which seems to be the latest I can find.

David Carr · ‎07-30-2009

Where are you pointing your dns? Are you pointing it internally or are you pointing it to your isp. You might try changing the dns servers to possibly 4.2.2.1 or 4.2.2.2 and see if that resolves the issue.

nickovs123 · ‎07-30-2009

The router is being given DNS settings as part of its DHCP configuration (as it happens the servers are 204.14.152.2 and ...5) so the router DNS is pointing to the ISP. The client machines are all getting DHCP from the router and this comes with DNS settings too. Typically they are receiving these same ISP addresses, although it seems that the sometimes get given the address of the router instead. I will try manually forcing them to 4.2.2.1 and ...2 once I've had another cup of coffee.

streaves · ‎07-30-2009

Hi Nick --

Can you please provide your RVS4000 config file and network topology? This will help us troubleshoot.

You can email it to me if you prefer: streaves at cisco dot com.

Thanks,

--Stephanie

nickovs123 · ‎07-30-2009

Stephanie , I'll download the config file and mail it to you shortly. That said, I can reliably reproduce the problem with a single router attached directly the ISP by restoring factory configuration and then switching off the IPS. No other configuration change is necessary.

FWIW, all the client machines I've tried on which I can carry out any diagnostics are Apple Macs, since that is all we have here. It would appear that the same problems are also occurring on some random devices which contain embedded Linux systems too, but I've not tried Windows.

gorciscoliksys · ‎08-04-2009

I have the same problem!

I’m connected via PPPoe to a local internet provider; the speed is limited to 50Mbps.

Till now, I used with succes, a Linksys WRT54GL, withou any problem! But the WAN speed it’s hardware limitted at ~28Mbps.

I bought a new Linksys gateway-router RVS4000 to increase this WAN speed.

Everythins is fine, but the speed is very low with IPS (Intrusion Prevention System) – around 15Mbps. I tryed to disable it, but the DNS is not working enymore!

If I try to reach directly the IP, the speed is very good!

I tryed different firmwares (v1.1.14, 1.2.10, 1.2.11) with restore factory default before and after flashing, but without any success.

On static IP or DHCP on WAN, the speed is very good without IPS.

If I try to acces the web pages directry in web administration page at Diagnostics > Trace Route, the DNS seems to be OK.

But the problem is not present on all platforms...

I have a desktop and a laptop with Windows XP OS and IPS kill DNS.

On my LAN I have a satellite receiver with Linux OS and the DNS is fine with even without IPS.

Also my XBOX 360, reach everything in internet with IPS disabled, but my iPhone is not working.

For the moment, on computers and iPhone I put a manual DNS and everything is fine! For me, this one is ok: 209.244.0.4; 4.2.2.2 is not working sometimes.

nickovs123 · ‎08-31-2009

Well, after some more investigation I have made a little progress on this, although it is still not actually resolved.

Having attached packet capture tools on both sides of the router and watched the DNS packets going through it seems that there is a bug in the RVS4000 firmware. With the IPS enabled, DNS packets pass through the device with just the source IP address changed (as you would expect due to NAT being enabled) and the checksum values in the IP and UDP headers updated (because the source IP address changed). In this situation everything works fine but the throughput is throttled by the IPS software. If the IPS is disabled, the router not only changes the source IP address but also changes the source UDP port. Worse, it replaces the randomly chosen, non-privileged port number picked by the client with a sequentially chosen port number, starting at port zero. Thus with IPS disabled the outgoing DNS packets look exactly like the packets that would be used to bounce a DDoS attack off someone else's DNS server and my ISP is deciding that they look bogus and rejecting them.

It's also worth noting that replacing the randomly chosen, non-privileged port number with a sequentially chosen port number, starting at port zero, introduces a serious new security vulnerability. If an exploit is found for a bug in the DNS client software on PCs then the only thing stopping this being as dangerous as the recent bugs in the BIND DNS servers is that it's hard for attackers to know which port to attack. Unfortunately if the client machine is behind an RVS4000 then the attacker doesn't need to know which port is being used because he can just send exploit packets with the source address set the the victim's ISP's DNS server and low, sequentially numbered destination ports and the RVS will conveniently pass the exploit on to the correct port on the victim machine. Thus for this type of exploit, having an RVS with IPS disabled is actually less secure than not having the router there at all.

Re-mapping the source UDP port is unnecessary unless there is a port number collision, and the router manages just fine without re-mapping if IPS is enabled. If the port must be re-mapped it should be replaced with something that is randomly chosen and not a privileged port number, just like a good client would have picked. The current state on the RVS4000 1.2.11 firmware is broken and should be fixed.

David Carr · ‎09-01-2009

Thank you nick, for your input on the rvs4000. It is noted and they are working towards releasing a new firmware for fix this issue.

gorciscoliksys · ‎09-11-2009

With the new firmware v1.3.0.5 version, the DNS it's OK!

nickovs123 · ‎09-11-2009

That's great. When will it be available?

streaves · ‎09-11-2009

Firmware v 1.3.0.5 is now available for download here: http://www.cisco.com/en/US/products/ps9928/index.html.

Share your thoughts.

Thanks!

nickovs123 · ‎09-11-2009

Thanks for the pointer.

FWIW, I had looked there but neither the new firmware version number not the release date appear anywhere on that page. Perhaps it would be worth considering a web site policy about ensuing that links to firmware specify either version number, release date or both.

Anyway, I'll try the new firmware and see if it cures the problem for me.

nickovs123 · ‎09-11-2009

I can confirm that the V1.3.0.5 firmware cures this problem!

Thanks for the fix, Cisco.