Internet router CPU question

wilson_1234_2 · ‎06-18-2008

We recently had a situation where a vulnerability scanner went haywire and was trying to port scan across an entire class A subnet.

The scanning started at 10.0.0.0, and was scanning across a wide range of ports.

We have no routes in that subnet in our network, so the traffic was getting passed through the firewall to the Internet edge router.

We are getting a default route from Verizon, which is in this router's route table, which then points to the PE router that peers with my Internet edge router on a T1.

The CPU on this router would bounce up to 100% and stay there causing the serial interface to shut down.

Here is what I don't understand:

Why was this pegging the CPU and causing the interface to shut down, when I have seen data downloads, and we traffic also utilize 100% of the T1 without doing this?

The router has a default route to Verison's router, so this router was not having drop the packets, it was just passing them through as the data would be.

Unless, (as I am writing this I am thinking about it) it has to do with the Firewall is PATing the single IP address and probably just sourcing a few ports compared to the scanning of numerous addresses and dozens of ports continuously.

Did I just answer my own questions, the router could not process all of that?

ashok_boin · ‎06-18-2008

There could be several reasons for high cpu like packets not using CEF so that interrupting the CPU in very fast rate etc.

Hope the following doc helps...

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml#show_process_cpu

With best regards...
Ashok

hennigan · ‎06-18-2008

It could be a number of things. "show process cpu" would be a good starting point.

The Verizon router likely has RFC1918 addresses filtered and is returning ICMP unreachables to you. This may involve the CPU relaying the unreachable to the firewall. By default ICMP unreachables are rate-limited to one every 500ms. If they aren't rate-limited or if the scanner is generating a huge number of source IPs/ports, then the router may be overwhelmed.

"sh ip icmp rate-limit" may give some help here.

If the vulnerability scanner is looking to trigger a DoS attack to which your IOS is vulnerable, the scanner succeeded. ;-)

wilson_1234_2 · ‎06-19-2008

I really didn't see anything from "sh process cpu".

Look at this capture.

The CPU is up, but does not show what caused it, or I don't see it.

When I shut the serial interface down, the cpu would go down.

ashok_boin · ‎06-19-2008

From the given attachment, it's the interrupts which is causing high cpu, not processes which is why you are not able to see any process consuming more CPU cycles in the output.

CPU utilization for five seconds: 96%/89%; 89% is due to interrupts.

Here is the link which helps to troubleshoot CPU utilization due to Interrupts.

http://www.cisco.com/en/US/customer/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

I strongly feel "Router overloaded with the traffic" cause listed in the above doc for your problem. Just enable the interface again & observe throttles in "sh int". And also capture "sh int switching". There is accounting procedure as well if you want to find a source which is causing this issue.

Regards...

-Ashok.

With best regards...
Ashok

Giuseppe Larosa · ‎06-20-2008

Hello,

not all routers support this command but when it is supported is very useful:

show proc cpu sorted

it will show in the first rows the processes that are using the CPU the most.

Hope to help

Giuseppe

a.alekseev · ‎06-19-2008

I think you use NAT on your firewall.

The port scan across an entire class A subnet creates too many nat translations on it.

Try to identify the source of this scan and block it.

It's a best practice to block RFC1918 addresses on your Edge router or firewall.