Code Red Worm Smoked Our Shun-Test Router

[crossmanj]

How can we throttle shun attempts? That may be a better title for this message... I left last night with my sensors configured with the string match sigs for the ISAPI overflow bug that Code Red uses, plus a handful of other signatures flagged for shunning with a 2600 configured as my test router for shunning. There were four sensors speaking to it to mirror our production environment.

The probes were live, the router was a test box running the same code (IOS 12.0(7)T) as the production 7200's. I had planned to run a vulnerability scanner against the configuration to certify the design. (Last night's testing went well.)

Before I could do anyting this morning, we were hit with Code Red and some customer servers (not patched) were infected. Now at the end of the day I have 35000 detects, and the router has been reduced to ROMMON status twice now. (I have 13Mb+ Sniffer logs of the conversations between sensors and router from the first pre-failure run at about 20,000 detects into it, I haven't checked the latest)

It seems to be a load problem, and I would not worry so much if it were test data - after all how could something like this hit in the real world. SHEESH! The live data smoked my router. Now, how do I get my infrastructure team to trust this with the live 7200's?

Is there a way to throttle shuns similar to mail (x events over y seconds only), or route all the shuns through the Director so it could order and at least help the concurrent requests?

[stleary]

The current version of the sensor can not throttle consecutive shun attempts of multiple hosts.

Consecutive shunning of the same host is not a problem, since If the same host is shunned multiple times, an internal timer is updated but the router ACLs are not updated.

Shun handling is improved a bit in 3.0. Prior releases would update the router ACLs for each and every shun. 3.0 will consolidate new shuns into a single ACL update, if the shuns arrive while the router ACLs are being written for a previous shun. Some routers can take up to 5-10 seconds to commit ACL updates, so this may help.

But it sounds like you would like to see real throttling of shuns that can be tuned by the user. I will discuss this with the team, let me know if you have any preferences for throttle parameters.

[marcabal]

Did you say you have 4 sensors controlling the same router?

If so then your implementation needs to change, and this could definately help your problem.

Right now you have four sensors which are each trying as fast as possible to change the routers configuration, and lower end routers such as the 2600 have problems when multiple users are changing and saviong it's configuration (can result in rebooting into rommon mode because of corrupted configuration.)

Your request to have a single box do the shunning is the correct answer, and CSIDS has a feature just for this type of implementation that you should use.

The sensors have the ability to forward shun requests to a single shunning server sensor so that only a single sensor is controlling the router.

To do this you would specify one of the four sensors as your shunning server.

Configure the shunning server sensor to control the router.

The other 3 sensors should have the router information removed from their configurations.

The other 3 sensors should have the shunning server sensor configured as their shunning server.

The other 3 sensors are then considered shunning clients of the shunning server.

If you are using CSPM see the following CSPM User's Guide link to configure:

http://www.cisco.com/univercd/cc/td/doc/product/ismg/policy/ver23i/idsguide/ch03.htm

Look for the "Specifying Master Blocking Sensors" section.

NOTE: Instead of shunning, CSPM uses the word blocking.

If you are using the Unix Director the master shunning sensor is configured in the Shunning Tab of the Device Management configuration.

Caveats to keep mind during implementation:

1) Never configure a shunning server to be a shunning client of another shunning server unless you can guarantee that you have not created a shunning request loop. For example: If sensor1 is configured as a client for sensor2, but sensor2 has also been configured as a client for sensor1, then this creates a loop where the requests get passed back and forth from sensor1 and sensor2 continuously.

2) All of the sensors have to be configured from the same management machine (CSPM or Unix Director). Because the management machine will have to be updating all of the sensors.

3) When 3.0 comes out, you shouldn't have a 3.0 client sending requests to a 2.5 server.

4) The client sensor should have the shunning server's information in it's hosts, and routes file (and possibly organizations if in different origanizations). The shunning server should have the shunning client's information in it'd hosts, routes, (possibly organizations) and auths file (in the auths file the client has to be given at least EXEC authorization).

This should be done automatically by CSPM, but in the Unix Director you may need to update each configuration area through nrConfigure. In older versions of nrConfigure you had to add the information to each file yourself (through nrConfigure), but newer versions may update the files automatically.

5) Whether using CSPM or Unix Director you should push the new configuration to both the clients and server.

[crossmanj]

Thanks for the replies. I believe you may have answered my first question which was what files need to be updated for the sensors to see one another in the Shunning Tab. When I was first configuring the test environment, each sensor could only see themselves and the Director on the drop-down menu.

Your explaination lets me revive my shun testing, so that is good. :-) However, what about this?

I have all of my sensors' management interfaces on a private network which is unroutable and not currently connected to the production network. I did this for obvious security measures. The Unix Director is multihomed to a (reasonably) secured internal network and the private IDS network. So to activate shunning I must either add a router between this private network and the production, internal network -OR- if there was a way to get the Director to shun for me...? Any chance of being able to do it through the director?

[marcabal]

Current director versions will not support shunning from the director, and I don't believe that it is scheduled for a future version either. The only thing to do would be to call the TAC and submit it as an enhancement request.

To do shunning I would reccomend a small Pix 506 be used to route between the IDS and Internal networks, and use NAT to translate the IDS addresses. The shunning server should be mapped to a static NAT address, and this address would then be placed in the NAT Address field when adding Routers to be managed by the sensor.