Using SigWiz to detect crawlers and robots.

[ismael.lezcano]

Hi. I'm trying to use STATE.HTTP to detect the User-Agent header of all my inbound Web traffic and parse the value of it to detect crawlers, robots, etc. Any suggestions?

[anthall]

I assume you know what will appear in the 'User-Agent' Field. If you don't that would be your first step by using tcpdump over a sustained period of time. With that information you could craft a simple HeaderRegex that would match on what you are looking for. The following example would fire upon detecting a Mozilla browser.....

Tune Signature Parameters : CSIDS Signature Wizard

___________________________________________________________________________

Current Signature: Engine STATE.HTTP SIGID 20000

SigName: Mozilla User Agent (example)

___________________________________________________________________________

0 - Edit ALL Parameters

1 - AlarmInterval =

2 - AlarmThrottle = FireOnce

3 - ArgNameRegex =

4 - ArgValueRegex =

5 - ChokeThreshold =

6 - DeObfuscate = True

7 - Direction = ToService

8 - FlipAddr =

9 - HeaderRegex = User[-]Agent[:][ ]Mozilla

10 - LimitSummary =

11 - MaxArgFieldLength =

12 - MaxHeaderFieldLength =

13 - MaxInspectLength =

14 - MaxRequestFieldLength =

15 - MaxUriFieldLength =

16 - MinHits = 1

17 - RequestRegex =

18 - ResetAfterIdle = 15

19 - ServicePorts = 80,3128,8000,8010,8080,8888,24326

20 - SigComment =

21 - SigName = Mozilla User Agent (example)

22 - SigStringInfo = Mozilla User Agent detected

23 - ThrottleInterval = 15

24 - UriRegex =

25 - WantFrag =

d - Delete a value

u - UNDO and continue

x - SAVE and continue

___________________________________________________________________________

Selection>

Hope that helps!

[ismael.lezcano]

It almost did. My previous attempts did not take in all the rules for Regular Expressions. Following your example I have configured the sensor, however packetd now refuses to start. I get "W WARNING suppressed while parsing global parameter" messages in the error log.

[klwiley]

Could you please cut and paste the SigWizMenu screen that describes your signature for us. This will help us to see what might be wrong. If you can't do that (it might prove to be fairly difficult) could you please forward your SigUser.conf file to anthall@cisco.com.

KLW

[anthall]

I think I found you problem. My example had a space in the brackets before Mozilla:

User[-]Agent[:][ ]Mozilla

Your SigUser.conf line did not:

User[-]Agent[:][]lwp

If you use SigWizMenu and edit that sig it should work.