Cisco Support Community
Showing results for 
Search instead for 
Did you mean: 

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

Using SigWiz to detect crawlers and robots.

Hi. I'm trying to use STATE.HTTP to detect the User-Agent header of all my inbound Web traffic and parse the value of it to detect crawlers, robots, etc. Any suggestions?

Cisco Employee

Re: Using SigWiz to detect crawlers and robots.

I assume you know what will appear in the 'User-Agent' Field. If you don't that would be your first step by using tcpdump over a sustained period of time. With that information you could craft a simple HeaderRegex that would match on what you are looking for. The following example would fire upon detecting a Mozilla browser.....

Tune Signature Parameters : CSIDS Signature Wizard


Current Signature: Engine STATE.HTTP SIGID 20000

SigName: Mozilla User Agent (example)


0 - Edit ALL Parameters

1 - AlarmInterval =

2 - AlarmThrottle = FireOnce

3 - ArgNameRegex =

4 - ArgValueRegex =

5 - ChokeThreshold =

6 - DeObfuscate = True

7 - Direction = ToService

8 - FlipAddr =

9 - HeaderRegex = User[-]Agent[:][ ]Mozilla

10 - LimitSummary =

11 - MaxArgFieldLength =

12 - MaxHeaderFieldLength =

13 - MaxInspectLength =

14 - MaxRequestFieldLength =

15 - MaxUriFieldLength =

16 - MinHits = 1

17 - RequestRegex =

18 - ResetAfterIdle = 15

19 - ServicePorts = 80,3128,8000,8010,8080,8888,24326

20 - SigComment =

21 - SigName = Mozilla User Agent (example)

22 - SigStringInfo = Mozilla User Agent detected

23 - ThrottleInterval = 15

24 - UriRegex =

25 - WantFrag =

d - Delete a value

u - UNDO and continue

x - SAVE and continue



Hope that helps!

New Member

Re: Using SigWiz to detect crawlers and robots.

It almost did. My previous attempts did not take in all the rules for Regular Expressions. Following your example I have configured the sensor, however packetd now refuses to start. I get "W WARNING suppressed while parsing global parameter" messages in the error log.

Cisco Employee

Re: Using SigWiz to detect crawlers and robots.

Could you please cut and paste the SigWizMenu screen that describes your signature for us. This will help us to see what might be wrong. If you can't do that (it might prove to be fairly difficult) could you please forward your SigUser.conf file to


Cisco Employee

Re: Using SigWiz to detect crawlers and robots.

I think I found you problem. My example had a space in the brackets before Mozilla:

User[-]Agent[:][ ]Mozilla

Your SigUser.conf line did not:


If you use SigWizMenu and edit that sig it should work.