it would be http or https type of traffic

check this :-

http://www.cisco.com/en/US/docs/security/ips/5.0/configuration/guide/idm/dmsigwiz.html

thanks, good link. I know about the custom signature configuration and will advise client to provide information from their web logs to asssit in an attempt to create a custom signature. It does not appear there are any pre-configured sigs. for "web crawler type traffic". I looked at what is available in version 6.0(3)E1 S315.

I have heard of people embedding hidden URL's in their HTML and then firing when something requests the resource. You could also create something based on requests per second.

hmm, sounds good. Would creating a sig based upon # of requests per second be in the custom signature documentation?