Web crawler/scraper signatures

Unanswered Question
Feb 8th, 2008

Running version 6.0(3)E1 S297. Is there a specific signature that will recognize web crawler/scraper traffic? If not one already defined what in particular would I need to create a customer signature.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
5creedus Tue, 02/12/2008 - 15:32

thanks, good link. I know about the custom signature configuration and will advise client to provide information from their web logs to asssit in an attempt to create a custom signature. It does not appear there are any pre-configured sigs. for "web crawler type traffic". I looked at what is available in version 6.0(3)E1 S315.

mhellman Thu, 02/14/2008 - 07:23

I have heard of people embedding hidden URL's in their HTML and then firing when something requests the resource. You could also create something based on requests per second.

5creedus Thu, 02/14/2008 - 14:06

hmm, sounds good. Would creating a sig based upon # of requests per second be in the custom signature documentation?


This Discussion