I have a search engine spider that runs on my server that is protected by a PIX 501 with a basic configuration. The spider needs HTTP outbound access and back inbound again to spider a web site that is on the same server. I am a newbie, but I believe that somehow the firewall is blocking the inbound (re-entry so to speak) of the spider and therefore the spider is giving me errors that it cannot find the web site. Any ideas on how I could verify this and/or make a setting to allow this in a specific secure manner. Thanks.
There are a couple things that can prevent this.
1. PIX routing. Unless running 7.x and even then only with configuration changes to the default, the PIX doesn't allow routing back out an interface it received the inbound packet on. So if the web client(WebKeepAlive) on your web server is essentially making an http request to itself, it'll resolve DNS(assuming your using public) and receive it's Public IP. It will then route it's packet to it's default gateway (unless you have it specified in your web server route table) and that will probably be the PIX. The PIX will receive this and will eventually drop it due to security not allowing routing back out its source interface.
The easiest way to get around this for your scenario is to update the HOSTS file on the server with the Web Site FQDN using the Private IP and not the Public. DNS will never get invoked because the HOSTS file will resolve first. You will never hit the PIX and will be able to Spider your website for your reports or whatever.
I'm not going to discuss the other things that could block it because I'm pretty sure you ain't running 7.x on a 501 because it isn't supported. If it was 7.x you could loop the connection and then the thread could go on and on with Static commands and access-lists. Though you could technically use the DNS fixup on the static when it makes the DNS request but I would have to look that up. You could also configure routing on your web server for the Public IP but the HOSTs file is your best bet.
Please rate any helpful posts