- Bronze, 100 points or more
Hi there. I have an interested problem I"m hoping someone could help out with. I have a a server dual homed to two 4507R's, which are connected to a failover pair of Pix 535's. This particular server receives a TCP data feed from a data source that sends about 3Mbps continuously throughout the day.
We've been noticing that every 2 hrs (+- 10 mins), the feed gets disconnected. The interesting thing is that all we have to do to recover the feed is stop/start the server application that is receiving the data.
After observing the condition for a couple of days, I noticed that the problem happens every two hours, plus or minus 10 mins. We can practically set our watches to it. When the traffic stops flowing, I notice that one of the interfaces on the firewall increments a receive discard error (as observed through Solarwinds Orion). The interface in question is a trunk link that carries traffic to the firewall for 3 different vlans. The link is fiber.
I was able to capture traffic right at the time the feed stopped receiving data, and I observed the following:
1. There long periods where the source was sending data, and seq #'s were increasing, but my server wasn't sending anything back.
2. After a second or two of the behavior seen in #1, I started receiving a few TCP Previous Segment lost messages, and a large numbers of TCP Dup Ack's.
The packet sniffer is looking at the interface on the 4507R that the firewall is connected to. The traffic flows like this:
Server ---> 4507R ---> PIX 535 (inside) ----> PIX535 (DMZ) ----> 4507R ----> provider network device.
I'm really at a loss as to what to look at next. The server get's rebooted each night, so if it was related to the something going awry on the server I wouldn't expect to see it each day. The firewall is running 6.3.5, but we do not see any problems on any other feeds or connections that pass through the firewall. The predictability of the error is also puzzling. Can anyone offer some insight into where I could look next?
Thanks for your help!