***UPDATE...I was able to convert it to .zip and attach.  Thanks.

!

!

!

Hi, I tried to attach the .pcap file however it won't allow me.  Below is a list of allowed file types.

You can change the extension of the original attachment to .pcapng and open with Wireshark.

Unless there is another way for me to attach the capture file?

txt rtf doc docx xls xlsx ppt pps ppsx pptx pdf odt odp ods jpg jpeg png gif arf zip

So all clients have their IPs translated to 172.18.123.4 ?

What is the firewall ie. make, model and if ASA version number of code ?

Jon

It's actually a McAfee Sidewinder (being decommissioned and replaced with ASA 5515x in the coming weeks).

Yes all clients resolve the website address via global DNS to the WAN IP of this firewall which then sends it to 10.10.7.0/24 network.  This network is in another office however firewall has connectivity to it via our DMVPN network.

It could literally be a thousand and one things to be honest.

Packets are clearly not be being received in time but if this is across a WAN with DMVPN and a firewall between the server and clients it's very difficult to say from a packet capture on the server.

Is it just one office that cannot connect. If there are other offices that can then concentrate on that specific office and it's connectivity.

If all offices are experiencing the same issue then concentrate on the firewall and the DMVPN connection in that site.

Do you have other servers behind the firewall that are accessible ?

If they are and you are accessing them via https that suggests the firewall is okay.

If they are but you don't use https then that would suggest looking at the firewall.

With these things it's really just a question of narrowing it down to find out exactly what does and doesn't work and this usually points to where to start looking.

Apologies if this is what you have already done and I am telling you something you already know.

Jon

It's actually from anywhere.  The DNS resolves the website address to a global address.  So regardless of the source (inside or out), you hit the firewall and get routed to 10.10.7.0 network.  The firewall's LAN interface shares the same VLAN as the DMVPN head-end router's LAN interface.  From the DMVPN head-end router, it goes over the DMVPN cloud (i.e. back over the internet) to our office in Florida where this site is being hosted. 

The capture I grabbed was by SPAN port between the two LAN interfaces showing transactions between the firewall's LAN interface and the server's IP address on the 10.10.7.0 subnet.

Site uses HTTPS and we have other servers in the same subnet (10.10.7.0) that are accessible in the same manner.  I did SPAN the ports for another webserver and did see a lot of TCP OOO and re-transmissions however not as bad as this one. 

I do have a theory, please feel free to correct me.  Request comes in on the WAN interface, gets NATed by the firewall and sent to the DMVPN router, router encrypts the packet and places it on the wire, once the remote DMVPN peer receives the packet, it decrypts it and then sends it out it's DMZ interface connected to another application firewall. This firewall checks the packet and then sends it to the web server hosting the content.  The process is reversed for reply traffic. On top of all this, the content is served over HTTPS therefore more encryption/decryption. This seems like too much handling of the packet to me?  When the source computer sends a request, it simply times out or spends too much time within our own network causing the source to resend the request?

This seems like too much handling of the packet to me?  When the source computer sends a request, it simply times out or spends too much time within our own network causing the source to resend the request?

I tend to agree. Specifically the encryption decryption on the already encrypted stream in addition to having to go through two firewalls.

Is your DMVPN network hub and spoke or is it also spoke to spoke ?

If the actual server is hosted in different spoke and is still protected via a firewall could you not simply go direct from the spoke to the other spoke and not via the hub ?

Obviously this would also mean changing the DNS entry to point to the remote site.

I appreciate you have other servers setup but I'm not seeing what benefit you get going via the hub other than a central point of control which may be the reason.

Out of interest does each spoke have local internet or is that centrally controlled as well ?

I only ask because with it already being encrypted if they did could you not simply use the internet connection to get from one spoke to the other ie. no DMPVN involvement.

If the spokes had fixed IPs you could lock down which IPs were allowed to access it.

I appreciate neither may be possible alternatives, just suggestions really but I would have thought anything you can do to reduce the amount of time spent getting from one site to the other should help assuming that is the main cause.

Jon

Hi Jon, yes our network is spoke-to-spoke.  

If the actual server is hosted in different spoke and is still protected via a firewall could you not simply go direct from the spoke to the other spoke and not via the hub ?

This is exactly what my initial thought was and I did bring this up. Basically what they (our server team) are trying to accomplish is to have a DR site setup at another office and mirror the data from the original office to the DR site.  They want to have a single globally routable address coming into our Hub site and then from there they get to decide where they want to send it to.  According to them, should a disaster occur at the original site, it's "faster" to make the change internally and point the traffic to the DR site rather than updating the global DNS with the global IP address of the DR site.

Each spoke site split tunnels to the internet directly from their router.  

What I was planning on trying was to have them change the global IP of the server to the actual spoke site. This will bypass all the DMVPN back and forth traffic from the hub site. If this fixes everything, then we will know for sure the issue is either with the McAfee Firewall OR the actual handling of the packet inside the DMVPN cloud.