I troubleshoot performance problems on our XEN Servers and I see some weird problems. Attached you can find a drawing with the network layout.
The Xenserver is connected with 4 x 10Gbit/s to 2 Nexus 5k switches with FEXes. The 6500's are a VSS cluster. I've created an ERSPAN session with all 4 XE interfaces as source towards the 6500 cluster, config is on the drawing.
Now I did expect to see all the traffic going through the 2 x 10G interfaces on the XENServer and 2 x 1G for management, but I see a lot of "Acked missing segment" messages in the trace (see attached screenshot, it's taken from Cascade Pilot). The bandwidth over time never exceeds 25Mbit/s, with microbursts up to 600Mbit/s. I don't see any drops or interface errors on the N5k or the 6500.
The linux server where tcpdump is running:
tcpdump -i eth4 -n -B 1048576 -w trace.pcap -C 100M
tcpdump: listening on eth4, link-type EN10MB (Ethernet), capture size 65535 bytes
12909810 packets captured
12910471 packets received by filter
561 packets dropped by kernel
I have no idea why I see these acked missing segment messages, where do they get lost?
A well-behaved TCP stack implementation should never acknowledge a segment that has not yet been received. The question, of course, is whether all packets captured by the ERSPAN session indeed made it to your machine performing the tcpdump and whether the machine was fast enough to store them. Is there any option of doing a local SPAN session and comparing the results?
I also wonder if other traffic analysis tools would report the same issue. Wireshark should be able to detect such occurences using the tcp.analysis.ack_lost_segment filter expression.
This document gives several answers on frequently asked questions for PFRv3 channel state behavior.
Q1: What are all the channel operational states from a BR (border role) perspective and what are the rules/conditions to be in each st...
The need was to reach an host inside a LAN through a VPN connection managed by the LAN gateway (Cisco 1921).
The LAN gateway performs NAT and there was a dedicate nat rule for the host i wanted to reach through VPN.
I couldn't connect to the hos...
We have 3 identical switches configured by someone else and would like to claim some of the Gigabit ports(G1/G2/G3/G4) for use on servers. When we try to change the wiring and configuration, we run in to connectivity issues. Attached is a des...