Why Customer get too many "Connection reset by peer"

[tevfik]

Hi,

I have CSS 11503 V7.50 without SSL Module.

I have 4 real servers and LB method is Round Robin. Customer tests the methods by using two way.

First A requestor try to connect service via VIP address.Second requestor try to connect to real servers by using directly real server IP address (Bypass Load balacer)

Customer cheks the real server logs and sees that in first experiment (via VIP) there too many "connection reset by peer" messages. For second experiment there are few same messages.

I wonder why real servers get this messages. Plus, why CSS reset flows. I know flow timeout does that. But if there are too many idle flows, it says something wrong?

[Gilles Dufour]

you need to capture the traffic with a sniffer on the server side and client side.

Then compare the 2 methods.

Also, check the log before and after a test, just to make sure that the counter you are checking is really increasing during the test.

Gilles.

[tevfik]

So, is there any command or output that shows idle flow count or smoething like this?

[Gilles Dufour]

There is a command, but I was more thinking about the server side counter or log info.

If you really want to see if the CSS sent a RESET or not, you can use the following engineering command

CSS11503-2(debug)# symbol-table load SPRITZ

Loading string table... 660,706

Loading symbols........ 24,937

Load complete.......... 1,259,194

CSS11503-2(debug)# shell 1 1 FMShowReset

FM RESET COUNTERS:

Last FM Reset Case was 7

-----------------------------

0 Case 1: Server Flow not in WCC

0 Case 2: Client Flow not in WCC

0 Case 3: FM_ContentHandleUnexpectedTermination

0 Case 4: Server not responding on backend, reset client

0 Case 5: Server not responding on backend, reset server

0 Case 6: WCC returns empty vector, reset client

18 Case 7: Reset to server on DOS attack

2 Case 8: FM_HandleEarlySpoofTermination

0 Case 9: Reset client if join fails in spoof flow

0 Case 10: TCP SYN Exceeded

0 Case 11: Reset server on remap backend connection

0 Case 12: WCC did not return WCC_TCP_SPOOF for FAUX SYN

0 Case 13: FM_HandleFinalClose

0 Case 14: FM_CreateRedirectServerReset

0 Case 15: Flow reset, route change

0 Case 16: Peer reset, route change

0 Case 17: Peer Flow rejected, either WCC rejected or no egress port

0 Case 18: Reset flow, bridge forwarding change

0 Case 19: Reset flow, egress port went down

0 Case 20: Client side connection reset sent

0 Case 21: Server side connection reset sent

0 Case 22: Reset client due to server reset

0 Case 23: Received ACK to SYN, reset existing connection on server

0 Case 24: Server side reset, server port down

0 Case 25: Client side reset, client port down

0 Case 26: Server side reset, client port down

0 Case 27: Client side reset, server port down

0 Case 28: MIDNAT reject, reset server

0 Case 29: MIDNAT reject, reset client

0 Case 30: FM_HandleTcpResetVipUnavailable

0 Case 31: MID SPOOF reject, reset server

0 Case 32: MID SPOOF reject, reset client

Gilles.

[tevfik]

Thanks Giles

I can look at the server side(Customer side) logs.

They says that connection reset by peer caused to "Internal server Error".

So, I have to learn that this connection reset by peer messages or, RST from load balancer is normal or anormal?

I mean, what is wrong that server side get too many "connection rest by peer" messages.

Additionnaly A little question:

Is there any zero command to reset output of comamnd you given? And is ther any command that show how many reset occurs by Rule?

Besr Regards..

[Gilles Dufour]

the command is "shell 1 1 FMZeroReset".

Once again, you NEED a sniffer trace.

Don't try solving this by just looking at counters or using debug command.

Loadbalancing world requires knowledge of exactly what is going on at network level.

What flags are set or not set and stuff like that.

With the sniffer trace we can understand exactly what is going on and what the server is complaining about.

Then we can act to solve it.

Gilles.

[tevfik]

Hi,

I got sniffer trace.

But i am little confused.

Altough flow-timeout-multiplier is 4, CSS 11503 does not end flows after 2 min.

I expect that if a flow is idle more than 64 sec. CSS send RST both side to close connection. Right? If not, how works flow-timeout-multiplier?

[Gilles Dufour]

the flow is marked reusable after the flow-timeout. It does not mean it will be reset or removed. The flow is simply move at the end of a long list of deleted flows [but still usable] and when the css needs to setup new flow it takes the first elements of the list.

So, it may take a while before a delete flow gets reused.

Once reused, you do not see a RESET until the client or server sends a new packet belonging to this old flow.

Gilles.

[tevfik]

Gilles,

Thank you for this explanation. But I have a last question to undesrtand flow timeout.

If a flow moves to deleted flows and it does not use again, how many time later or which conditions occurs before to delete this flow?

I mean flow deleting is depended on time or another condition?

[Gilles Dufour]

the flow is moved to the spoof list which is a FIFO. So the flow is deleted when getting out of the FIFO.

The time the flow spends in the FIFO is dependent on the number of connection per sec.

Gilles.

[tevfik]

Tnak you Gilles for your assistances...