Tcp Flow Reject & Error Messages

[john.pepper]

We have a problem whereby our 2 * CSS11503 content switches are continually logging the following error messages:

FLOWMGR-6: FM_UtilGenericTcpFlowReject: Handling Generic Flow REJECT

FLOWMGR-6: FM_UtilGenericTcpFlowError: Handling Generic Flow ERROR

I have searched this forum and found only 'one' post on this subject (posted by Gilles) which says the problem could be either on the Client side or the Server side with the possible cause being as follows:

Handling Generic Flow REJECT:

1.) This error is on the Client side and something caused the CSS to reject the connection. Examples might be; the content request spanned more than the configured maximum (default is 6). The delayed

ACK to the client (at 200ms) could not be sent or was responded to with a TCP SYN/FIN or RST or the client closed down unexpectedly

Handling Generic Flow ERROR

2.) This error is on the server side. The CSS is sending the spanned content request to the server and either did not get an acknowledgement or got an unexpected response or more likely in this case the client side was torn down.

Gilles suggests a sniffer capture - I am due to go to site tomorrow (Wednesday) to troubleshoot this problem..

Has anyone got any information or resolution tips on this error and what I'm supposed to be looking for in the sniffer trace..?

The errors are constant and occur approx every second.

Help very much appreciated....

Cheers...John

[stevehall]

John,

Well, there is a lot to look for, given the above possible problems.

Look to see if the content request spanned more than the configured maximum (default is 6).

Look to see the ACK to the client. Was it a SYN/FIN or something unusual? Was it more than 200 ms?

Did the client end the connection unexpectedly (before all data has been sent)?

Did the CSS not get an ACK (or some other unexpected response) from the server?

Basically, look at a failed connection and look for distinctions from a working connection...

-Steve

[john.pepper]

Steve,

Good to hear from you again..

Thanks for the helpful pointers. I'll do as you suggest tomorrow (Wednesday) and see what I can find.

I'll no doubt have some more questions once I've done the trace.

All the best for now..

Cheers....John

[Gilles Dufour]

John,

one remark.

This message is not considered an error.

The level of the message is 6 as you can see.

It is therefore considered informational.

There are valid reasons to see this message.

If there is no problem reported by the people going through the CSS, I would simply reduce the logging level to 4-error.

Going beyond level 4 should only be done for troubleshooting purposes.

Lab tests proved that when logging too many messages, the performance of the CSS can be severely affected.

Regards,

Gilles.

[john.pepper]

Gilles, thanks for this information.

For the record, I carried out a trace on the packets going through the CSS.. From the client side I noticed many unusual tcp conversations. Here is sample of a dodgy conversation (ip addresses changed to Host & CSS):

Host to CSS TCP 52336 > 8000 [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460 WS=1 TSV=380721923 TSER=0

CSS to Host TCP 8000 > 52336 [SYN, ACK] Seq=0 Ack=1 Win=8760 Len=0 MSS=1460

Host to CSS TCP [TCP Out-Of-Order] 52297 > 8000 [FIN, ACK] Seq=0 Ack=0 Win=65535 Len=0

CSS to Host TCP [TCP ZeroWindow] [TCP Dup ACK 287#1] 8000 > 52297 [ACK] Seq=0 Ack=1 Win=0 Len=0

I also noticed what appear to be many bad tcp connections. These are some of the tcp messages I captured during a trace of approx 2 minutes:

[TCP ZeroWindow] 2533 > 8000 [RST] Seq=3259 Ack=173349072 Win=0 Len=0

[TCP Dup ACK 8683#1] [TCP Previous segment lost] 8000 > 3827 [RST, ACK] Seq=2 Ack=0 Win=65535 Len=0

[TCP ACKed lost segment] 2583 > 8000 [ACK] Seq=0 Ack=1 Win=64512 Len=0

[TCP Out-Of-Order] 52316 > 8000 [FIN, ACK] Seq=0 Ack=0 Win=65535 Len=0

[TCP Retransmission] 8000 > 38490 [ACK] Seq=0 Ack=0 Win=65535 Len=1460

[TCP Retransmission] Continuation

Although the capture was a short trace (approx 2 mins), there were approx 65,000 conversations. Out of these 65,000 conversations over 4,500 contained the above tcp messages.

There are no complaints from users however I would like to understand why this is happening. We will take your advice and reduce the logging to the Default (4) but if you or Steve could provide any feedback on the above I would be grateful.

many thanks....John

[Gilles Dufour]

could you uploade the trace to ftp://incoming.cisco.com and give us the file name.

We can review it and see if there is nothing abnormal.

When command to capture is 'show dos'.

It will tell you if the CSS detects dos attack.

Regards,

Gilles.

[john.pepper]

Gilles,

Sorry for the delay.

I have uploaded the packet trace for you to review as requested. The file name is :

EDS CSS Problem Client side.xls

I will do a 'show dos' and let you know what I output I get.

Thanks...John

[Gilles Dufour]

sorry - but may I ask you to upload a .cap file.

ethereal or tcpdump compatible.

Thanks,

Gilles.

[john.pepper]

Gilles,

tried to ftp the Ethereal file over RAS but it's 181 MB in size. I have put some of it up on your 'incoming' web site - not sure if you can read it..?

File name is - CSS Client Side

If you can't read it I'll have to do it when I'm next in the office.

Thanks...John

[Gilles Dufour]

got the file.

[the first 4 seconds].

for each RST being sent the window size is zero.

So your sniffer will complain but there is no reason.

This is correct.

Personally, I don't see anything wrong.

Do you have packets in this trace that you are concerned with.

Gilles.

[john.pepper]

Thanks for the update.

I'll try and ftp more of the trace up to you so you can have a bit more data to review.

I did see a lot of the following messages:

TCP ACKed lost segment

TCP DUP ACK

TCP Out-of-Order

TCP Previous Segment Lost

Cheers...John

[john.pepper]

Gilles,

Just to wrap this one up.

I've checked for Denial of Service (show dos) and noticed a number of syn attacks occuring every few seconds.

I will look into this but for the moment I am satisfied that the CSS' are running ok. We have had no user complaints. Therefore, I would like to close this message thread.

Many thanks for your help..

John