04-12-2011 01:36 AM - edited 03-10-2019 05:19 AM
Hi all – Have picked up something really strange on our IPS
IPS Info NME-IPS-K9 Version 7.0(4)E4 sig 552.0 (configured inline)
We have seen on our mail servers connecting that we get the following error
conversation with <remotemailserver> timed out while receiving the initial server greeting
This happens on a few remote mailservers.
The log on the IPS shows
Severity Date Time Device Sig. Name Sig. ID Attacker IP Victim IP
informational 04/10/2011 09:04:29 myipsname TCP SYN Host Sweep 3030/0 <mymailserverip> <remotemailserverip>
Note, no action taken… according to the logs.
Now I have “whitelist” the mailserver IP’s
Ie
<myipsname>(config)# service network-access
<myipsname>(config-net)# general
<myipsname>(config-net-gen)# never-block-host <mailserverip>
The added the remote server as a test, no luck.
However the moment I take the IPS out of inline mode and put it in promiscuous mode the mail servers work fine.
This only happens to some remote mailservers. This is what's so strange.
I am open for ideas / things to try.
Solved! Go to Solution.
04-13-2011 08:21 AM
Martin -
You can try turning on logging for all the normalizer engine signatures. This should let you see which signature is fireing in the alert logs. Once you know that you can tune th eoffending signture out.
- Bob
04-12-2011 09:43 AM
The normailzer engine has some signatures set to drop but not log.
Try disabling Signature 1330/12 (TCP drop - segment out of order)
It has caused problems like this in the past.
- Bob
04-12-2011 11:39 PM
Thanks for the tip Bob. I seem to picked up another issue - I had this a month ago.
In short the IPS Manager tells me that the "Application Failed" Now last time I did not spend alot of time on trying to figure out what was wrong.
i just reimaged the IPS and loaded my config back again.
As this just happened again, I think I need to try and figure out why this is happening. A reset of the IPS does not help.
My other IPS that I installed running the similar config has no issues. I am starting to wonder if there is not an underlying issue with this one.
04-13-2011 12:10 AM
Found something
Error Message Output from show statistics analysis-engine
Error: getAnalysisEngineStatistics : ct-sensorApp.424 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.
Error Message Output from show statistics anomaly-detection
Error: getAnomalyDetectionStatistics : ct-sensorApp.424 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.
Error Message Output from show statistics denied-attackers
Error: getDeniedAttackersStatistics : ct-sensorApp.424 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.
Possible Cause These error messages appear when you run the show tech support command and Analysis Engine is not running.
Recommended Action Verify Analysis Engine is running and monitor it to see if the issue is resolved.
To verify Analysis Engine is running and to monitor the issue, follow these steps:
Step 1 Log in to the sensor.
Step 2 Verify that Analysis Engine is not running:
sensor# show version
-----
MainApp N-2007_JUN_19_16_45 (Release) 2007-06-19T17:10:20-0500 Running
AnalysisEngine N-2007_JUN_19_16_45 (Release) 2007-06-19T17:10:20-0500 Not Running
CLI N-2007_JUN_19_16_45 (Release) 2007-06-19T17:10:20-0500
Check to see if AnalysisEngine reads Not Running.
Step 3 Enter show tech-support and save the output.
Step 4 Reboot the sensor.
Step 5 Enter show version after the sensor has stabilized to see if the issue is resolved.
Step 6 If Analysis Engine still reads Not Running, contact TAC with the original show tech support command output.
Ref http://www.cisco.com/en/US/docs/security/ips/6.0/installation/guide/hwTS.html
Will advice what I did once mine is sorted.
04-13-2011 12:47 AM
IPS is back up and running seems the reset with in the IPS sorted the ClientPipe.
Back to the original issue. I have disabled 1330/12, put the IPS inline mode and started seeing the mail issue to some servers again.
If we telnet from the mail server to a mail server that gives and issue u can see the port opening. However u can't type the "helo"
Put the IPS in promiscuous, then you can say "helo" to the mail server and it responds as per the norm.
04-13-2011 08:21 AM
Martin -
You can try turning on logging for all the normalizer engine signatures. This should let you see which signature is fireing in the alert logs. Once you know that you can tune th eoffending signture out.
- Bob
04-13-2011 01:14 PM
Hi Bob,
Is there a global setting for this, or will this be a case of doing them one by one?
04-18-2011 03:18 AM
Ok, found the guilty one
Disabled
1330/17 <-
1330-17: TCP segment out of state order. If a packet in a stream causes this signature to produce an alert, processing will cease for that stream. This signature will not produce an alert in promiscuous mode regardless of the signature status.
Where can I find more information on this?
P.S Bob, thanks for helping in tracking this one down!!!
04-19-2011 01:29 PM
Hello Martin,
Good job identifying 1330.17 as the culprit. The signature indicates that your sensor is seeing out-of-order TCP traffic. Disabling this signature temporarily to identify it as the root cause of network issues is an acceptable troubleshooting step. However, I want to caution against leaving it disabled beyond the scope of troubleshooting. 1330.17 alerts you to the fact that out-of-order traffic exists on your network and that the sensor cannot properly inspect that traffic. Multiple network paths, hop devices that mis-order traffic, or misconfigured host stacks are the cause of out-of-order traffic. You will want to fully investigate this issue so that you can default signature 1330.17 as soon as possible. If the out-of-order traffic is coming from your ISP, you will want to work with them to correct this issue.
In regard to the other issue you noted, Analysis Engine Not Running, you will want to open a TAC case so that we can isolate the root cause. We can then provide you with remediation steps or a bug ID which will be used to document the issue.
Thank you,
Blayne Dreier
Cisco TAC Escalation Team
**Please check out our Podcasts**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast
TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758
04-20-2011 02:05 AM
Hi Blayne,
Out-of-order tcp traffic, I think I might know what is causing that. Let me try and explain see if you agree.
Right now we run a multihome solution with two alternate ISP connections via BGP.
Each of these connections come in to a 3945 – Same routers where my IPS is located on.
Now the one link is a 30 meg and the other a 10 meg. Configured in failover mode (as prepend) with each of my address blocks. Now if I make use of an international looking glass, I can see traffic coming back towards me the right way. In this case my 10meg.
However, if I make user of the mail servers where we experienced the issue their ISP’s looking glass I see the traffic coming over my 30meg. I can only think it’s due to the fact that they are somehow directly connected to the ISP supplying my 30meg. They pushing the traffic directly to me, and not across via the internet to my other ISP, then back to me.
So I think what is happening is I am accepting the traffic via my 30 meg, and trying to communicate back via my 10 meg, and I think this is where the IPS picks up the out-of-order traffic. (Asymmetrical routing )
I have a failover test date setup in about 2 weeks’ time. I think part of the test might need to re-enable that 1330.17 on the IPS. And when I shut the link to one ISP to ensure failover of the BGP, (ie running on a single link) test the mail servers I know gives us issue to try and confirm this. Then do the same with the other link and retest
If this is the case, The IPS is doing what it’s supposed to be doing. And not an IPS issue but rather something that’s more BGP related.
What do you think?
Oh P.S if the IPS gives that Analysis Engine Not Running issue again I’ll log the call directly to TAC – Thanks.
04-26-2011 08:31 AM
Hello Martin,
This sounds possible, depending on your configuration. Do you have the "ids-service-module" command applied to both ISP link interfaces?
Thank you,
Blayne Dreier
Cisco TAC Escalation Team
**Please check out our Podcasts**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast
TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758
05-03-2011 12:57 AM
Hi Blayn,
Yes I do... We got failover planned for the weekend. I'll see if I can't test this and feedback
10-04-2012 02:28 AM
Hi Guys,
I am facing a strange problem with the IPS-4240 which is sitting in promiscous mode.
It runs for sometimes and the analysis engine freezes where it blocks the gui access to the device throwing the following error message.
01Oct2012 09:38:02.016 70.010 collaborationApp[456] IoPipe/E errSystemError connect timed out [ClientPipe::connect]
01Oct2012 09:38:02.016 0.000 collaborationApp[456] Syslog/F ct-sensorApp.446 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.
When I reboot the system,it starts to run again for sometimes and again the same thing happen.
I am trying to monitor the internet traffic in my network outside the firewall where the IPS is left in the wild configuration.
please advice me if you guys come across this issue.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: