IPS strange behaviour ?

Answered Question
Apr 12th, 2011

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin-top:0cm; mso-para-margin-right:0cm; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0cm; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-fareast-language:EN-US;}

Hi all – Have picked up something really strange on our IPS

IPS Info NME-IPS-K9 Version 7.0(4)E4 sig 552.0 (configured inline)

We have seen on our mail servers connecting that we get the following error

conversation with <remotemailserver> timed out while receiving the initial server greeting

                This happens on a few remote mailservers.

The log on the IPS shows

Severity               Date      Time      Device  Sig.         Name    Sig. ID    Attacker IP          Victim IP

informational     04/10/2011         09:04:29               myipsname TCP SYN Host Sweep             3030/0  <mymailserverip>                <remotemailserverip>

Note, no action taken… according to the logs.

Now I have “whitelist” the mailserver IP’s

Ie

<myipsname>(config)# service network-access

<myipsname>(config-net)# general

<myipsname>(config-net-gen)# never-block-host <mailserverip>

The added the remote server as a test, no luck.

However the moment I take the IPS out of inline mode and put it in promiscuous mode the mail servers work fine.

This only happens to some remote mailservers. This is what's so strange.

I am open for ideas / things to try.

I have this problem too.
0 votes
Correct Answer by rhermes about 3 years 2 days ago

Martin -

You can try turning on logging for all the normalizer engine signatures. This should let you see which signature is fireing in the alert logs. Once you know that you can tune th eoffending signture out.

- Bob

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (1 ratings)
rhermes Tue, 04/12/2011 - 09:43

The normailzer engine has some signatures set to drop but not log.

Try disabling Signature 1330/12 (TCP drop - segment out of order)

It has caused problems like this in the past.

- Bob

martin.bosch Tue, 04/12/2011 - 23:39

Thanks for the tip Bob. I seem to picked up another issue - I had this a month ago.

In short the IPS Manager tells me that the "Application Failed" Now last time I did not spend alot of time on trying to figure out what was wrong.

i just reimaged the IPS and loaded my config back again.

As this just happened again, I think I need to try and figure out why this is happening. A reset of the IPS does not help.

My other IPS that I installed running the similar config has no issues. I am starting to wonder if there is not an underlying issue with this one.

martin.bosch Wed, 04/13/2011 - 00:10

Found something

Analysis Engine Not Responding

Error Message    Output from show statistics analysis-engine
Error: getAnalysisEngineStatistics : ct-sensorApp.424 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.
Error Message    Output from show statistics anomaly-detection
Error: getAnomalyDetectionStatistics : ct-sensorApp.424 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.
Error Message    Output from show statistics denied-attackers
Error: getDeniedAttackersStatistics : ct-sensorApp.424 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.

Possible Cause   These error messages appear when you run the show tech support command and  Analysis Engine is not running.

Recommended Action   Verify Analysis Engine is running and monitor it to see if the issue is resolved.

To verify Analysis Engine is running and to monitor the issue, follow these steps:

Step 1 Log in to the sensor.

Step 2 Verify that Analysis Engine is not running:

sensor# show version

-----
MainApp N-2007_JUN_19_16_45 (Release) 2007-06-19T17:10:20-0500 Running 
AnalysisEngine N-2007_JUN_19_16_45 (Release) 2007-06-19T17:10:20-0500 Not Running 
CLI N-2007_JUN_19_16_45 (Release) 2007-06-19T17:10:20-0500

Check to see if AnalysisEngine reads Not Running.

Step 3 Enter show tech-support and save the output.

Step 4 Reboot the sensor.

Step 5 Enter show version after the sensor has stabilized to see if the issue is resolved.

Step 6 If Analysis Engine still reads Not Running, contact TAC with the original show tech support command output.

Ref http://www.cisco.com/en/US/docs/security/ips/6.0/installation/guide/hwTS.html

Will advice what I did once mine is sorted.

martin.bosch Wed, 04/13/2011 - 00:47
/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin-top:0cm; mso-para-margin-right:0cm; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0cm; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-fareast-language:EN-US;}

IPS is back up and running seems the reset with in the IPS sorted the ClientPipe.

Back to the original issue. I have disabled 1330/12, put the IPS inline mode and started seeing the mail issue to some servers again.

If we telnet from the mail server to a mail server that gives and issue u can see the port opening. However u can't type the "helo"

Put the IPS in promiscuous, then you can say "helo" to the mail server and it responds as per the norm.

Correct Answer
rhermes Wed, 04/13/2011 - 08:21

Martin -

You can try turning on logging for all the normalizer engine signatures. This should let you see which signature is fireing in the alert logs. Once you know that you can tune th eoffending signture out.

- Bob

martin.bosch Wed, 04/13/2011 - 13:14

Hi Bob,

Is there a global setting for this, or will this be a case of doing them one by one?

martin.bosch Mon, 04/18/2011 - 03:18

Ok, found the guilty one

Disabled
1330/17 <-


1330-17: TCP segment out of state order. If a packet in a stream causes this signature to produce an alert, processing will cease for that stream. This signature will not produce an alert in promiscuous mode regardless of the signature status.

Where can I find more information on this?

P.S Bob, thanks for helping in tracking this one down!!!

Christopher Dreier Tue, 04/19/2011 - 13:29

Hello Martin,

Good job identifying 1330.17 as the culprit. The signature indicates that your sensor is seeing out-of-order TCP traffic. Disabling this signature temporarily to identify it as the root cause of network issues is an acceptable troubleshooting step. However, I want to caution against leaving it disabled beyond the scope of troubleshooting. 1330.17 alerts you to the fact that out-of-order traffic exists on your network and that the sensor cannot properly inspect that traffic. Multiple network paths, hop devices that mis-order traffic, or misconfigured host stacks are the cause of out-of-order traffic. You will want to fully investigate this issue so that you can default signature 1330.17 as soon as possible. If the out-of-order traffic is coming from your ISP, you will want to work with them to correct this issue.

In regard to the other issue you noted, Analysis Engine Not Running, you will want to open a TAC case so that we can isolate the root cause. We can then provide you with remediation steps or a bug ID which will be used to document the issue.

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

martin.bosch Wed, 04/20/2011 - 02:05
/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin-top:0cm; mso-para-margin-right:0cm; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0cm; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-fareast-language:EN-US;}

Hi  Blayne,


Out-of-order tcp traffic, I think I might know what is causing that. Let me try and explain see if you agree.

Right now we run a multihome solution with two alternate ISP connections via BGP.

Each of these connections come in to a 3945 – Same routers where my IPS is located on.

Now the one link is a 30 meg and the other a 10 meg. Configured in failover mode (as prepend) with each of my address blocks. Now if I make use of an international looking glass, I can see traffic coming back towards me the right way. In this case my 10meg.

However, if I make user of the mail servers where we experienced the issue their ISP’s looking glass I see the traffic coming over my 30meg. I can only think it’s due to the fact that they are somehow directly connected to the ISP supplying my 30meg. They pushing the traffic directly to me, and not across via the internet to my other ISP, then back to me.

So I think what is happening is I am accepting the traffic via my 30 meg, and trying to communicate back via my 10 meg, and I think this is where the IPS picks up the out-of-order traffic. (Asymmetrical  routing )

I have a failover test date setup in about 2 weeks’ time. I think part of the test might need to re-enable that 1330.17 on the IPS. And when I shut the link to one ISP to ensure failover of the BGP, (ie running on a single link) test the mail servers I know gives us issue to try and confirm this. Then do the same with the other link and retest

If this is the case, The IPS is doing what it’s supposed to be doing. And not an IPS issue but rather something that’s more BGP related.

What do you think?

Oh P.S if the IPS gives that Analysis Engine Not Running issue again I’ll log the call directly to TAC – Thanks.

martin.bosch Tue, 05/03/2011 - 00:57

Hi Blayn,

Yes I do... We got failover planned for the weekend. I'll see if I can't test this and feedback

raslan.a.s Thu, 10/04/2012 - 02:28

Hi Guys,

I am facing a strange problem with the IPS-4240 which is sitting in promiscous mode.

It runs for sometimes and the analysis engine freezes where it blocks the gui access to the device throwing the following error message.

01Oct2012 09:38:02.016 70.010 collaborationApp[456] IoPipe/E errSystemError connect timed out [ClientPipe::connect]

01Oct2012 09:38:02.016 0.000 collaborationApp[456] Syslog/F ct-sensorApp.446 not responding, please check system processes - The connect to the specified Io::ClientPipe failed.

When I reboot the system,it starts to run again for sometimes and again the same thing happen.

I am trying to monitor the internet traffic in my network outside the firewall where the IPS is left in the wild configuration.

please advice me if you guys come across this issue.

Actions

Login or Register to take actions

This Discussion

Posted April 12, 2011 at 1:36 AM
Stats:
Replies:12 Avg. Rating:5
Views:1980 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard

Rank Username Points
1 816
2 668
3 603
4 526
5 367
Rank Username Points
10
5
5
5