ASK THE EXPERT - TROUBLESHOOTING IPS

Unanswered Question
Sep 11th, 2009

Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to get more information on troubleshooting IPS. with Cisco expert Srinavas Mallu. Srinivas is a senior customer support engineer in High Touch Technical Support (HTTS) within the Technical Assistance Center (TAC). He has a double CCIE in Routing & Switching and Security (CCIE# 8914). Mallu has been in TAC for the past eight years supporting security related products such as PIX, ASA, FWSM, Security on IOS, IPSec, ACS and IDS. He also trains people on his team on security technologies.


Remember to use the rating system to let Srinavas know if you have received an adequate response.


Srinivas might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through September 25, 2009. Visit this forum often to view responses to your questions and the questions of other community members.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
ganesh.myakala Fri, 09/11/2009 - 22:25

I want to know the bugs realated to IOS c1841-advipservicesk9-mz.124-12c.bin

As i am facinf some problem within VPN tunnels

smallu Mon, 09/14/2009 - 13:14

Ganesh,


There are several bugs, and to narrow down the search, I need a clear cut criteria. What exactly is the problem you are running into. What is the peer routers IOS version?

Is this a VPN issue, or an IPS issue?


Best regards,

Srinivas.



serkancanbaz Sun, 09/13/2009 - 04:07

Hello, we have ASA 5540 with aip ssm module and its version is 6.1(1)E1. We want to upgrade it to 6.1(1)E3 and

get all of the new signatures which are belong to E3. In order to upgrade, there are three titles in your site which are called

"system upgrades", "system software" and "signature updates".

In system upgrades page, there are two title, "IPS-K9-6.1-1-E3.pkg" and "IPS-engine-E3-req-6.1-1.pkg"

In System software page, there is one title, "IPS-SSM_40-K9-sys-1.1-a-6.1-1-E3.img".

In signature updates page, there are many signatures up to S433.

What is the correct order to upgrade 6.1.1(E1) to 6.1.1(E3) and get the last signature updates?

Now our device's signature version is S329 in E1. How can we go to S433 in E3?


Best Regards

smallu Mon, 09/14/2009 - 15:30

Hi There,


To upgrade your IPS version on the ASA 5540, you need to go to the System upgrades page, and download the "IPS-K9-6.1-1-E3.pkg" file and upgrade the version. The system software page is for the SSM image itself, that goes into the SSM module. The signature update page, has signature updates for various IOS levels, if you need to be on the latest signature update, you can download the latest update. By downloading it, it patches the IOS level on the ASA with the latest signature.


Hope this helps!


Thanks,

Srinivas.

smallu Mon, 09/14/2009 - 15:33

Once you download and upgrade to "IPS-K9-6.1-1-E3.pkg", you can then go to the signature update page, and then go to S433 in E3 and download the "IPS-sig-S433-req-E3.pkg" and apply the patch.


Srinivas.

daniel.litwin Wed, 09/16/2009 - 11:10

Hello.


When tuning the IPS on our 5520, what is a good recommendation for active rules? Is there baseline setup of rules that should be enabled, ie SIG2158/0 set to alert. Do people find some rules better set to "Deny Attacker Inline" or just log the information?


smallu Wed, 09/16/2009 - 16:49

Daniel,


It really depends on the risk rating for the signature. Here is a good document that talks about the recommended action based on the risk rating and how the risk rating is computed.


Hope this helps!


Thanks,

Srinivas

Dileep Sivadas ... Thu, 09/17/2009 - 21:54

I have following questions


1.i have recently enabled global correlation feature on AIP-SSM , i can able see the information on IME regarding the number of packet blocked by this feature, how can i get more details like attacker ip , victim ip , nature of attack etc..


2. Its abt cisco secuirty manager ,we planning to install security manager 3.3.0 with standard edition license


2.1 Does this verison support IDSM2 and FWSM


2.2 Does it support the ips 7.x Engine


3. When configuring IDSM2 on 6513 with following config sup720,IOS 12.2(18)sfx16 on promiscuous mode how many monitor session (SPAN)can we configure on this

smallu Fri, 09/18/2009 - 15:46

Hi There,


I'll try my best to answer all your questions.


1. Have you configured Network participation, in 'full' mode, which is a requirement to collect the data you are looking for. Here is more information on this;

http://www.cisco.com/en/US/docs/security/ips/7.0/configuration/guide/cli/cli_collaboration.html#wp1053292


2.1 Yes. This version supports both IDSM2 and FWSM

2.2 Yes again. CSM 3.3 supports the latest IPS 7.x


3. You can have only two span sessions no matter what the IOS is or what sup you use.


Hope this helps!


Thanks,

Srinivas.

Dileep Sivadas ... Mon, 09/21/2009 - 20:33

Now IPS is configured in partial mode, i will enable full mode and check this out.

-------------

Another issue regarding IDSM


Getting lots of syslog message displaying following error


"syslogMessage:

description: Note: /etc/modules.conf is more recent than /lib/modules/2.4.30-IDS-smp-bigphys/modules.dep"


Appname : modprobe


i think this error is due to timestamp issue of linux kernel module configuration file.


how can i correct this.


thanks

Dileep






smallu Tue, 09/22/2009 - 14:25

Dileep,


This is a warning that the modules.conf file, which contains a list of

kernel modules and their command-line arguments, has been modified and

therefore depmod may need to be run if the changes to modules.conf

affect any dependencies between the modules or the location of the

module's binary. Modules.conf can be modified at boot time by the boot

scripts, which updates its timestamp but it will not alter the

dependencies or location of the modules.


To fix this, run depmod in the service account (su to root first).


Hope this helps!


Thanks,

Srinivas.

Dileep Sivadas ... Thu, 09/24/2009 - 01:32

Regarding Global-correlation


As you told , i have enabled full network participation mode.


but still not getting any details about denied packets.


I can see how many packets are denied due to GC on IME . But not getting any incident detail either on IME or MARS.


please clarify ,what will happen if following two conditions occur.


1. When packets are dropped due to reputation filtering.


2. when packets are dropped due to increase in RR.



Thanks


Dileep


smallu Thu, 09/24/2009 - 15:42

Dileep,


You can place the sensor in a global correlation audit mode which will result in events being generated for these instances where the deny actions are currently being performed. To enable this audit mode, just enter the following commands:

config t

service global-correlation

test-global-correlation on


As another option, Global Correlation works by adding risk to the event based on the attacker's reputation. Purely as a debugging step, you can


enable logging to display the packets for the HIGHRISK events. In general, having logging enabled impacts performance so you wouldn't want to keep logging enabled by default.


The sh stat ana command will display the global correlation stats on the actions being performed.


Please let me know if this helps in identifying the reputation filtered

events.


Thanks,

Srinivas.

marcabal Thu, 09/24/2009 - 22:19

For packets dropped due to reputation filtering try executing "show statistics analysis-engine".

At the bottom of the statistics should be a section on Malicious Site Deny Hit Counts. I think this section will tell you either the IPs or at least Networks being denied by Reputation Filtering and packet counts denied for each.


NOTE: The Reputation Filter denies do not cause alerts to be created. So the only way to get the IPs is to use the command above.



As for when packets are dropped due to an increase in RR by reputation.

There are 2 ways this can happen.

Either the reputation brings the RR up to 90 or higher (would have been below 90 without reputation), and because it is 90 or higher it gets denied by the default event action override for deny-packet-inline.


OR reputation raises the RR (maybe not as high as 90), but high enough that internal to the reputation code it matches a reputation override that adds deny-packet-inline.

The reputation override works similar to the configurable event action overrides.

Except that instead of a default 90-100 for the deny-packet-inline override it has a lower range.

The range is determined by the global-correlation-inspection-influence.

Permissive mode will likely prevent reputation itself from adding a deny (the default override might, but the reputation code itself is less likely to).

Standard mode which is the default and where quite a bit of alert from bad reputation addresses wind up with deny action added.

Aggressive mode which lowers the range even more and adds deny actions to a larger range of alert risk ratings.


In order to determine if reputation itself denied a packet look in IME for the following:

Filter for alerts that have negative Reputation.

Look for alert with either a drop or deny action listed.

And look at their Risk Rating.

If the Risk Rating is 90 or higher then the default event action override (assuming you still have it enabled) is likely the cause.

If the Risk Rating is less than 90 (assuming you haven't change the default event action override) then it is likely denied by reputation itself.


In either case you can select the alert itself, and click Show All Details.

At the bottom of the alert it will tell you the reputation information, and if reputation added a deny action.



Eduardo Aliaga Fri, 09/18/2009 - 09:10

Hello Srinavas. We are using 4 IPS 4270 in our network for over a year now. We have realized that the operating system is very buggy and several times we have found out the IPS went into bypass mode because the analysis engine stopped working and other times the IPS freezed and don't let us even use the console.


Because of that we've opened several cases in Cisco TAC, they suggest an upgrade, we perform the upgrade, but a couple of weeks later comes another issue, Cisco TAC suggest another upgrade... and so on, we have performed three or four upgrades throughout this year and now we're using 6.1(3)E3.


However currently we have a new issue. I opened Cisco case 612459443 because the AnalysisEngine stopped (yet another time) and this is the error message:


getAnalysisEngineStatistics : ct-sensorApp.460 not responding, please check system

processes - The connect to the specified Io::ClientPipe failed


The engineer assigned to the case told me this is likely to be a new issue and he's awaiting the response of the BU to give me a formal answer.


What can we do? There are no further upgrades in train 6.1 and we don't think an upgrade to 6.2 o 7.0 is a solution because those are really new operating systems and it's likely they're even more buggy.


We have really lots of Cisco equipment, and very different technologies and all of them are stable. All but the IPS.


Also, I would like to know what the number "460" means in the error "getAnalysisEngineStatistics : ct-sensorApp.460 not responding" cause we have seen several similar errors but with different numbers, for example earlier this year we got the message "getAnalysisEngineStatistics : ct-sensorApp.397 not responding". So I guess the number "460" and "397" means something relevant and we currently don't know what those numbers are up to.

smallu Fri, 09/18/2009 - 16:05

Hi There,


I see your frustration, but please be patient. The fact that you have a SR open means you'll get a resolution as a workaround or a bug fix. I just peaked into the SR, and it seems like the engineer got an update from the development team. Just ask him for an update, as you are working with him.


The development team only has access to tools which help them decipher these log codes. We normally send the error messages to them. It normally refers to a piece in the code which it is executing at that time.


Clearly communicate to the engineer who are working with, what your constraints are, and he can give you options which will work for you under your present circumstances.


Thanks,

Srinivas.

campbech1 Wed, 09/23/2009 - 09:27

Eduardoaliaga, we had the exact same problems. Since moving our IPS modules to version 7 they have been rock solid. You might want to consider upgrading.

clausonna Fri, 09/25/2009 - 09:19

I'll 2nd that - I have 15 AIP-SSM sensors and on 6.x was getting random lockups and application fails. Since going to 7.x things have been stable and haven't seen any hangs.


I'm pretty sure 7.0 is just 6.x with lots of bugfixes and the Global Corelation features.

Eduardo Aliaga Sat, 09/19/2009 - 10:25

I noticed there are three different types of logs: the file /usr/cids/idsRoot/log/main.log, the directory

/var/log/messages and the event-store. What are the meaning of those different logs? Are there any more logs?


If I were to use Cisco MARS, what logs will MARS receive from IPS? My guess is it will only receive logs from event store (via SDEE over SSL, since IPS doesn't support syslog) is that right?


My interest in this is that I found the following interesting logs in the file "main.log"


25Aug2009 14:33:29.523 1.841 interface[429] Cid/W errWarning Inline data bypass

has started.

25Aug2009 14:35:39.532 18000.001 interface[429] Cid/W errWarning Inline data

bypass has stopped.


I was wondering if MARS could receive those logs so I could create a rule to alert us whenever bypass starts. (IPS analysis engine has failed so many times without us noticing it for several days, so we DO need those alerts).

smallu Tue, 09/22/2009 - 14:21

Hi There,


The IPS applications use LogApp to log messages. LogApp sends log messages at any of five levels of severity: debug, timing, warning, error, and fatal. LogApp writes the log messages to /usr/cids/idsRoot/log/main.log, which is a circular text file.


The main.log is included in the show tech-support command output. If the message is logged at warning level or above (error or fatal), LogApp converts the message to an evError event (with the corresponding error severity) and inserts it in Event Store.


Event Store keeps track all the policy violations. SensorApp performs packet capture and analysis. Policy violations are detected through signatures in SensorApp and the information about the violations is forwarded to the Event Store in the form of an alert.


When you are using MARS, it should report the log file, which is /usr/cids/idsRoot/log/main.log. MARS can be configured to act like a syslog server. So, you should be able to receive all these logs.


If you need further help with the configuration, I would recommend opening a TAC case and get live help with the MARS configuration.


Thanks,

Srinivas.



smallu Tue, 09/22/2009 - 16:01

Eduardo,


Correction to my previous reply..


MARS actually only pulls alerts from the event store. MARS is Not able to receive syslogs from the sensor (the sensor will not send syslogs off the box).


And even if it did, the main.log is not syslogs and is just an internal debug log.


You have to go into the box periodically and manually pull these debugs.


The inline data bypass messages are actually written in at least 2 places.

They are written in main.log and they are also written as Error or Status events in the eventstore.You should be able to see these events using the CLI “show events”, or in an IDM event screen. Unfortunately none of our event viewers (neither MARS or IME) are built to monitor for Error or Status events.


The inline data bypass state is also included in Health Dashboard information in IME. I would recommend running IME and periodically check the sensor health in IME to see how the sensor is doing.


Thanks,

Srinivas.





Eduardo Aliaga Wed, 09/23/2009 - 13:24

Thanks for your answer. The "interesting" logs are "warning" level, as you could see


25Aug2009 14:33:29.523 1.841 interface[429] Cid/W errWarning Inline data bypass has started.

25Aug2009 14:35:39.532 18000.001 interface[429] Cid/W errWarning Inline data bypass has stopped.


You told us that logs from "warning" level and above should be sent to the "event store", so then MARS could use "SDDE over SSL" protocol to pull the logs from IPS, and MARS could alert us if data bypass starts.


Please confirm if that's right.

smallu Thu, 09/24/2009 - 15:27

Thats correct eduardo.


As long as it is in the event store, you should be able to pull the logs in the MARS.


Thanks,

Srinivas.

Eduardo Aliaga Sat, 09/19/2009 - 10:57

Our IPS are protecting public DNS servers. Every now and then we notice an extremely high number of DNS queries to invalid or non-existent domains. So to avoid Denial of Service to our DNS servers we created atomic signatures using regex to identify the particular URL of the invalid or non-existen domain, so those DNS queries won't arrive to the DNS server. I attached the signature configuration. I was wondering if there was a better signature engine, or another preferable method in order to block those invalid domains.



Attachment: 
smallu Tue, 09/22/2009 - 14:27

Eduardo,


For some reason, I am not able to open that signature.txt file. Can you paste the contents of the text file, if its not a big file.


Thanks,

Srinivas.

Eduardo Aliaga Wed, 09/23/2009 - 13:26

signatures 60013 0

alert-severity high

sig-fidelity-rating 100

promisc-delta 0

sig-description

sig-name Attack darkbaron.be

sig-string-info darkbaron.be

no sig-comment

release S176

sig-creation-date 20050620

exit

engine atomic-ip

event-action produce-alert|deny-packet-inline

specify-l4-protocol yes

l4-protocol udp

specify-udp-valid-length no

specify-udp-length-mismatch no


specify-dst-port yes

dst-port 53-53

exit

specify-src-port no

specify-udp-length no

exit

specify-payload-inspection yes

specify-min-match-length no

regex-string [Dd][Aa][Rr][Kk][Bb][Aa][Rr][Oo][Nn].[Bb][Ee]

specify-exact-match-offset no

specify-min-match-offset no

specify-max-match-offset no

exit

exit

exit

specify-ip-payload-length no

specify-ip-header-length no

specify-ip-tos no

specify-ip-ttl no

specify-ip-version no

specify-ip-id no

specify-ip-total-length no

specify-ip-option-inspection no

specify-ip-addr-options no


exit

event-counter

event-count-key AaBb

specify-alert-interval no

exit

alert-frequency

summary-mode fire-once

summary-key AaBb

specify-global-summary-threshold no

exit

exit

status

enabled true

exit

vulnerable-os mac-os|unix|aix|bsd|hp-ux|irix|linux|solaris|windows|windows-nt-2k-xp

exit


smallu Thu, 09/24/2009 - 15:51

Eduardo,


The signature file looks ok. It can be fine tuned, provided you have more details on the specifics of the attack. But as it is, it looks ok.


Thanks,

Srinivas.

Eduardo Aliaga Thu, 09/24/2009 - 20:45

Yes this signature works. But it's using an atomic engine. Most times, using "atomic engines" are not good for overall IPS performance. So do you think we can use another engine that could block URL but with better performance? maybe "string engine" or "dns engine"? Regards

wsulym Fri, 09/25/2009 - 06:37

Using atomic-ip is fine, its a very powerful engine, but you do have to be careful. It's a per packet inspection engine, and where you can get into trouble is when you use it specifying high volume common ports, or very broad port ranges (for instance inspecting all 65k ports for inspection)


But for what you are trying to do, which looks like trigger on the DNS name query, it's the correct choice.


Now, your signature works, but its not exactly correct. Your regex:

[Dd][Aa][Rr][Kk][Bb][Aa][Rr][Oo][Nn].[Bb][Ee]

uses the "." (dot). And that dot is being interpreted by the sensor as a wildcard and not as the dot in between pieces of the name (possibly that was your intention, but I'm not sure). So the sensor interprets your regex as:

"darkbaron", followed by any single character that is not \x0a, followed by "be", case insensitive.


Your regex works because in a name query, what you would actually see there is x02, which satisfies the wildcard dot. But it could be tighter, and thus consume just a little bit less resource.


This question gets asked a lot, and I've posted this same reply a few times on the forum before, I'm going to paste it here as well.


/begin/

A DNS query for something.


sig-name DNS query

> engine atomic-ip

> event-action produce-verbose-alert

> specify-l4-protocol yes

> l4-protocol udp

> specify-dst-port yes

> dst-port 53

> specify-payload-inspection yes

> regex-string (see below on what should be here)



For the dns regex, you need to be aware that the query will take the form of:

length-byte -- characters -- length-byte -- characters


So something like:

my.domain.com

2 characters, 6 charcters, then 3 characters.


Gets strung together as such:

\x02[Mm][Yy]\x06[Dd][Oo][Mm][Aa][Ii][Nn]\x03[Cc][Oo][Mm]


That is the regex to catch my.domain.com regardless of case in a dns query (UDP).

(note that the dots in the name, do not appear in the regex string)


/end/

The AD is detecting our internal DNS servers doing standard queries but this has started happening when we moved to IPS ver 7. What would be a good way to prevent these false positives from occuring? We are using the asa aip-ssm.


event_id=1253662775291897275

severity=high

device_name=########

app_name=sensorApp

receive_time=09/25/2009 02:19:57

event_time=09/25/2009 02:19:57

sensor_local_time=09/25/2009 02:19:57

sig_id=13004

subsig_id=0

sig_name=AD - External UDP Scanner

sig_details=Single Scanner

sig_version=S262

attacker_ip=#.#.#.#

attacker_port=

attacker_locality=OUT

victim_ip=0.0.0.0

victim_port=53

victim_os=

victim_locality=Unknown

summary_count=0

initial_alert_id=

summary_type=

is_final_alert=

interface=ge0_1

vlan=0

virtual_sensor=vs0

context=

actions=denyPacketRequestedNotPerformed

alert_details=. adExtraData: numDestIps=200; currentThreshold=200; destPort=53 ;

risk_rating_num=100(TVR=medium)

threat_rating=100

reputation=0

global_correlation_audit_mode=false

global_correlation_deny_attacker=false

global_correlation_other_overrides=false

global_correlation_risk_delta=0

global_correlation_modified_risk_rating=false

global_correlation_deny_packet=false

protocol=udp


smallu Fri, 09/25/2009 - 11:21

Hi There,


I am not sure what code changes have gone into IPS 7.x thats causing this. There is no easy answer to this. This is a good question for the development folks.


I would recommend you open a TAC case and have them engage the development folks to get an answer for this question. I am sure there is a way to prevent these false positives from occuring.


Hope this helps!


Thanks,

Srinivas.

Is there a way to autobackup the IPS? I noticed the copy command from the command line. If I do the copy configuration command does that cover all my changes or should I also copy the as-knowledge-base, anomaly-detection, etc?


copy ?

/erase Erase the destination file before copying.

Location of source file to be copied.

ad-knowledge-base Copy an anomaly detection knowledge base file.

anomaly-detection Anomaly Detection configuration.

backup-config Copy from backup config.

current-config Copy from current config.

event-action-rules Event Action Rules configuration.

iplog IP log.

license-key Copy from license key.

packet-file Captured packet-file.

signature-definition Signagure Definition configuration.

Actions

This Discussion