Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Step-by-Step Configuration and Troubleshooting Best Practices for the NGFW, NGIPS and AMP Technologies A Visual Guide to the Cisco Firepower Threat Defense (FTD)
Cisco Employee

Resolving AMP for Endpoints TETRA Definition Issues

 

Hi Everyone,

 

Recently we were made aware of a TETRA AV definition update which caused the Windows AMP for

Endpoints service to crash. 

 

Note:  Customers who do NOT have TETRA enabled are not affected by this issue. 

 

While we have already removed the problematic definition set, which was available for ~30 minutes (see further notes below), affected systems will need to be fixed manually by uninstalling/re-installing the Connector (instructions below). Once the connector has been re-installed, a non-affected definition set will be downloaded and resolve the issue.

 

How to determine if you are impacted:

The issue causes the AMP for Endpoints service to crash or hang. The best way to determine if you have an affected system is to determine if any Connectors have been offline since the bad definition set was published.

 

To get the Last Seen Timestamp from the AMP Console, go to the Management tab and select Computers. From here you can download a CSV file using the "Export to CSV" option. The CSV will contain the Last Seen Timestamp. You can sort and filter on Connectors that have not been seen since 16:00 UTC February 06 2018 – these are likely Connectors that have been affected by this issue.

 

Resolution:

We urge all customers who are affected by this issue to open a TAC case immediately.

 

Resolving this issue does involve uninstalling and reinstalling the Connector.

 

Uninstall via Add/Remove Programs:

a) Uninstall the connector (choose "No" when asked if you plan to install the Connector again)
b) Re-install connector

 

Uninstall via Command Line:

<installer> /R /S /stopservicecoe 1 /remove 1

 

Uninstall via Command Line with Connector Protection Enabled:

<installer> /R /S /stopservicecoe 1 /remove 1 /uninstallpassword <INSERT YOUR PASSWORD>

 

Affected Software Versions:

All Windows Connector versions with TETRA enabled are affected on both 32bit and 64bit versions of Windows 7/8/10, Windows Server 2008R2 and Server 2012

 

Notes:

 

TETRA Definition Sets:

Faulty TETRA definition revision (16:20 UTC)

32bit = 101032, 64bit = 70876

 

Updated TETRA definition revision (16:50 UTC)

32bit = 101034, 64bit = 70878

 

A Root Cause Analysis (RCA) document will be prepared and shared with affected customers. 


 

 

44 REPLIES
Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

When was the update available? How can we tell through the console which systems may be exhibiting the behavior? We have around 4,500 connectors in our environment. 

Cisco Employee

Re: Resolving AMP for Endpoints TETRA Definition Issues

As stated above, the Replacement definition was posted within 30 minutes of the original problematic definition. This happened before 17:00 UTC on February 6. The versions are also referenced above.
Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

We also have 20k+ connectors with tetra offline scanning enabled. Would be nice to know which ones were affected.

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Affected clients are getting this error.

Affected clients are getting this error.  If you have connector protection enabled like we do it looks like resolution is going to be a nightmare.  

error.png

Cisco Employee

Re: Resolving AMP for Endpoints TETRA Definition Issues

I had this problem also, and my PC was really slow. I removed AMP (with no plan to reinstalled) and then re-installed as suggested and both problems went away and APM is now working.
Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

While this fix is for the endpoints, can something be done on the backend to prevent this from spreading any further?

 

With this deployed across all endpoints in our network, working on every machine individually is going to be extremely tedious to say the least......

Cisco Employee

Re: Resolving AMP for Endpoints TETRA Definition Issues

Please see below
Todd

Re: Resolving AMP for Endpoints TETRA Definition Issues

It seems that only 32bits OS would requires the fix to be executed. We are using a powershell script to detect if the event is present in the windows logs if so clear the content of the update directory. The script is to be deployed with SCCM and the compliance check is that the service is present and not running.  AND FINALLY we will disable TETRA and deploy SCEP: 3rd strike, we are not beta tester.

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Pascal,

 

All our machines that have so far been affected are all Windows 7 64 bit. For the most part we have no 32 bit machines deployed.

 

The batch file works but, only when the machine is in booted in safe mode.

 

This is going to be  a real pain in the ***

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

What is the name of the event that you are searching for in the Windows logs? 

Re: Resolving AMP for Endpoints TETRA Definition Issues

Event ID 7031 that matches cisco AMP

 

$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7003;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if ($ampLog.Count -gt 0) {
    $svc = Get-Service | Where-Object {$_.name -match "amp"}
    if ($svc.status -eq "running") {
        Stop-Service $svc -Force
    }
    # clean the **bleep**...
}

Re: Resolving AMP for Endpoints TETRA Definition Issues

Correction: 7031

$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

Re: Resolving AMP for Endpoints TETRA Definition Issues

FINAL VERSION FOR SCCM THAT APPLY TO 32-bit only

___________________________

TO APPLY ON ALL : just remove

-and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" )

___________________________

# compliance check:
$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if (($ampLog.Count -gt 0) -and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" ) ) {
    return $false
} else {
    return $true
}

# remediation:
$svc = Get-Service | Where-Object {$_.name -match "ciscoamp"}
if ($svc.status -eq "running") {
    Stop-Service $svc -Force
}
Remove-Item "$env:ProgramFiles\Cisco\AMP\tetra\Plugins\*"
Remove-Item "$env:ProgramFiles\Cisco\AMP\update\Plugins\*"
Start-Service $svc

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Does this sccm script work for password protected instances?


pascal.bourbonnais@mcgill.ca wrote:

FINAL VERSION FOR SCCM THAT APPLY TO 32-bit only

___________________________

TO APPLY ON ALL : just remove

-and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" )

___________________________

# compliance check:
$ampLog = @(Get-WinEvent -FilterHashtable @{logname='system';id=7031;startTime=$((Get-Date).AddHours(-1)) } | Where-Object { $_.message -match "Cisco AMP"})

if (($ampLog.Count -gt 0) -and ( (Get-WmiObject Win32_OperatingSystem).OSArchitecture -eq "32-bit" ) ) {
    return $false
} else {
    return $true
}

# remediation:
$svc = Get-Service | Where-Object {$_.name -match "ciscoamp"}
if ($svc.status -eq "running") {
    Stop-Service $svc -Force
}
Remove-Item "$env:ProgramFiles\Cisco\AMP\tetra\Plugins\*"
Remove-Item "$env:ProgramFiles\Cisco\AMP\update\Plugins\*"
Start-Service $svc


 

Re: Resolving AMP for Endpoints TETRA Definition Issues

Your could change the resolution script like this:

 

    # remediation:
   
    $sfc = @(Get-ChildItem -Path "$env:ProgramFiles\Cisco\AMP" -Recurse -Include "sfc.exe")
    $sfc | foreach { .$($_.fullname) -k <YOUR PASSWORD> }
   
    $svc = Get-Service | Where-Object {$_.name -match "ciscoamp"}
    if ($svc.status -eq "running") {
        Stop-Service $svc -Force
    }

    Remove-Item "$env:ProgramFiles\Cisco\AMP\tetra\Plugins\*" -Force
    Remove-Item "$env:ProgramFiles\Cisco\AMP\update\Plugins\*" -Force
    Start-Service $svc

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Same Problem

6261711391345530340.jpg

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Trying to modify the provided script in the OP using the documentation here ( https://www.cisco.com/c/en/us/support/docs/security/sourcefire-fireamp-endpoints/118588-technote-fireamp-00.html ) since we have connector protection enabled. Currently not working (access denied) but still trying.

Cisco Employee

Re: Resolving AMP for Endpoints TETRA Definition Issues

  1. Try this addition to the script to get it to disable the connector protection.
  2. :: START cmd.exe /c "cd C:\Program Files\Cisco\AMP\<version you're running > & sfc.exe -k <connector protection password>" 

    START cmd.exe /c "cd C:\Program Files\Cisco\AMP\5.1.13 & sfc.exe -k Cisco123456"
    timeout 10 > nul
    del "C:\Program Files\Cisco\AMP\tetra\Plugins\*" /q
    del "C:\Program Files\Cisco\AMP\update\Plugins\*" /q
    timeout 5 > nul
    wmic service where "name like 'CiscoAMP%%'" call startservice > nul
Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

That one solved the problem - thanks.

 

Uninstall.

Choose no - (all files including quarantined are deleted).

Reboot.

Download and install connector.

Highlighted
Cisco Employee

Re: Resolving AMP for Endpoints TETRA Definition Issues

Fixed my problem and the Window's security definition error, and the very slow response my PC was having.  All is well now thankfully.

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

It is not possible to delete definition files after stopping the Service.

Looks like the Kerneldriver is preventing changes in AMP-Directory

 

 

thanks Cisco for this great Product

 

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

I haven't encountered duplicates in AMP by using the 'say no'-solution.

 

Advise: Try a few machines and check AMP before continuing.

 

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

you are right

changed my reply

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Are there any "Mass deployment" solution to this mess? this is really bad for a corporate environment to Uninstall the connector, Force a reboot, Install the connector and force another reboot..

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Need a way to manage definitions through the console. Also need status information in the console on whether a connector is connected or not. Not really an enterprise solution with so little control over the endpoints. 

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

The delete_defs.bat script does not work for mass deployment purposes as it relies on manual elevation of the command prompt to delete the problematic files.

Alternatively, I'm unaware of a CLI parameter that can be set with the execution of the uninstall.exe executable in the local Cisco AMP version folder that would allow for the scripting of the AMP's uninstallation that includes removal of the definitions as well.

Cisco AMP customers need a better solution than has been provided to accommodate the large scope of machines that have been impacted. A manual machine-by-machine resolution is not optimal for companies managing hundreds or thousands of computers.

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

Can we get more details on this issue?  Are all versions of connector afffected?  What is the TETRA definition version number that is the issue?  Does the connector stop working completely until the fix is applied or does a reboot temporarily get it going again with the possibility of another crash? 
That information would greatly help us determine our exposure and assist in our remediation planning.

Community Member

Re: Resolving AMP for Endpoints TETRA Definition Issues

I would really like to get some explanation of the 0016 timestamp. Are you saying anything that has checked in from 0017 on is okay? Is it anything in a certain window?

Cisco Employee

Re: Resolving AMP for Endpoints TETRA Definition Issues

Rather than manually touching each endpoint, please remember you can push uninstalls and reinstalls via SCCM or any other deployment  tool using Command Line Switches. These are listed in the latest Deployment Strategy Guide on page 26.

 https://docs.amp.cisco.com/en/A4E/AMP%20for%20Endpoints%20Deployment%20Strategy.pdf

 

/remove 1       Uninstalls the Connector and removes all associated file

So you can use these commands:

<installer> /r /S /stopservicecoe 1 /remove 1

 <installer> /r /S /stopservicecoe 1 /remove 1 /uninstallpassword <INSERT YOUR PASSWORD>

 to remove the application. The <installer> should be the original installation file for the AMP for Endpoints Connector.

 

Todd

 

6943
Views
180
Helpful
44
Replies
CreatePlease to create content