Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

DRS on Publisher

I am having an issue attempting to backup my Publisher and Subscriber. When I got to do a backup I get the following error:

Status: Local Agent is not responding. This may be due to Master or Local Agent being down.

I downloaded the logs and found this series of entries.

2013-12-03 15:16:48,584 DEBUG [main] - drfNetServerClient: setupSSLSettings: /usr/local/platform/.security/ipsec/trust-certs/.ipsec-trust.keystore and /usr/local/platform/.security/ipsec/keys/.ipsec-trust_trust.passphrase files exists

2013-12-03 15:16:48,584 DEBUG [main] - drfNetServerClient: setupSSLSettings: set up truststore(password) properties

2013-12-03 15:16:48,584 DEBUG [main] - drfNetServerClient: setupSSLSettings: Create SSLContext and get SocketFactory

2013-12-03 15:16:48,584 DEBUG [main] - drfNetServerClient: setupSSLSettings: Number of entries in truststore : 0

2013-12-03 15:16:48,585 DEBUG [main] - drfNetServerClient: Reconnect: Creating socket from Host: OLSDCM1 port: 4040

2013-12-03 15:16:48,585 DEBUG [main] - drfNetServerClient:Reconnect, Unable to connect to host : [OLSDCM1], message: Connection refused, cause: null

2013-12-03 15:16:48,585 DEBUG [main] - drfNetServerClient.closeSocket, closing connection to: [OLSDCM1]

I have verified the services are running

I have rebooted both boxes

I had to reset the password on the box for the disaster recover system and at one point reset the share password. 

I have tried following the IPSEC reset trick with no change

I am running version 6.1.5.12900-7

At a loss.  Any help would be greatly appricated

Everyone's tags (1)
18 REPLIES

DRS on Publisher

Have you regenerating any certs recently? Or restored a DRS on "new" hardware/VM? When I have seen this issue I have had to restart ipsec (which you have done) or I have had to regenerate the ipsec cert and then restart the service (which has worked for me in all instances).

HTH.

-Bill (http://ucguerrilla.com)

HTH -Bill (b) http://ucguerrilla.com (t) @ucguerrilla

Please remember to rate helpful responses and identify

New Member

DRS on Publisher

Hi Dennis,

The Error is looks like IPSec Certificate related one, might be existing certificate got corrupted or something problem with the certificate.

Generate a New IPSec certificate from Publisher and install it in all Subscribers.

Regards,

Madhusudhana Rao. M

New Member

DRS on Publisher

I have tried regenerating the certificate before with no luck. I will try again to see if I have any better luck.

I am basically just deleting the IPSec cert from the publisher.  Regernerating it.  Then downloading it and the uploading it the subscriber correct?

I beleive I know which one is the publisher but want to verify to make sure I'm not running in circles making chanes to a subscriber.

These are the servers in the cluser.  10.20 would be my publisher correct?

olsdcm2.olsdipt.loc 172.19.10.21 olsdcm2 Secondary

olsdcm1.olsdipt.loc 172.19.10.20 olsdcm1 Primary

VIP Purple

DRS on Publisher

Hi Dennis,

From the CLI , u can run command utils service list.

At end of output, u would find Primary Node=TRUE for Publisher and for Subscriber , PRIMARY NODE=FLASE.

regds,

aman

New Member

DRS on Publisher

So I verified that I am working off of the Publisher 10.20

I clicked on the ipsec_cert and regenerated the certificate

It then update the certificate with the publisher host name on the publisher server

I deleted the certiface with the host name off of the subscriber

It automatically created a new one

Serial numbers match

I then restarted DFS master and local on the publisher

No change

New Member

DRS on Publisher

Went through and tried to redo the IPEC on the Pub and Sub

On Publisher

Regenerated the IPSEC_cert

Deleted certificate with Publisher name that was an IPSEC_trust

Downloaded IPSEC_cert

On subscriber

Regenerated the IPSEC (probally doesn't matter but thought Id give it a try anyways)

Deleted the IPSEC_trurst with Publisher name

Uploaded IPSEC_Cert as a IPSEC Trust

Rebooted Subscriber and then Publisher

Still getting the same errors

Local Agent is not responding. This may be due to Master or Local Agent being down. on the DRS screen

2013-12-04 11:44:50,100 INFO [drfLocalRegMonitorThread] - drfLocalWorker.isLAConnected(): Unable to contact server. Master or Local Agent could be down...sleeping for 10 seconds

2013-12-04 11:44:55,811 DEBUG [main] - drfNetServerClient: setupSSLSettings: /usr/local/platform/.security/ipsec/trust-certs/.ipsec-trust.keystore and /usr/local/platform/.security/ipsec/keys/.ipsec-trust_trust.passphrase files exists

2013-12-04 11:44:55,811 DEBUG [main] - drfNetServerClient: setupSSLSettings: set up truststore(password) properties

2013-12-04 11:44:55,811 DEBUG [main] - drfNetServerClient: setupSSLSettings: Create SSLContext and get SocketFactory

2013-12-04 11:44:55,812 DEBUG [main] - drfNetServerClient: setupSSLSettings: Number of entries in truststore : 0

2013-12-04 11:44:55,812 DEBUG [main] - drfNetServerClient: Reconnect: Creating socket from Host: OLSDCM1 port: 4040

2013-12-04 11:44:55,813 DEBUG [main] - drfNetServerClient:Reconnect, Unable to connect to host : [OLSDCM1], message: Connection refused, cause: null

2013-12-04 11:44:55,813 DEBUG [main] - drfNetServerClient.closeSocket, closing connection to: [OLSDCM1]

The system is on the same hardware for the last five years.

I had to reset the password for the webpages for the DRS and OS Administration

I actidently reset the secuirty password and pretty sure they are the same on both Pub and Sub (Could this cause issues)

Other than that nothing has been done to the system. 

Thanks

VIP Purple

DRS on Publisher

Hi Dennis,

Check if DB replication is fine after u have changed the security password.

Suggest opening TAC case.

regds,

aman

Cisco Employee

DRS on Publisher

Hi Dennis,

Try the following steps

Regenerate the PUB's IP sec certificate

wait for 5 minutes as it will push the cert to the sub.

restart master and local agent services in all your nodes.

Rate if it helps

Thanks

Ajay

DRS on Publisher

Hi Dennis,

When did you reset your security password? Before taking backup?

Please check your DB replication from RTMT as well and share the status.

If DB is fine then re-verify your certificate serial number on all the nodes (it should be similar).

During backup, check what alert you are getting in RTMT.

First, verify if the Certificate Serial Number in the keystore of Publisher is present in the Truststore of all Subscribers. Complete these steps:

  1. Log on to CUCM OS Administration page of Publisher server of the cluster setup. Choose Security > Certificate Management. The Certificate List window displays.
  2. You can use the Find controls in order to filter the certificate.
  3. Click on the ipsec.pem file and check the serial number of the certificate.
  4. Log on to CUCM OS Administration page of each node of the cluster. Choose Security > Certificate Management. The Certificate List window displays.
  5. You can use the Find controls in order to filter the certificate.
  6. Click on ipsec-trust.pem file with the file name of hostname of the publisher and check the serial number of the certificate.
  7. Certificate Serial Number should be same on all the nodes of the cluster. If Serial Number of any node is mismatched, complete these steps.
    1. Log on to CUCM OS Admin page of affected node.
    2. Choose Security > Certificate Management. The Certificate List window displays.
    3. You can use the Find controls in order to filter the certificate.
    4. Click on ipsec.pem file and download that certificate.
    5. Find the existing ipsec-trust with the filename of the hostname of the publisher,click on the file name and Delete.
    6. Upload the downloaded ipsec.pem file with the caption ipsec-trust.
    7. Restart the DRF Master Agent(MA)/DRF Local Agent (LA).

Thanks & Regards,

Nishant Savalia

Regards, Nishant Savalia
New Member

DRS on Publisher

I checked replication and it appears to be fine.

Changes made on the Subscriber or Publisher replicate

RTMT reported no issues with replication.

I deleted the trust certificates and regenerated the certificate on the publisher

It created a certificate on both the Pub and Sub right away..

Verified the certificate serial numbers are the same.

Rebooted the Publisher

Rebooted the Subscriber

Same thing

Status: Local Agent is not responding. This may be due to Master or Local Agent being down.

BUG [main] - drfNetServerClient: setupSSLSettings: /usr/local/platform/.security/ipsec/trust-certs/.ipsec-trust.keystore and /usr/local/platform/.security/ipsec/keys/.ipsec-trust_trust.passphrase files exists

2013-12-05 08:44:29,091 DEBUG [main] - drfNetServerClient: setupSSLSettings: set up truststore(password) properties

2013-12-05 08:44:29,091 DEBUG [main] - drfNetServerClient: setupSSLSettings: Create SSLContext and get SocketFactory

2013-12-05 08:44:29,091 DEBUG [main] - drfNetServerClient: setupSSLSettings: Number of entries in truststore : 1

2013-12-05 08:44:29,092 DEBUG [main] - drfNetServerClient: Reconnect: Creating socket from Host: OLSDCM1 port: 4040

2013-12-05 08:44:29,092 DEBUG [main] - drfNetServerClient:Reconnect, Unable to connect to host : [OLSDCM1], message: Connection refused, cause: null

2013-12-05 08:44:29,092 DEBUG [main] - drfNetServerClient.closeSocket, closing connection to: [OLSDCM1]

Is there other logs I can pull from? Right now I'm giong into RTMT Trace and Log Central.  View Real Time Data and selecting Publisher or Subscriber DRF Local.  DRF Master never returns data.

Cisco Employee

DRS on Publisher

Hi Dennis,

Please run the following commands from the CLI

open ports regexp 404

utils disaster_recovery status Backup

file list activelog /platform/drf/trace/

after you run this command you would see file name like drfla.log and drfm.log

then run

file view activelog /platform/drf/trace/

Rate if it helps,

Thanks

Ajay

New Member

DRS on Publisher

Thanks I had pulled up those logs before.  Has the same error.  I will contact Cisco and open ticket.  If I get a resolution Ill post it back here. 

Bronze

Re: DRS on Publisher

Did you perhaps also change IP addresses on the boxes? Any chance DNS is out of sync?

Regards,
Erik

Sent from Cisco Technical Support iPad App

New Member

Re: DRS on Publisher

I didn't change the IP.  There is no record setup in our main DNS servers for the equipment.  It shows its self as the main DNS server

New Member

DRS on Publisher

Hi Dennis,

Did you manage to fix this?

We are facing a similar problem. The DRF Local service on our subscriber stops randomly.

We have to manualy start it to be able to backup the subscriber.

New Member

Hi, did you get this fixed?

Hi, did you get this fixed?

New Member

Hi wotifadmin,in our case the

Hi wotifadmin,

in our case the problem turned out to be NTP related.

We were using a windows server as an NTP server and that wasn't working particularly well.

After moving to another NTP server the problem was fixed.

Hope this helps.

Regards.

New Member

I had to call support who

I had to call support who remoted into the box with root access and deleted a config file that was corrupted.

Once the file was recreated everything started working again

2267
Views
14
Helpful
18
Replies
CreatePlease to create content