Distributor configuration is out-of-date - AW problem

[Kenan Muharemagic]

I could realy use your help right now...

We upgraded ICM to 7.5.9 because of the bug fixes...

After the upgrade i can not do any config changes using the config manager...

i can see every config but cannot change anything

tried to init local database, that goes through without any errors, but still i am locked out

When i try to change anything (ex. delete a campaign or create a new one) i get:

"Distributor configuration database is out-of-date. Wait a moment any then try again. If this message appears repeatedly check on the status of Updateaw on the distributor"

the updateaw process reports:

Admin Workstation configuration database is out-of-date with the central controller. Please update your local ICM database. Aborting...

I tried to recreate the awdb but it fails saying that awdb is in use...

anyone have any ideas?

[geoff]

Very strange, and I have not seen this.

With the Distributor running, when you do "Initialize Local Database" it will completely drop the tables in the AWDB and rebuild them from the Logger. At the end of this process, updateaw should be saying "Waiting for new work".

Just wondering - are the AW and the Central Controller as members of the Domain synchronized to the PDC emulator, normally running on the backup Domain Controller (where the GC is not), and showing exactly the same time.

Something like

w32tm /config /syncfromflags:domhier /update

then stop and restart Windows Time service.

Regards,

Geoff

[Kenan Muharemagic]

The time is syncronized with the domain controllers, i checked again...

I reinstalled the AW without any effect...

I tried to recreate the awdb before reinstalling the aw but the icmdba says cannot delete awdb, database in use...

Rolled back to 7.5.7 but the problem is still present...

It's 7 am here and i've been at it all night...

update:

i get a message from RouterA router process:

UpdateCC from Administrator on RBBH-AW rejected because wrong key used. Expected 576864042017, got 576864042017.

it shows the same number expected and got...

Kenan Muharemagić

[Abu Hadee]

Hi

You need to check following

use rttest command on router

rttest /cust /node routera

config_status

compare the number with output of aw_control table on Logger and AW.

In most cases, you need to do "exit_router on router and then recreate or initialize aw db.

thanks

- abu

[Irfan Tariq Javed]

HI Abu

We are also facing this issue using ICM 8.0.3 . When we try to save any change it gives this error "

"Distributor configuration database is out-of-date. Wait a moment any then try again. If this message appears repeatedly check on the status of Updateaw on the distributor"

Abu we followed the steps you mentioned . AW & Router keys are same.

Regards

Irfan Tariq

[twschafer]

Look at this bug CSCtj51228

[Kenan Muharemagic]

We solved this issue with the help of TAC. The team at TAC mailed us a detailed procedure to correct this problem by manualy deleteing certain keys from the AW and Logger databases.

You can find the procedure at this link

http://www.cisco.com/en/US/ts/fn/633/fn63371.html

Or bellow:

Regarding defect CSCtj65060:
Currently no software resolution is available for this defect. As a safeguard, it is strongly recommended that ICM/UCCE administrators uncheck the "Truncate Config Message Log" option while using the ICMDBA tool to export the configuration database from the Logger. However if you are experiencing symptoms described above, please see the defect CSCtj65060 for workaround.

If you are experiencing the symptom mentioned below, perform these steps noted to workaround the issue:

1. Stop RouterA, RouterB, LoggerA, Logger B.
2. Stop AWs.
3. Take the back-up of the Logger database.
4. Delete the rows from Config_Message_Log table from LoggerA, except the first row with RecoveryKey=0. Sample SQL Delete Query:
   delete Config_Message_Log where RecoveryKey >= 1
5. Restart the LoggerA, RouterA and One AW which is connecting to the A side.
6. Perform configuration updates and verify they are successful.
7. Now Sync the logger database from A->B using ICMDBA.
8. Start LoggerB and RouterB, followed by any other AWs.

Best of luck,

Kenan

[Irfan Tariq Javed]

thanks yes I have solved the issue by using above mentioned steps.

Regards

Irfan Tariq

[Oleh Doroshenko]

just completely stopped Problem SIde(Side B in my case)

Stopped Router B, Logger B, AW B.

Started Router B,Logger B, waited couple minutes and started AW B. After while databases synced and i was able to update database without error.