Errors with UCCX 7.0

Unanswered Question
Jun 29th, 2010
User Badges:

For the last couple of Monday's when the staff have came in they could not log into the CAD. The error message was 'The Cisco Unified CCX application server may be offline'. We also could not log in via browser to the Appadmin screen. AXL errors were mentioned.


On a reboot of the UCCX server everything started working again. The worry is that there is an underlying problem that may be causing a service to stop and this will continue to happen. I have attached our event logs. We also have an other couple of issues that has been outstanding for couple of weeks where the Voice CSQ Summary is not being displayed within the Supervisor program. Also the VoIP Monitoring and VoIP Recording are not working.


To my knowledge there were no power downs or any network problems over the two weekends. The contact centre closes on a Fridaty night at 5pm and there is no one in again until 8am on the Monday.

Attachment: 
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
pcaltagirone22 Thu, 08/19/2010 - 07:53
User Badges:

Hey Barry,


Not sure if you are still expereincing the same issues but we recently upgraded from CRS 5.0(2) to UCCX v8.0(2) and saw the same UCCX server may be offline message on a single agent workstation - all other agents were fine and we had full access to AppAdmin. I was able to resolve the issue by rebooting the station phone - figure it may be worth a shot for you if you are still seeing this happen.


Paul

joesnyde Fri, 08/20/2010 - 12:03
User Badges:
  • Cisco Employee,

The next time you have an issue with login, go to the CCMUser page and try to login. If this fails you have a problem with CUCM authentication or AD if integrated. As for the Supervisor issue, try opening the CDA web interface and do a directory sync. The reason for doing the directory sync through the web gui is that it will throw an error code.


I am attaching the CAD error code guide which might help.


Regards,


Joe

Attachment: 
pcaltagirone22 Wed, 10/27/2010 - 06:42
User Badges:

I have run into this issue on a few more stations (3-4) since then and rebooting the phones unfortunately does not always clear up the issue. I opened a TAC case and worked with them for hours and we still couldn't get to the bottom of it. The only way I resolved it is by rebooting the UCCX cluster after hrs - Since I rebooted the cluster I have not seen that error message, although it has only been about 10 days since the reboot. Good luck!

Brett Hanson Mon, 02/07/2011 - 01:05
User Badges:

I had this same issue, with 2 users now over a period of about 4 months - using UCCX 8.0.2.


I tried everything except assigning a new extension to the person (which will be what I try next time it happens...if it happens).


My most recent workaround was to create a new user in AD and created everything again for the person (deleted device, etc.)

We use LDAP sych for our users - hence the new user in AD.


E.g. problem user is Joe Bloggs username jbloggs

Created minimal account Joe Bloggs2 username jbloggs2, then setup the same phone with them (and extension) - works now.

They login to their PC with their normal jbloggs account but sign into CAD using jbloggs2.


I consider this a workaround and hope to resolve - otherwise it's a pain to have to create additional AD users.


My next step (and if the issue hadn't shown so late in the day - when crew were changing shifts - I would've done this) - will be to try assign a new extension to them.

As it's call centre - doesn't make a huge difference.

In fact - I might try provision them a new directory number, delete the old one, test their login, then restore the old DN and re-assign to see if it's simply the DN.


But, if all else fails - you can pursue the new user option.

Sux - but it's pretty quick to configure.

Brett Hanson Wed, 02/23/2011 - 19:17
User Badges:

Hi all,


Follow on from my previous post on the 7th of Feb 2011...


Yes - we have UCCX 8.0.2, CUCM 8.0.2, and CAD 8.0(2) Premium Version - and suffer this and similar issues intermittently.


Our workaround for the "The Cisco Unified CCX application server may be offline. Please verify your system is online and try again" error (which happens to some staff and not others intermittently) was this ...

(same as the last update here before mine) - create a new AD account (solely for the purpose of signing into CAD).


Before we figured out that workaround, we tried heaps of things:

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";}

  • Removing user from UCCX and adding them back in - Error
  • Completely removing from UCCX and CUCM and adding them back in - Error
  • Same as last step but with a new extension - Error
  • Having someone else log in on the CAD which is erroring – Success
  • Have User with issue log into known working CAD – Error
  • Giving User new AD account (Copied from previous account) – Success


Interesting thing that happened today...

A user who previously had the problem jbloggs, had the issue resolved by creating a new AD account jbloggs2.

Today, jbloggs2 had the issue happen again.

As a test, we had them use their normal windows account (the old account they used to use to login to CAD) - jbloggs....it WORKED!

Weird.


Now, the other day our DC provider had a total loss of power (fire suppression system issue) and our CCX server was power cycled unexpectidly.

The issue didn't surface at this point - it only surfaced again today.

I suspect, however, that the CCX box reboot resolved the jbloggs account issue.


Another thing you mentioned in the original post was "We also have an other couple of issues that has been outstanding for  couple of weeks where the Voice CSQ Summary is not being displayed  within the Supervisor program."

After the power cycle the other day - we had this exact same issue.

This was resolved by logging into UCCX Serviceability and restarting 2 services under "Tools" > "Control Center - Network Services".

Service 1: Cisco Desktop Enterprise Service

Service 2: Cisco Desktop Recording and Statistics Service


Interestingly, none of our Service Desk folk reported any drops during the quick restart of said services.

We do have a HA cluster setup with our main box here in Perth and our other in Sydney - so can't be 100% sure that there was no drop due to this.

PS. I restarted the services on both our Perth and Sydney boxes in turn.


My real issue is WHY is this happening to start with?

I made a change to a bunch of the users phones in CUCM (removed a second line association from all their devices) - anyone who didn't log out yesterday did not have the problem. Anyone who did log out yesterday came into work today and had the problem. I also modified the script which those particular staff are running with - but only changed the times calls are diverted away from them - can't imagine it is related to the script.


I will be logging a TAC case today - will see what they can come up with.

vincent.morton Mon, 03/21/2011 - 03:54
User Badges:

Brett,


How did you get on with TAC?


We're seeing the same issues with a new install of UCCX 8.0.2 with CUCM 8.0.2 linked to AD.


Can't log into the appadmin web page using the app admin account, error message "AXL Authentication Failed. Please try again." and can't log a user into desktop agent, error message "...server may be offline...".

Brett Hanson Fri, 07/22/2011 - 22:15
User Badges:

Yikes, sorry about the late reply - notification must've disappeared amongst the thousand unread emails!


Yeah - TAC weren't much use for this issue - they couldn't explain WHY it was happening (the CAD user login issue) - even given my workaround. I forgot the specifics but recall them wanting us to bounce the box which just wasn't an option at the time.


Anyway - I forgot to mention that we didn't lose access to our APPADMIN page.

Our trouble was with individual users only.


With respect to the AXL auth failed error you guys had - I can only imagine that maybe the connection between your UCCX box and CUCM could have been an issue (not the network link specifically - but the 'connection') - as the AXL user is a CUCM 'Application User' system. Restarting your CCX box would likely have restored the connection - but there is likely a better way. Read on...


Question - when you were unable to login to the APPADMIN page of your CCX box, were you also unable to login to CCX Serviceability (UCCXSERVICE page)?

If you could get into serviceability - then I would've tried restarting the 'Cisco Unified CCX Engine' service before thinking about bouncing the box. This would've restarted the subsystem manager service 'Unified CM Telephony Subsystem' - which could quite likely have achieved the same outcome.

If servieability was unavailable - you should still have been able to SSH into the CCX CLI and restart the service as follows:


utils service restart Cisco Unified CCX Engine


It also may have been worth checking your syslog logs (external) for 'OOS' messages to see if it is a specific service or services.


Whenever anything goes astray with our CCX box - I immediately jump  onto our remote SYSLOG (RHEL with SYSLOG-NG) server and run the  following command:

cat | grep -i oos

It shows any services which are in trouble - including CCX  triggers (route points). I then look to restart those services or  investigate further with that knowledge.


Obviously you need to have configured Remote Syslog on your CCX box beforehand.

To do this - connect to CCX Serviceability [https:///UCCXSERVICE] - click on "ALARM" > "CONFIGURATION" > check "Enable Alarm" under "Remote Syslogs" heading and set Alarm Event level.

We use 6 (informational) - and the IP address of your syslog server.

Once you apply - your CCX box will send syslogs to your remote syslog server. (we have cron jobs to gz and archive the logs daily - otherwise the size of informational CCX logs can get out of hand).


Interestingly - we haven't had this problem show itself since my last post on this discussion.

We did update our UCCX from 8.0.2.10000-41 to 8.0.2.11004-12 (via Service Update) shortly after this problem (to resolve other issues) - which may help explain it.

Andrew Skelly Wed, 07/27/2011 - 08:44
User Badges:
  • Silver, 250 points or more

We've run in to this issue also and opened a TAC case.  The fix they  gave us for it was to modify the PhoneDev.cfg file.  This worked for our  users that were having the issue.


C:\Program Files\Cisco\Desktop\config


Open the “PhoneDev.cfg” file in Notepad.  Once the file opens,  paste the below 2 lines to the bottom of the file (after the  “AgentWorkMode=0” line)


[ReqTimeout]

Milliseconds=60000


Save the file and then try to open CAD.


We had the  issue happen another time and found that the users were unsynced from  LDAP in UCM (User Management - End User).  They showed as Inactive.   After making them active again in LDAP (they were expired) and then  resyncing the LDAP service, the users were able to get back in.

Brett Hanson Wed, 07/27/2011 - 17:39
User Badges:

Hi Andrew,


Thanks for taking the time to post these tips.

Appreciate the details.


Thankfully our company hasn't had this trouble with its phones for a while - but will def. keep these in mind for if/when it happens again.


Cheers,

Brett

shh5455 Tue, 11/15/2011 - 10:40
User Badges:
  • Bronze, 100 points or more

The username is also case sensitive with CAD.  We have some users whose usernames in AD were uppercase.  They were previously able to login using an uppercase username.  It appears that they have been changed to lowercase in AD now.  We received this error whenever trying to continue using the uppercase username.  Using the lowercase username allowed us to login.

Matheus Lopes Wed, 05/21/2014 - 13:38
User Badges:

I got the same issue, some users couldn´t log in on CAD "server may be offline" and i coundn´t log into the appadmin and serviceability web page using the app admin account, error message "AXL Authentication Failed.

I reboot the phone stations and CCX box, not successful. Then i realised that i changed the password of the CUCM (Application Users) and this account was associated to CCX AXL Service Provider (CCX/Subsystem/Cisco Unified CM Telefony/Provider/Modify Cisco Unified). So i set back the previous password on CUCM and it was fixed. Or if you like you can configure a new account on CUCM (Application Users) and set the same acc on CCX AXL Service Provider.

Actions

This Discussion