I have two independent cases of SPA525G1 and G2 being used as teleworker phones spontaneously rebooting during calls. They are running the latest software (7.4.8). It doesn't happen at any other time, only during phone calls. The SPA525G1 is my office phone which I use from our remote office and the 525G2 belongs to a customer using it remotely to their UC540. Both UC540's are running 15.0(1)XA3a. They typically will reboot at some point during every phone call. I haven't opened a case on this yet.
This is turning out to be a consistent theme at the moment
Is there any way possible you can run a debug on an active call and capture the data and provide it here?
Is there any errors messages showing up on the phone itself when it does reboot or it is just a simple clean reboot?
I would really love to see some solid data on this, this is now 3 cases that I am aware of that is active with what seems to be the same issue, potentially more than that out there.
Have you seen Cisco engineers or TMEs comment on the other cases? I will open a case tomorrow and attempt to get it escalated.
There are no errors on the phone. I haven't been watching the UC540 itself. I was tolerating it for now, but when a customer called me today, the priority changed.
What are the steps to capture debug data and I will capture?
Have you seen Cisco engineers or TMEs comment on the other cases?
No sorry I have not, I do not have viability over the case numbers either, I have though inspected two of the systems configuration and couldn't find anything that sticks out like a sore thumb, the configs looked very sound.
I will open a case tomorrow and attempt to get it escalated.
YES PLEASE!!! I cannot encourage you anymore to do so, the more people who are effected with this problem that log a case, the more support become aware of it and know that there is a problem out in the wild, and not with just a selected few... Please do
What are the steps to capture debug data and I will capture?
I understand that priorities can change The following debugs may not be the entire set either.
Firstly run the following command: ROUTER# term mon
This has to do it, it is a fair bit and do not want to overload the system, but I would like to see what information is produced on these, although they may not elude to the actual problem, but there could be some hints in there.
If the system becomes unstable or is producing unwanted results, please run the following commands imediatly:
If you are using CCA make sure that if you get kicked from the system you can easily or quickly reload it, please make sure that you have a backup on hand (Should not be needed but it would be madness to any diagnosis on a system without a full backup before hand).
The IOS is from Nov 2010 (SWP 8.0.5) https://supportforums.cisco.com/docs/DOC-9827
I doubt highly the combination of that IOS and Phone FW ever went through testing, and I would upgrade.
If this is SSL VPN, disable split tunnel on the UC500 VPN Server side.
You can try what Marcus suggested, and report back since there should be no reason to downgrade SPA525 from 7-4-8 (for SPA50x, we have 7-4-8a) so I would be interested in the result there.
This sounds very much like the problem I am having.
Have you looked into the logs to see if there are any issues with memory and buffers when this happens? I have a load of these appear:
116375: May 23 11:52:25.970: %TCP-6-NOBUFF: TTY0, no buffer available -Process= "
116376: May 23 11:52:25.994: %TCP-6-NOBUFF: TTY0, no buffer available -Process= "
116377: May 23 11:52:26.014: %TCP-6-NOBUFF: TTY0, no buffer available -Process= "
116378: May 23 11:52:26.038: %TCP-6-NOBUFF: TTY0, no buffer available -Process= "
What happens to me is:
1) the phone connects fine on the VPN - it registers normally and everything appears fine
2) I can even use web services through it (e.g. timecard) and nothing happens
3) When I make a call everything appears to work for the first 10 to 20 seconds then the following happens:
i) The incoming traffic appears to stop (I can't hear incoming voice - but outgoing still works for a while)
ii) The phone stops responding (i.e. press a button on it and nothing happens)
iii) Eventually the status bar at the bottom of the screen says 'network error'
During this time the router's buffers start to fail - here are two sho buffers during this phase:
A few seconds Later:
Public buffer pools:
Small buffers, 104 bytes (total 113, permanent 50, peak 129 @ 1d00h):
29 in free list (20 min, 150 max allowed)
8158905 hits, 11417 misses, 1067 trims, 1130 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 47, permanent 25, peak 82 @ 5d16h):
39 in free list (10 min, 150 max allowed)
53853278 hits, 288 misses, 714 trims, 736 created
8 failures (0 no memory)
Big buffers, 1536 bytes (total 3342, permanent 500, peak 3362 @ 5d00h):
0 in free list (500 min, 1000 max allowed)
3463874 hits, 82703 misses, 74599 trims, 77441 created
44588 failures (75750 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
0 in free list (0 min, 100 max allowed)
5249 hits, 44515 misses, 0 trims, 0 created
44515 failures (69189 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 1 @ 4d21h):
0 in free list (0 min, 10 max allowed)
7 hits, 44508 misses, 42 trims, 43 created
44508 failures (69187 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 1 @ 4d21h):
0 in free list (0 min, 4 max allowed)
15 hits, 66065 misses, 42 trims, 43 created
66065 failures (70570 no memory)
4) It will continue in this state for about 1 minute and then the following happends:
i) The phone appears to loose itself - it doesn't unregister it just say's 'connecting' - the VPN stays 'up' according to both the router and the phone.
ii) All connections on the SSL VPN drop (they stop routing but don't disconnect)
iii) The router appears to grind to a halt, these results in phones in the office loosing their line button's and packets getting lost. the phones don't unregister it seems that the router isn't producing a heartbeat while it recovers and the phones don't like it.
4) after about 3 - 5 minutes the router recovers, and everything returns to normal.
If I reload my router the first couple of calls work fine, and then it starts to go down hill.
I am using a UC520 which may be less powerful than your UC560 - this might be why I experience more diabolic consequences....
I have opened a TAC.
Is there a possibility of making a combined thread on this. Looking out on some other techie boards out there we aren't the only people to experience these problems.
I am using a UC520 which may be less powerful than your UC560 - this might be why I experience more diabolic consequences....
I don't believe this to be the case as the original UC-520's were actually pretty robust in specification, but alas I could be wrong on this
Please let me know how you go with TAC, also do let us know when you complete the upgrade as well, note that this is not a big job, 10 minutes of downtime is all it should take so long as the updated IOS is on the flash card you would just reboot and it will upgraded, expect it to take 5-10 minutes to complete... If you use CCA it may take longer (Don't know why).
I hope TAC get to the bottom of it.
I need to correct the initial versions I reported. The customer is using 7.4.6 on a 525G2 and I'm using 7.4.8 on a 525G1. Again, both are being used as teleworker phones in remote sites. Split tunneling is disabled and as Marcus suggested the 'Use as teleworker phone" is not checked for the users.
I can't upgrade the system until an after hour opportunity, but will do that soon. However, I assume phone firmware will remain at the same version as the upgrade is IOS only.
So, with version 7.4.6 and 7.4.8 experiencing the problem, should I spend the time trying 7.4.7 before upgrading IOS?
When you eventually get access to the downloads you'll see a pack for your particular UC. It will be a zip file with all the phone loads in it, the IOS and software to drive the integrated-services-engine (that's the bit that does the voicemail/autoattendant stuff). If you use the CCA software it will do all the upgrading for you. When the phones next connect they should automatically download the latest phoneload file. Sometimes they will spend a while with 'upgrading' on their screens after you reload the router after you do the install.
the other piece is localization. You can get a language pack which will convert the nice american lady on the voicemail's to a nice british lady (or possibly a nice Aussie Lady if that's your fancy). You can do this through the CCA and it can be done at the same time as the software pack is installed.
I would check the teleworker box.
When you upgrade the SWP, you will go back to the 7-4-7, and will then need to drag and drop newer phone FW with CCA. Just a heads up. The Next SWP will have the latest FW.
Technical Solutions Architect - Partner Sales, USA
7025 Kit Creek Road
Research Triangle Park
North Carolina, 27709
I upgraded the UC540 to the latest posted release, 15.1(2)T2. The 525G firmware went back to 7.4.6. I'm now having problem with my two remote sites calling each other. They can initiatiate a call, but there is no audio. I called SBSC and opened a case and they said there were known issues in that area with that version and to roll back to what I had previously, 15.0(5) XA3A.
I will be taking it back to the previous version and to try to resolve the original problem of the 525G phones spontaneously rebooting. I will start by taking the 525G's back to 7.4.7.
I will start by taking the 525G's back to 7.4.7.
That is a wise decision, I would be weary of rolling back the IOS myself.
I have to roll back, the problems I have now are worse than before using this new IOS version. My two remote sites can't call each other. Then I have to solve the original problem and will try 7.4.7 in the phones.
Sorry to hear that mate... I hope you can sort it out soon though.
I rolled back IOS successfully, and it had the 7.4.4 525 Firmware. I upgraded the phones firmware to 7.4.7 but both of my remote 525G1 and G1 had no softkeys. Called SBSC and said there is a know problem in User locale for the phones in the latest version, 15.1(2)T2, and that remanant remained from the upgrade.
The bad user local looks like this:
So the commands to fix the problem are:
no create cnf-files
no user-locale U5 load CME-locale-en_US-English-22.214.171.124.tar
no network-locale U5
user-locale US load CME-locale-en_US-English-126.96.36.199.tar
Apparently there was a typo in the latest software and this would cause 7937 Conference phones to now work, but it was affecting the 525Gs as well.
SBSC also recommended staying with 7.4.7 and not going to any 7.4.8 or 7.4.8a and wait for 7.4.9.
One other tip I received from SBSC was if the phone is a teleworker phone, click the teleworker checkbox on the User/Phone configuration as that will improve voice quality. I don't have it checked for any of my SSL teleworker phones and had some problems with that.
Now I'm back to the original problem of seeing if 7.4.7 fixes the spontaneous reboot problem. I will keep you posted.
We have had the same or similar issues at multiple sites for quite some time.
Cisco doesn't care. Their best suggestion is to basically jump from firmware to firmware randomly till it works. (both IOS and phone load)
We have multiple SPA525G cases open with this problem and other freezing/restarting problems. I even have one thats been stuck at level 2 for well over a month. The cisco dev team seems uninterested -- they apparently don't see the problem in their labs -- so it couldn't possibly exist!
The best combination we have found is IOS15.0(1)XA3a and phone load 7.4.7. This seems to fix the issue in *most* cases -- some still persist though. The only other thing I can suggest is something like a SA520 router with an ipsec tunnel -- definetely not as easy(or cheap) as the sslvpn... but it works.
I have been watching these cases for a while, and seems very sad (the support) reminds me of the Linksys days. These are phones!, What if you need to dial 911 and it drops com'on Cisco!
If it seems to be a buffer issue, can you not set auto tune on the buffers or is that already set?
I hope for your customer's sake this gets resolved soon, we are not recommending the 525's and even though they are much more expensive the 79xx series of phones are rock solid, but it's a tough sell.
Keep these threads alive for the rest of us.
These SSL VPN phones are a major FAIL. I've been battling issues with these phones since day one when we got them. We were able to iron out a lot of the previous issues with call quality, but now with this spontaneous restarting issue it's just unacceptable. The higher ups aren't liking this situation, especially after spending thousands of dollars on a system that's supposed to work. I feel like they released these things without knowing how the phone will work with SSL. The phones work just fine in the office using the internal network.
"The phones work just fine in the office using the internal network."
Its funny you should say that -- we have had some AWFUL issues with SPA525's directly connected to the UC... replacing them with 7900 series phones or SPA504's fixed the issues right away.
Our main issue was 2 phones that would freeze when you picked up a call... the "call timer" would still be going on the screen but the call was dead and none of the buttons worked. It generally took about an hour to get the phones to reboot... if you power cycled the phone it would sit at the cisco logo screen for forever. STAC was completely unable to resolve this issue -- currently Level 2 support has these phones in their possession -- they have been unable to reproduce the problem... so as far as I know the case is being dropped and we just have to eat the cost of 2 new 7975's... with sidecars no less....
On the ssl vpn topic,
Last week STAC gave me a beta firmware to test that is supposed to fix the "refreshing voice components" issue when ssl vpn is in use. Not only did the upgrade not fix it -- it killed the entire system during the upgrade... I had to call the customer and have them power cycle the box. Awsome. (This was after confirming with cisco REPEATEDLY that I could do this over a VPN connection -- the customers location is not geographically near us.) Anyway, on the first call the customer tried on the new firmware the phone reset within the first minute.
I don't understand why Cisco thinks this is OK? How are we supposed to sell this stuff -- I've only been working with Cisco equipment a fairly short time now... but I honestly COULD NOT recommend them to anyone for any reason -- this kind of stuff is simply UNACCEPTABLE for a phone system. It would be one thing is this 525 issue was a standalone issue -- but its not; the UC500 series seems to have an endless supply of bugs.
Sorry for ranting.
After embarking on a rebuild over the weekend with another user of the Cisco community forums, helping out someone who could do with a friendly person on the other end of the line, I decided to do some trial testing (Spurred on by Michael's dedication, and he himself having this issue) the results were very interesting.
The field result was conducted on 3 SPA-525G2's off our UC-560 system 2 of them located at my home and another one located down the road at a friends house. However the UC-560 was controlling the xDSL connection with the Cisco 857 in Bridge mode (Yes might seem like an overkill with an 857 but there is some method in the madness).
No matter what I tried I could not get the phones to do the spontaneous reboot, but if the modem was not in bridge mode and was routing all the traffic, the phones would do the spontaneous reboot roughly around the 10-15 minute mark, and on ocasion around the 2-3 minute mark, although at times they might have lost some connectivity, I just attributed this to the poor Internet connection I have at home. However in saying that I do notice that using EZ_VPN over SSL VPN is much more stable, I don't know why as I am not a data expert, but the SPA-508G had no issues what so ever, no drop-outs and very few audio drop-outs.
Given that the wife was yelling at me working on the weekend, the tests did not get too far indeepth analysing the traffic moving back and forwards, but the results none-the-less were still interesting.
If anyone has the ability to give this scenario a shot, I would be interested to see the results of it, although I do appreciate that most of these systems are in production and you may not have that opportunity to do it, but still worth a shot posting this and seeing if anyone has the ability to do it.
I'm familiar with that thread, in fact we should merge the two if possible. I'll try to do some testing on my own and let you know. As for our setup, we have a 50Mbps dedicated fiber connection directly attached to the UC. It is a shared connection between our main firewall but it doesn't go through it. We have about 4 users on SSL VPN with SPA525G phones running 7.4.7 firmware. We've been experiencing the same issues with the call drop out, etc.
Do you still have this test setup in place? If so, I would like to have you try one thing to see if the spontaneous reboots stop.
I have a setup in my lab at work with a DSL attached to a basic Netgear wireless router and the SPA525G behind that. The other end is our UC560 on a 50Mbps dedicated fiber. I just tested it a few minutes ago, and the phone rebooted after 10 minutes into a call.
No not setup any longer, it turns out that the system ends up in a High CPU loop anyway and system becomes un-responsive, based on secondary tests carried out, I hadn't posted it because I wanted to investigate it further, but since this system is used in our demo room the boss would not be too happy if I continued to kill the system as I hadn't made him aware that I was testing this and would rather not expose this problem to him, anything that sways his mind to go back and sell LG phone systems full time would be VERY BAD for me.
The conclusion is that we are not going to sell the SPA-525G (Both derivatives) as remote handsets with in-built VPN capabilities, we will continue to encourage the clients to purchase a Cisco 857 and use EZ_VPN (IPSEC), simply put IT JUST WORKS and we just cannot afford to throw money at supporting an endless cycle, on top of that looking silly in front of the clients all the time is also not a fun thing.
Given I dont have a LAB specific system to do testing that has potential ramifications, I have to stop, I will do ones that will not bring the system down as I can get away with it, but in this day and age asking for expensive kit to just sit around for testing purposes is getting too hard to get through the budgets, margins are too tight and operating expenses are going up (I miss the old days)...
I badly wanted to help everyone out with this problem, but I have reached certain limits
I just got off the phone with Cisco support, they have a non-public update to IOS (T3c) and the ROMMON. I will test it tonight and post the results.
If you are using firmware version 7.4.8 please try defaulting the phone and putting on 7.4.7 to resolve this issue. You can do this by simply dragging and dropping the firmware onto the UC540 icon in the topology view via CCA. Once the firmware is uploaded then simply reboot the phone via CCA by right-clicking on the icon for the phone in topology view and selecting the "reboot" option. As an additional step, try going through the factory reset process by selecting this on the phone itself and then rebooting. This is what I prefer to ensure that I am in fact only using the parameters of the 7.4.7 release.
All the best,
We tried the T3c firmware at a site that was having the "refreshing voice components" problem.... FIXED!
(of course this is after they gave us T3b -- which killed the system, but whos counting...)
I've applied the update last night, tested on call for 45 minutes and today one for 1 hour and so far no drops or reboots.
All is well so far.