Potential Bug Found with SPA-525G2 and FW 7-4-8

[David Trad]

Hi All,

Today whilst working with SBS trying to troubleshoot a problem with a reception phones (SPA-525G2 with 2X 500s Modules), nothing in the obvious could be found, until the support call ended (Murphy's law they call it).

The Issue:

(SBS Case# 618585561)

Reception Phone a SPA-525G2 with 2X SPA-500s modules places a call on Park slot 1 (SPA-500s Module 2), Receptionist tries to retrieve the call again and the phone appears to be alive but is actually dead, no dial tone and no response to key strokes, then a message appears on the screen where the company message usually resides "Network Error" and then the phone reboots.

Now since this is an easily reproducible problem, I decided once the support call ended to do some debugging on the CLI (Old habits die hard) and the following results were yielded.

001261: Aug  2 01:50:46.359: peer_vdb is NULL for DN 597, chan 1 at SkinnyIsPeerNonSCCP

001262: Aug  2 01:50:46.363: peer_vdb is NULL for DN 596, chan 1 at SkinnyIsPeerNonSCCP

001263: Aug  2 01:50:46.363: peer_vdb is NULL for DN 556, chan 1 at SkinnyIsPeerNonSCCP

001264: Aug  2 01:50:46.379: peer_vdb is NULL for DN 597, chan 1 at SkinnyIsPeerNonSCCP

001265: Aug  2 01:50:47.615: peer_vdb is NULL for DN 596, chan 1 at SkinnyIsPeerNonSCCP

001266: Aug  2 01:50:47.615: peer_vdb is NULL for DN 556, chan 1 at SkinnyIsPeerNonSCCP

001267: Aug  2 01:50:47.643: %ISDN-6-CONNECT: Interface Serial0/3/0:4 is now connected to 414474751 N/A

001268: Aug  2 01:50:51.115: peer_vdb is NULL for DN 597, chan 2 at SkinnyIsPeerNonSCCP

001269: Aug  2 01:50:51.119: ephone-1[0/52][SEP405539A27D38]:SkinnyMohTransfertoPhone transferee_dn=-1, transferee_chan=1

001270: Aug  2 01:50:51.135: PARK DN 547 SKINNY_CALL_RINGING->OffHook pending ref 224

001271: Aug  2 01:50:52.971: sccp_endpoint_vsa_query:Invalid phone for dn 547 chan 1

001272: Aug  2 01:51:50.287: %ISDN-6-DISCONNECT: Interface Serial0/3/0:4  disconnected from 414474751 , call lasted 62 seconds

001273: Aug  2 01:51:52.855: socket 52 fatal error 254! can't read msg header with size -1, fd 52 phone tag 1 device name SEP405539A27D38 state 1 keepalive 61

001274: Aug  2 01:51:53.019: %IPPHONE-6-REG_ALARM: Name=SEP405539A27D38  Load=7.4.8 Last=Hard+Unknown

001275: Aug  2 01:51:53.039: Bring down DN 597 chan 1 (800) with the following traceback -Traceback= 0x8081CE78z 0x8081D3E4z 0x8081FC30z 0x80852050z 0x80852A4Cz 0x80854EC8z 0x808001E4z 0x8080A5D8z 0x802D560Cz 0x802BBEB4z

001276: Aug  2 01:51:53.039: Bring down DN 597 chan 2 (800) with the following traceback -Traceback= 0x8081CE78z 0x8081D3E4z 0x8081FC30z 0x80852050z 0x80852A4Cz 0x80854EC8z 0x808001E4z 0x8080A5D8z 0x802D560Cz 0x802BBEB4z

001277: Aug  2 01:51:53.063: %IPPHONE-6-UNREGISTER_ABNORMAL: ephone-1:SEP405539A27D38 IP:10.1.1.10 Socket:52 DeviceType:Phone has unregistered abnormally.

001278: Aug  2 01:51:53.063: %IPPHONE-6-REGISTER: ephone-1:SEP405539A27D38 IP:10.1.1.10 Socket:43 DeviceType:Phone has registered.

This issue is reproducible on this particular phone, other phones are SPA-508G's and do not suffer the same issue, which also explains why I could not re-produce this issue at other sites because the SPA-525G2 was not in use at those sites.

This problem also only happens when parking and or retrieving a parked call, and it is the same error every time.

The next step is to down-grade the firmware to 7-4-7 and see if that resolves it, sadly due to a Fortigate system being in place, I was unable to do a TFTP to the system, and Drag and Drop with CCA 3.1 with the firmware does not work like 3.0.1 any longer.

Not sure if anyone else has come across this issue, but if you are having problems that resemble this issue, then debug it on the CLI and see if you are getting the same error, the following would need to be done to debug it:

ROUTER# term mon

ROUTER# debug tftp event

ROUTER# debug ephone error mac-add <insert ephones mac here>

This will show you the above results or what ever the error is that is being produced, if it is the same please log a case so Support can see a pattern and identify it as a bug... PLEASE.

Resolution:

None yet, but an upgrade to the system to SWP-8.1 and a potential downgrade of the phones FW to 7-4-7 may be in order, will update the post if the downgrade resolves the issue.

Cheers,

David.

[bill.roland]

I can't add anything useful to this post, but I can add that we have seen our SPA525Gs (as well as our now gone SPA508s) display "Network Error" on the screen and drop calls.  I have also had cases where the SPA525G would become "deceased" according to the UC560.  The screen would be normal and the call timer would just keep on running.  Couldn't do anything until you rebooted the phones.  Interestingly, but not suprisingly, the 7900 phones have never experienced either of these issues. 

All I can say is, good luck. 

[David Trad]

Hi Bill,

Bill Roland wrote:  
All I can say is, good luck.  

I am going to need it, even the SBS staff was like just over it, and I could only feel for him, despite the real crappy VoIP line back to Australia so far he has been a treat to work with, with the exception of not getting the time zones right, he woke me up at 4am and I couldn't fall back a sleep... I'll get him back though, just you wait and see

Finding the problem out is the hard part, now that I managed to do that, I will get them to escalate this up too engineering and have them work on it from there, it is a bad...bad...bad bug in the system that needs to be arrested straight away, and I suspect resolving this problem will also resolve other outstanding issues as well, mainly SSL VPN because this is an almost identical error to what was being produced on a few remote SPA-525G2 a little while back.

Cheers,

David.

[Anna Tsukerman]

Hi David,

Can you please email me the TAC case number offline. I want to forward it to the SPA team to take a look at.

Thanks,

Anna

[David Trad]

Hi Team,

I wish to give a HUGE thanks to the Cisco Dev team, today they gave me a lend of a firmware that is work in progress, and after applying it and upgrading the SPA-525G2, the problem could not be reproduced at all.

I guess we will see if the fix is a true fix and over the next couple of days the results will be shown.

Just as an FYI, I have been playing round with this bug to see what its potential consequences are, interesting to know that when you have 3X SPA-525G2 phones connected and if at least 2 of them at the same time park a call, the UC-500 reboots.... WARNING!!! only test this in a LAB environment, the client would probably take exception to you doing this to them (Speaking from first hand here LOL)

I also should point out that the Dev team turned the testing firmware around in pretty good time, but I suspect I have Anna to partly thank for that

Here's to hoping an official/permanent fix for this comes out soon, it is a ugly and nasty bug to have linger in your system.

Cheers,

David.`

[kbaker-mlr]

Hi David,

We have also fell victim to this bug!!! are there any updates to where the sollution lies??

Our issue is totally random, when incoming calls hit a blast group of 525G2's, you lift the receiver on a phone and it is silent, even though the phone suggests you are connected!!! then it will do the 'network error' thing and a quick reboot. We are also seeing the 'deceased' notifications in CCA.

FYI - Phones are running firmware 7-4-8 and UC560 has software pack 8.0.2. The same issue occued with phones on firmware to 7-4-6, thats why I upgraded!! have downloaded .9 from Cisco web site and that just messed up the softkeys?

Any update on this bug would be much appreciated, after reading this thread and the following link I'm a bit nervous as the restaurant the system is installed in is about to go live!!

https://supportforums.cisco.com/message/3444085#3444085

Cheers,

Keith

[David Trad]

Hi Keith,

I am using a firmware supplied to me by engineering, so far it has halted the issue, although I point out that the problem is not noticed on a UC-540 like it is on a UC-560 so my concern now is redirecting to what are the differences between the system to have this type of result.

I have attached the firmware given to me by the engineering team, use this as it should resolve that problem, if it does not then your issue is something else and you need to get SBSC involved

Cheers,

David.

[kbaker-mlr]

Hi David,

Thankyou for the rapid response, I will let you know how I get on.

Much appreciated,

Keith

[David Trad]

Good Luck

Cheers,

David.