SPA 9000 randomly reboots

Unanswered Question
Jun 9th, 2009
User Badges:

Hello


I'm currnetly working with a customer on a new install (live for 2 weeks), the system appears to reboot at random times where all the phones will reset. I've attempted to setup the log server but the SPA9000 will not send any messages (I've tried all log levels), I get responses from the 2 SPA400 so I've ruled out the server or firewall issue. Is it possable to have a bad system or partially failed firmware update, upgraded to 6.1.5 before install and did not see this while bench testing the setup.


Thanks

Joe

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Alberto Montilla Wed, 06/10/2009 - 03:00
User Badges:
  • Cisco Employee,

Hi Joe;


Are you using remote provisioning at all for this setup?


Frequent reboots are not common at all, unless you are using remote provisioning and affecting some low level parameters. Debug logs definitively will help to determine the reboot cause (and then whether the box is broken or if it is a config issue).


Regards
Alberto

j.lanzon Wed, 06/10/2009 - 07:24
User Badges:

Hi Alberto,


No remote provisioning, one of the 400s connects to PSTN.


When I tried to setup loggin and was getting nothing that's when I thought there may be an issue with the 9000 itself as the setup I use here in the office is identical and have never seen it behave like this, since it does look like an issue with the 9000 now I am going to swap it and see if it fixes the problem; if not then it's something with my config.


Thanks for the reply and I'll update after the swap.

Joe

Patrick Born Wed, 06/10/2009 - 08:21
User Badges:
  • Cisco Employee,

Hi Joe,


Can you confirm:

- if you performed a factory reset on the SPA9000 before starting [so you have a good, known starting point]

- if you used the SPA9000 Setup Wizard v 2.1 to configure the SPA9000?

- if you're using 6.1.5 firmware on the SPA9000?

- if the SPA9000 has a static IP address

- that you have nothing connected to the SPA9000's ETHERNET port


WRT logging, did you follow the instructions described in https://www.myciscocommunity.com/docs/DOC-3032


Regards,



Patrick

----------

j.lanzon Wed, 06/10/2009 - 08:54
User Badges:

Hi Patrick


I did the factory reset twice, I just remembered that at one point half the phones during the bench testing would not go live; the line keys stayed orange like the 9000 was disconnected. When I reset the phones the system registered them as new and gave the phone a new extension.

I did use the 2.1 setup wizard.

Firmware is 6.1.5, HW is 1.0.5

Static IPs are used on everything but desktops and phones, everythings on the same VLAN; only 8 people.

Ethernet is unplugged.


I just double checked and the SIP Debug option was set to none, I have changed it to full on both lines 1 (voice mail) and 2 (PSTN) and still so not recieve any messages. I also changed the 400s and do get messages so the syslog server is working.


Joe

Patrick Born Wed, 06/10/2009 - 11:06
User Badges:
  • Cisco Employee,

Hi Joe,


Everything sounds good. Could it be that something on your network is getting between your SPA9000 and the syslog server?

Are you using VLANs?

Is the SPA9000 on the same network switch as the syslog server?


Please reboot the SPA9000 with http:///admin/reboot and watch every phone.

Did every phone reboot too? If no, something on your LAN is interfering with multicast.


What LAN switch are you using? Please make sure that multicast is not inhibited in any way.


Can you capture a Wireshark trace on the SPA9000 LAN? This will show what, if anything, the SPA9000 is doing on the LAN.


Regards,



Patrick

----------

j.lanzon Wed, 06/10/2009 - 11:58
User Badges:



Hi Patrick,


Can’t rule out something network related but doesn’t seem likely. The SPA9000, both SPA400s, RVS4000, Microsoft 2008 SBS server and single SPA942 are all connected to a SRW208P; all the desktops and phones are on  a SFE2000, common VLAN for all.


All phones did reboot after the system.


I can do the capture when I’m onsite tomorrow, the only machine I have access to remotely is the server and cannot mirror the SPA9000 port to the server but the only packets I see from the SPA9000 right now are ARP requests, no syslog like I see from the SPA400’s


Thanks

Joe

Patrick Born Thu, 06/11/2009 - 14:18
User Badges:
  • Cisco Employee,

Joe,


It's good that the phones reboot when you reboot the SPA9000. This proves that multicast is getting through.


On the SPA9000 please verify that you  have the correct syslog server address in Voice tab > System > Miscellaneous Settings > Debug Server:

Also verify that you've set Voice tab > System > Miscellaneous Settings > Debug Level: 3


Is there any chance that the power supply / power strip for the SPA9000 is either faulty, or getting bumped, causing the device to restart?

Have you noticed any pattern to the random reboots? How many times a hour/day/week does the device reboot?


Could it be that the DHCP server that provides the phones with IP addresses has a short DHCP lease time? The phones will reboot when their IP addresses change...


Regards,



Patrick

-----------

j.lanzon Thu, 06/11/2009 - 15:06
User Badges:



Patrick,


A few updates… I was able to get to the customer today and swapped the SPA9000 and so far it’s good news the system has yet to reboot on them for the 6 hours after I left.


Now before I did the swap I was able to get a 10 minute capture, mirrored the port the SPA9000 was on, capture is close 10MB.

The debug server was setup with the correct address and is now sending data to the logging server.

No power strips are being used, directly plugged into the back of the UPS; one thing I noticed that surprised me and may change things, in the Info tab->System Status ->Elapsed time was 11 days…was not expecting that.

Reboots were told are random as well as how often, nothing to lead me in a direction.

DHCP is set for 8 days


I have the system here now so if there’s anything you would like me to try I have more flexibility now.


Joe

Patrick Born Thu, 06/11/2009 - 16:06
User Badges:
  • Cisco Employee,

Hi Joe,


Wow, 11 days of uptime... that sounds like it's not the SPA9000 rebooting, causing the phones to reboot.

I can't think of anything for you to try with the SPA9000 aside from connecting it up and running it. If it stays up for 11 days, it's more stable than my computer...


Please keep me posted on how the new SPA9000 behaves. Sounds like the site may still have issues...


Regards,



Patrick

----------

j.lanzon Thu, 06/25/2009 - 12:00
User Badges:

Just wanted to report 2 weeks later and all is well, no reboots; only minor complaint now is with the white noise being a little loud and 2 ring delay.


I have also been using the 'problematic' SPA9000 with no problems here in the office so looks to be just a quirk in the original programming that the factory reset cleared.

Patrick Born Thu, 06/25/2009 - 13:01
User Badges:
  • Cisco Employee,

Hi Joe,


Thanks for the update. I'm glad to read that the site has settled somewhat.

WRT two rings... this is to allow caller id information to be fully received in some geographies. We're busy testing new SPA400 firmware that allows you to reduce the answer after field to less than 2 rings if you don't care about caller id. We're hoping the firmware will be available in about a month...


Regards,



Patrick

-----------

j.lanzon Thu, 06/25/2009 - 13:21
User Badges:

Hi Patrick,


I already explained about the 2 ring issue and they were OK with it knowing a fix was in the works, unless you call the office while inside the office you don't realize it anyway. Are there any rumors about the white noise, I know removing it is not the best choice but being able to adjust it would be helpful.


Thanks for all the help

Joe

j.lanzon Wed, 07/01/2009 - 11:41
User Badges:

Hi Patrick,


Bad news, customer just called and says the problems continue. The list of complaints...




-Incoming call dropping mid way.

-Incoming call has no one on the other end.

-Phones ring after completed call with same caller-id but no ones there.

-Outbound calls do not complete, full numbers dialed but no ring back.

-Phones randomly reboot, not always all of them.


I've re-started the syslog server and am capturing events now, I've also asked them to record the times these issues happen.

I am going to head over there tomorrow and isolate the phones to thier own VLAN and leave a sniffer on the air to capture everything.

One thing I noticed as I was starting the syslog server, the web interface of the SPA9000 no longer responds.


Any other items you can think on to assist in troubleshooting?



Thanks

Joe

Patrick Born Wed, 07/01/2009 - 14:57
User Badges:
  • Cisco Employee,

Hi Joe,


Please confirm:

  • 1 SPA9000 no ITSP
  • 1 SPA400 no PSTN voicemail only
  • 1 SPA400 4x PSTN lines
  • How many phones are configured in this system?


Assuming my understanding about the SPA400 config above is correct, please try spread the PSTN line load between the two SPA400s to determine if that makes any difference.


The site has been running for several weeks. Have they had problems and not reported to you, or has all been well up until a few days ago?


Regards,



Patrick

-----------

j.lanzon Wed, 07/01/2009 - 15:05
User Badges:

Hi Patrick,


Yes site setup is correct, currently have 7 phones in use; 6 SPA941 and 1SPA942.


From the latest call they have been dealing with is the entire time, today has been the worst day where the system completely froze at one point and they have to reset the SPA9000. The issue of calls coming in with no one there has happened to me as well, they pick up but neither side can hear the other.


Will split the PSTN lines when I am there.


Joe

smbdatanetwork Fri, 12/18/2009 - 16:48
User Badges:

Hi Patrick or Joe,


Just wonder if you have resolve the SPA9000 issue, it is because I have just installed a SPA9000 with 2 SPA941 a few weeks ago at my customer's location. My customer experienced the same issue. Both SPA941 randomly reboot when receive calls, it does not happen all the time, only if there are many calls come in at the same time, what I suspect is the SPA9000 will send a reset to both SPA941 and the phone reboot. The worst is all the calls that is on hold will also drop as the phones reboot.


Thanks and hope you can help.


Paul

smbdatanetwork Fri, 12/18/2009 - 20:12
User Badges:

Hi Patrick,


Not sure if my last post is able to get to you, so I just post it again directly to you. I have the similar problem as Joe which both SPA941 reboot at the same time and they both are connected to a SPA9000. This only happens if there is a few calls come in at the same time and my customer is using ITSC with 2 call paths and no SPA400. If both lines on the phones (as they both are using Share Line Appearance) are being used and when the third or fouth calls come in, both SPA941 will reboot and it will also drop all the calls that are being on hold. The SPA9000 is using the latest firmware 6.1.5


If I can get some direction on this, it will be great.


Thanks


Paul

Patrick,

I have a customer suffering for the same circumstances.  What was the final corrective actions that solved this problem?  I have twleve telphones on the SPA9000.  10 SPA502s and two SP504s.  I have two SPA400s.  One is for voicemail.  The other for the 4 (four) analog trunks.  The SPA9000, SPA400s and the phones are running the latest firmware.


Thanks,

Bart Salng

smbdatanetwork Sun, 02/07/2010 - 20:34
User Badges:

Hi Bart,


I contacted the Cisco support, ITSC provider and I have also tried different setting, it is not able to fix the issue. My customer has two SPA 941 with Share LIne Appearance enable and the phones reboot and reinitialize during a conversation. The phones only reboot when there is a high volume of calls coming in, such as 5 or 6 calls at the same time, even though my customer subscribes only 2 lines from an ITSC. Based on my observation, it seems the SPA9000 has a limitation on the hardware to handle how many calls within a short period of time, such as 1 or 2 sec. I am going to replace the unit that has the problem (hardware version 1.05) with another SPA9000 that has hardware ver 2.05 and see if it can resolve the issue. Otherwise, I will need to take back the SPA 9000 and the SPA 941s and refund my customer, port their number back to the original telco, it will be tough.


Thanks


Paul

Actions

This Discussion