Problems with EX90 and MX300 devices dropping registration

Unanswered Question
Jul 24th, 2013
User Badges:

Hello


My company purhcased this TeleP solution. For the most part, it has worked fine. However, I have had several devices lose registration.

These devices were setup and workign well.

There have been no network configuration changes.


TAC case support wants me to console into the device, however the console cable is a special kind that I have been advised I need to make myself.


I did receive one of these cables from TAC and was advise d to download software. I did, but had no success. I "googled" this

software and found ALOT of people had problems with this console cable and software.


Troubleshooting steps I have taken are:


     I Have Restarted Device, no success

     I Have Power Cycled Device, no success

     I Have Replaced Ethernet Cable, no success

     I moved device To New Lan Jacks And Switch Ports, no success

     I Reset And Tried To Re-Provision, no success


So far, my only option has been to RMA the devices that stop registering


Questions I have are:

     What can I do to resolve this special console cable issue. I belive I can issue command to correct CODEC, once I gain console access


     Are there any other troubleshooting steps I should try?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Bryan Deaver Wed, 07/24/2013 - 10:01
User Badges:
  • Cisco Employee,

Hey Steve,


Normally for registration issues I would not recommend nor see the need for accessing the console.


You can find some information on the console cable that was likely sent to you and the drivers at this link:


https://supportforums.cisco.com/docs/DOC-27936


The cable will allow you access the codec even when the codec is not accessible on the network.  For example, in some instances where the codec will not boot up properly, the console cable can help to understand why and in many cases recover the codec.


This Cisco Live presentation covered some of the use and access for the console cable:


https://supportforums.cisco.com/docs/DOC-29766


Now, if the device is losing registration because it is crashing, then possible the console cable would be good but normally the system should come back up and in that case you would want to work with the tac to review the historical log files to determine why the codec crashes.  In later release of the software, the codec may go into "maintenance mode" when it crashes which provides choices of what you want to do to collect information and recover the device.  Yet another reference link for you for maintenance mode:


https://supportforums.cisco.com/docs/DOC-34116


Normally for registration issues, you would want to look more on the protocol being used.  For example SIP and determine when and why the device is losing registration. This may entail collecting more information such as SIP packet traces from the codec.  Check this troubleshooting guide and specifically look at the section on enabling SIP/H323 traces:


http://www.cisco.com/en/US/docs/telepresence/endpoint/codec-c-series/tc6/troubleshooting_guide/tc_troubleshooting_guide_tc60.pdf


It is unclear to me how RMAing the codec would relate to issues with it not being able to register unless there was something going on with the lower layers such as the NIC failing on the device.  Maybe you can unicast me the tac case number where the troubleshooting of a registration problem led to an RMA to get better context of this issue.


Hope this helps guide you to resolution of your issues.


Bryan

Vernon Depee Wed, 07/24/2013 - 10:04
User Badges:
  • Cisco Employee,

Steve,


Have you tried to access these devices via SSH? If they lose registration, but still have network connection, SSH should still be available if enabled.


When the devices unregister, do they still respond to pings?

Steve Coady Wed, 07/24/2013 - 10:37
User Badges:

Bryan


Thanks for the response and the links.


The RMA ghas been my only option because TAC could not help unless they had a log. The only way to get the log was via consoie cable.





Vernon


Thanks for the response. The devices cannot be pinged and are not available to ssh.


Again, there was no network change attributable to this. No configs were changed. Other devices on the same LAN still work fine!

Steve Coady Wed, 07/24/2013 - 10:39
User Badges:

aded info:


The devices are attached to an Cisco 3750 switch with routing enabled.

Bryan Deaver Wed, 07/24/2013 - 10:50
User Badges:
  • Cisco Employee,

Does show cdp neighbor detail from the 3750 show information about what the IP address on the codec thinks it is?  What does the touch panel on show you?


Yes, console access in this case could be helpful if there is no other access to the device.  I would guess that something changed possibly on the codec with regards to the vlan it is on but again if you are running later versions of TC software we should have codec ip address infomation which is sent back to the 3750.  Again, what does the touch show you when in this state as well as what do you observe on the EX90 screen?


If are you at the point in an issue where you are going to RMA the device, as a last step I would recommend that you try a factory reset; on the EX90 it is tricky but doable from the power button method.  Again check the troubleshooting guide reference.  I normally don't recommend a facory reset as a troubleshoot step (since for one it wipes out good diagnostic information from the device) but again if you are going to be doing an RMA anyway, then try it.


In what briefly I am hearing from you though, there should not be a need for an RMA here if this is an issue that occurs more than once.

Steve Coady Wed, 07/24/2013 - 11:00
User Badges:

Bryan


Both the switch and the Touch screen show the same ip info.


Show cdp ne det


Device ID: SEP00506008580A

Entry address(es):

IP address: 10.x.x.30

Platform: CTS-CODEC-MX200/MX300, Capabilities: Host Phone

Interface: GigabitEthernet6/0/24, Port ID (outgoing port): eth0

Holdtime : 170 sec


Version :

TC6.0.1 65adebe


advertisement version: 2

Management address(es):


PP3750S1#ping 10.x.x.30

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.x.x.30, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)


The Vlan is /24

Bryan Deaver Wed, 07/24/2013 - 14:12
User Badges:
  • Cisco Employee,

Good that something is working; we see CDP packets coming in.  The codec is using DHCP and does get this address from the server?  This is not using static address, correct?


The address and vlan information you see from the touch panel settings menu reflects the reality of what is configured on the switch side?


As you said, restarting the EX90 from the touch panel does not cause it to come back up, yes?


There are a few things that can be checked from the codec via console connection.  Would be good to get the output of xstatus, xconfig, the /var/log/console file as well as ifconfig and ethtool -S eth0.


Nothing off on the switch side when the codec is in this state from the show interface stats?  Are you using port security on the switch?


The same issue has happened on more than one of the codecs then?  Do you have other codecs besides EX/MX systems?

Steve Coady Wed, 07/24/2013 - 15:06
User Badges:

Bryan


The device does have a static ip.


The switch interface is configured as follows:


interface GigabitEthernet6/0/24

description VIDEO MX300 LrgConf Alternate

switchport access vlan xxx

switchport mode access

srr-queue bandwidth share 10 10 60 20

queue-set 2

priority-queue out

mls qos trust dscp

auto qos voip trust

spanning-tree portfast

end



Retarting from Touch panel does not fix the issue


No port security

Bryan Deaver Thu, 07/25/2013 - 09:51
User Badges:
  • Cisco Employee,

Again, since we are showing CDP information on the switch, this means that the CDP multicast packets are coming in to the switch.


No guarantee that the codec is receiving packets though.  If you do a 'show arp | include ' from the switch does it show that it was able to resolve the address?


Again, nothing in the show interface for that switch port that looks abnormal?


Thinking at this point what a packet capture while spanning that port would show.  But before going doing that path, again getting access to the codec while in this state (again through the console port) will be a good step if there is nothing else that we can determine through the touch panel nor the switch side.  At that point though, I am going to recommend that you take the Q/A from this thread and open up a TAC case to have a CSE interactively troubleshoot this with you.

Steve Coady Thu, 07/25/2013 - 11:14
User Badges:

If you do a 'show arp | include ' from the  switch does it show that it was able to resolve the address?

     router#sh ip arp | inc 10.x.x.30


     router#sh mac add add 0050.6008.580a

          Mac Address Table

     -------------------------------------------

     Vlan    Mac Address       Type        Ports

     ----    -----------       --------    -----

     router#



     switch1# sh mac add int gi6/0/24

          Mac Address Table

     -------------------------------------------

     Vlan    Mac Address       Type        Ports

     ----    -----------       --------    -----

      x    0050.6008.580a    DYNAMIC     Gi6/0/24

     Total Mac Addresses for this criterion: 1

     switch1#


I am including this to show that ALL other devices on same vlan, several on same switch, others on different switches ARE being resolved and are working fine.


router#sh ip arp | inc 10.x.x


Internet  10.x.x.15            233   d867.d971.9113  ARPA   Vlanx

Internet  10.x.x.12              2   d867.d971.9138  ARPA   Vlanxxx

Internet  10.x.x.13            233   d867.d971.913c  ARPA   Vlanxxx

Internet  10.x.x.10            228   d867.d971.9446  ARPA   Vlanxxx

Internet  10.x.x.11            101   d867.d971.9428  ARPA   Vlanxxx

Internet  10.x.x.6               -   3cdf.1e8a.9f4c  ARPA   Vlanxxx

Internet  10.x.x.7             231   3cdf.1efc.9e42  ARPA   Vlanxxx

Internet  10.x.x.1             161   3037.a6ab.bdd1  ARPA   Vlanxxx

Internet  10.x.x.28            233   0050.6008.5803  ARPA   Vlanxxx

Internet  10.x.x.29             62   0050.6008.556a  ARPA   Vlanxxx

Internet  10.x.x.24             57   0050.6006.3c47  ARPA   Vlanxxx

Internet  10.x.x.23            230   d867.d971.988a  ARPA   Vlanxxx

Internet  10.x.x.20             26   0050.6008.5837  ARPA   Vlanxxx

Internet  10.x.x.21            233   d867.d971.98ac  ARPA   Vlanxxx

router#


Again, nothing in the show interface for that switch port that looks abnormal?


GigabitEthernet6/0/24 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is dc7b.940c.0498 (bia dc7b.940c.0498)

  Description: VIDEO MX300 LrgConf Alternate

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:17, output hang never

  Last clearing of "show interface" counters 14w1d

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 1000 bits/sec, 1 packets/sec

     125852 packets input, 13244462 bytes, 0 no buffer

     Received 125852 broadcasts (61410 multicasts)

     3 runts, 0 giants, 0 throttles

     3 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 61410 multicast, 0 pause input

     0 input packets with dribble condition detected

     2855367 packets output, 230517120 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out


There are (3) runts and input errors in last 14 weeks.


I do appreciate you assistance.

Bryan Deaver Thu, 07/25/2013 - 12:26
User Badges:
  • Cisco Employee,

OK, so a guess here is that the codec is not receiving the frames from the switchport.  Don't see it in the arp table; and I would assume that if you ping the address and then check the arp table you would see an incomplete entry for the ip address.


One off case, I could see issues possibly with the nic on the codec.  Happening more than once for you seems odd to me.  We don't currently have a way on the touch panel to see the ethernet stats for the codec so to dig further, it would be some of the information I had indicated before which once in this state would be the console connection.


How often have you see this issue with the same symptoms? 


Again, since you seem to have one in this state currently, I would recommend getting a case open on this so we can track it.  There is the factory reset option from the touch panel (again that would be my last resort option) but from what you seem to be seeing, I don't hold high hopes that it will make a difference.  You can ping my privately with the case number so I can track what happens on it.

Steve Coady Thu, 07/25/2013 - 14:39
User Badges:

router #ping 10.x.x.30

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.x.x.30, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)


router#sh ip arp | inc 10.x.x.30

Internet  10.x.x.30              0   Incomplete      ARPA


How often have you see this issue with the same symptoms?

     We implemented this solution last summer/fall. Since that time (4) devices have had this same problem.

     Several at my local site and 1 in Chicago site.


     A couple devices experienced this within a couple weeks of eachother.


I have already done a factory rest on the device in questuion, no success

Actions

This Discussion