Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Community Member

Cisco Phones and 3750V keep losing trust

I am working on a project to upgrade the phones at a location.  They phones are all plugged into a stack of 3750X/V all doing POE.  We recently upgraded the phones to 7942s and 8941s and the 3750s are running Universal on the Xs (masters) and IPbase on the Vs with 12.2(55)SE.  Since we put in the new phones they are able to connect, pull configs, make calls, but then for some reason they keep randomly dropping the voice for around a minute, then come back up.  The logs are not very help, the port never goes down, never goes into an error state, and only shows the following logs regarding the ports.

Jul 29 07:02:47.005 EDT: %SWITCH_QOS_TB-5-TRUST_DEVICE_LOST: cisco-phone no longer detected on port Fa3/0/13, operational port trust state is now untrusted.

Jul 29 07:03:29.222 EDT: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Fa3/0/13, port's configured trust state is now operational.

All the phones from what I can tell are having this same issue.

Firmware versions below:

6941       SCCP69xx.9-3-1-3

7942       SCCP42.9-3-1SR1-1S

8941       SCCP8941_8945.9-3-2-11


CDP timers are default and when I do a show CDP neighbors everything is working fine and up.

This is the physical interface setup, no errors, and its happening across mulitple phones.  It doesnt happen at the same time though, all the phones seem to be random.  The issues occurs on different switches in the stack.  I have other switches at this location with the same switch stack configuration that arent having any problems at all, with the same model and OS of phones attached.  Its isolated to a single switch stack.

interface FastEthernet3/0/13

switchport access vlan 760

switchport mode access

switchport voice vlan 769

switchport port-security maximum 3

srr-queue bandwidth share 10 10 60 20

priority-queue out

mls qos trust device cisco-phone

mls qos trust cos

auto qos voip cisco-phone

spanning-tree portfast

spanning-tree bpduguard disable

service-policy input AutoQoS-Police-CiscoPhone

end

FastEthernet3/0/13 is up, line protocol is up (connected)

  Hardware is Fast Ethernet, address is 18ef.6341.1f8f (bia 18ef.6341.1f8f)

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:31, output 00:00:12, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 248

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 1000 bits/sec, 1 packets/sec

  5 minute output rate 1000 bits/sec, 1 packets/sec

     7726801 packets input, 1340717194 bytes, 0 no buffer

     Received 72550 broadcasts (54632 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 54632 multicast, 0 pause input

     0 input packets with dribble condition detected

     14037571 packets output, 12173010547 bytes, 0 underruns

     0 output errors, 0 collisions, 1 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

Something I did notice is that when the issue occurs, the phones send out a DHCP request, that DHCP request is being responded to by the data vlan, not the voice vlan.  So I know that the switch is not tagging the traffic from the Phone with the voice vlan.

Here is the output from the log file of the phone, as you can see the first time I tries to pull a DHCP address from the data vlan, gets an IP, then releases it and requests a new one.

Jul 29 08:39:38 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4

DHCPOFFER from 10.75.180.1

DHCPREQUEST on eth0 to 255.255.255.255 port 67 interval 4

Jul 29 08:39:41 dhclient: DHCPOFFER from 10.75.180.1

Jul 29 08:39:41 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 interval 4

DHCPACK from 10.75.180.1

ARPING to 10.75.180.192 from 10.75.180.192 via eth0

Jul 29 08:39:41 dhclient: DHCPACK from 10.75.180.1

Sent 1 probe(s) (1 broadcast(s))

Received 0 reply (0 request(s), 0 broadcast(s))

bound to 10.75.180.192 -- renewal in 215999 seconds.

*****ASKEY: Our IP = [0x 0a 4b b4 c0]

08:39:42, 07/29/2013 : SEP20bbc0dff80a Get ip from DHCP server : 10.75.180.192

*****ASKEY: Our IP = [0x 0a 4b b4 c0]

Jul 29 08:39:42 dhclient: bound to 10.75.180.192 -- renewal in 215999 seconds.

route: SIOCDELRT: No such process

route del default gw 10.75.210.1 dev eth0 fail

--> dot1x_activate(468) :

EW: ew_client_task_create fail, either domain is null or invalid!

provision_task_delete_and_recreate(), recreate= 1

***********************

* Rate limit enabled *

***********************

08:39:44, 07/29/2013 : CTLSEP20BBC0DFF80A.tlv updating

08:39:44, 07/29/2013 : CTLSEP20BBC0DFF80A.tlv not found in 10.0.30.11

08:39:44, 07/29/2013 : ITLSEP20BBC0DFF80A.tlv updating

08:39:44, 07/29/2013 : ITLSEP20BBC0DFF80A.tlv (HTTP)

08:39:44, 07/29/2013 : ITLSEP20BBC0DFF80A.tlv updated successfully

08:39:44, 07/29/2013 : ITL installed

08:39:44, 07/29/2013 : SEP20BBC0DFF80A.cnf.xml.sgn (HTTP)

Authenticating configuration file SEP20BBC0DFF80A.cnf.xml.sgn successfully

--> dot1x_activate(468) :

Authenticating SCCP8941_8945.9-3-2-11.loads file successfully.

###############################################################

BOOT8941_8945.0-0-1-0.bin.sgn

SCCP8941_8945.9-3-2-11.bin1.sgn

SCCP8941_8945.9-3-2-11.bin2.sgn

SCCP8941_8945.9-3-2-11.bin3.sgn

SCCP8941_8945.9-3-2-11.bin4.sgn

SCCP8941_8945.9-3-2-11.bin5.sgn

SCCP8941_8945.9-3-2-11.bin6.sgn

SCCP8941_8945.9-3-2-11.bin7.sgn

SCCP8941_8945.9-3-2-11.bin8.sgn

SCCP8941_8945.9-3-2-11.bin9.sgn

###############################################################

cfgmgr_loads_task(): image_flag: 9

*** cfgmgr_loads_task 1826 callback return ***

No need to download locale resources. (English_United_States, 9.0.0.0(1), 10.0.30.11/Engl

sh_United_States, 9.0.0.0(1), 10.0.30.11)

08:39:47, 07/29/2013 : VPN Error: VPN is not Configured.

into subhandler_applications:provision_ready

ammu_get_cisco_sccp_ep_prof() video_capability=1

ammu_get_cisco_sccp_ep_prof() rtcp_enable=0

ammu_get_cisco_sccp_ep_prof() video_rtcp_enable=1

ammu_get_cisco_sccp_ep_prof() srtcp_auth_sha1_80=0

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone=0

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone_duration=65535

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone_local_vol=100

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone_rmt_vol=50

ammu_get_cisco_sccp_ep_prof() MissedCallLoggingOption=1100

(12)25

(12)6

(12)4

(12)2

(12)7

(12)8

(12)86

(12)15

(12)16

(12)11

(12)12

(12)257

(1)103

ffs_readfile 96 fd == ERROR (/CFG/ealarm.data)

accu_sccp_guard_term_spy_register_ok() x_miss_call_dn[1]=7204 , btn_dn=7204

accu_sccp_guard_term_spy_register_ok() fk_idx[1] keep the same

accu_sccp_guard_term_spy_register_ok() x_miss_call_dn[2]=7202 , btn_dn=7202

accu_sccp_guard_term_spy_register_ok() fk_idx[2] keep the same

accu_sccp_guard_term_spy_register_ok() fk_idx[3] is reset(dn empty).

accu_sccp_guard_term_spy_register_ok() fk_idx[4] is reset(dn empty).

accu_sccp_guard_term_spy_register_ok() fk_idx[5] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[6] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[7] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[8] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[9] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[10] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[11] is reset.

--> dot1x_activate(468) :

--> dot1x_activate(468) :

--> dot1x_activate(468) :

03:17:11:14.532 - AMMU: Checking Protocol: Device=SCCP. Config=SCCP (OK)

03:17:11:14.556 - ACCU: accu_sccp_ep_proc_tstone_config_resp_ok().

03:17:11:14.722 - ACCU: SCCP REG OK(-1, -1)

03:17:11:14.734 - ACCU: SCCP REG OK(0, -1)

03:17:11:14.754 - ACCU: SCCP REG OK(0, 1)

[MMI]Not update call ui on unregister

[MMI]Not update call ui on unregister

[MMI]Not update call ui on unregister

[MMI]Not update call ui on unregister

[MMI]Not update call ui on unregister

[MMI]Not update call ui on unregister

path = /nvdata is exist

FFS Commit to /nvdata/CFG1.

path = /nvdata is exist

FFS Commit to /nvdata/CFG2.

03:17:13:18.778 - ACCU: accu_mmi_cmd_unconfigure_ep(). epid=1. reason_uncfg=0

tstone_sessionstatus() cm_down act=-1 bak=-1

--> dot1x_activate(468) :

############# KILL /voip/dhclient eth0 -r ############# 4

Internet Systems Consortium DHCP Client V3.1.3

Copyright 2004-2009 Internet Systems Consortium.

All rights reserved.

For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/20:bb:c0:df:f8:0a

Sending on LPF/eth0/20:bb:c0:df:f8:0a

Sending on Socket/fallback

DHCPRELEASE on eth0 to 10.0.10.163 port 67

03:17:13:18.852 - ACCU: SCCP REG OK(0, -1)

03:17:13:18.862 - ACCU: SCCP REG FAIL(-1, -1)

03:17:13:18.874 - ACCU: accu_tstone_cmd_unconfig_resp(). ep_ref=0x4b6ffdf0. unconfig_ok=1

03:17:13:19.468 - AMMU: ammu_proc_cfgmgr_activate(). cfgmgr_pend_evt=unreg_ready. ignore

his message

Jul 29 08:41:52 dhclient: Internet Systems Consortium DHCP Client V3.1.3

Jul 29 08:41:52 dhclient: Copyright 2004-2009 Internet Systems Consortium.

Jul 29 08:41:52 dhclient: All rights reserved.

Jul 29 08:41:52 dhclient: For info, please visit https://www.isc.org/software/dhcp/

Jul 29 08:41:52 dhclient:

Jul 29 08:41:52 dhclient: Listening on LPF/eth0/20:bb:c0:df:f8:0a

Jul 29 08:41:52 dhclient: Sending on LPF/eth0/20:bb:c0:df:f8:0a

Jul 29 08:41:52 dhclient: Sending on Socket/fallback

Jul 29 08:41:52 dhclient: DHCPRELEASE on eth0 to 10.0.10.163 port 67

route: SIOCDELRT: No such process

route del default fail

route: SIOCDELRT: No such process

route del default fail

[0x30400000] 0xa1

[0x30400008] 0xa1

set_vlan_id vlan 1.1

[0x3041a030] 0x1d

[0x30400000] 0xa0

[0x30400008] 0xa0

Internet Systems Consortium DHCP Client V3.1.3

Copyright 2004-2009 Internet Systems Consortium.

All rights reserved.

For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/20:bb:c0:df:f8:0a

Sending on LPF/eth0/20:bb:c0:df:f8:0a

Sending on Socket/fallback

***ASKEY: dhclient.c, state_reboot(), Line714 DO initiation directly...

DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4

provision_task_delete_and_recreate(), recreate= 0

provision_task_delete_and_recreate(), recreate= 0

Jul 29 08:41:55 dhclient: Internet Systems Consortium DHCP Client V3.1.3

Jul 29 08:41:55 dhclient: Copyright 2004-2009 Internet Systems Consortium.

Jul 29 08:41:55 dhclient: All rights reserved.

Jul 29 08:41:55 dhclient: For info, please visit https://www.isc.org/software/dhcp/

Jul 29 08:41:55 dhclient:

Jul 29 08:41:56 dhclient: Listening on LPF/eth0/20:bb:c0:df:f8:0a

Jul 29 08:41:56 dhclient: Sending on LPF/eth0/20:bb:c0:df:f8:0a

Jul 29 08:41:56 dhclient: Sending on Socket/fallback

Jul 29 08:41:56 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4

DHCPOFFER from 10.75.210.5

DHCPREQUEST on eth0 to 255.255.255.255 port 67 interval 4

Jul 29 08:41:58 dhclient: DHCPOFFER from 10.75.210.5

Jul 29 08:41:58 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 interval 4

DHCPACK from 10.75.210.5

ARPING to 10.75.210.182 from 10.75.210.182 via eth0

Jul 29 08:41:58 dhclient: DHCPACK from 10.75.210.5

Sent 1 probe(s) (1 broadcast(s))

Received 0 reply (0 request(s), 0 broadcast(s))

bound to 10.75.210.182 -- renewal in 43199 seconds.

*****ASKEY: Our IP = [0x 0a 4b d2 b6]

08:41:59, 07/29/2013 : SEP20bbc0dff80a Get ip from DHCP server : 10.75.210.182

*****ASKEY: Our IP = [0x 0a 4b d2 b6]

Jul 29 08:41:59 dhclient: bound to 10.75.210.182 -- renewal in 43199 seconds.

route: SIOCDELRT: No such process

route del default gw 10.75.180.1 dev eth0 fail

--> dot1x_activate(468) :

EW: ew_client_task_create fail, either domain is null or invalid!

provision_task_delete_and_recreate(), recreate= 1

***********************

* Rate limit enabled *

***********************

08:42:01, 07/29/2013 : CTLSEP20BBC0DFF80A.tlv updating

08:42:01, 07/29/2013 : CTLSEP20BBC0DFF80A.tlv not found in 10.0.30.11

08:42:01, 07/29/2013 : ITLSEP20BBC0DFF80A.tlv updating

08:42:01, 07/29/2013 : ITLSEP20BBC0DFF80A.tlv (HTTP)

08:42:01, 07/29/2013 : ITLSEP20BBC0DFF80A.tlv updated successfully

08:42:01, 07/29/2013 : ITL installed

08:42:01, 07/29/2013 : SEP20BBC0DFF80A.cnf.xml.sgn (HTTP)

Authenticating configuration file SEP20BBC0DFF80A.cnf.xml.sgn successfully

--> dot1x_activate(468) :

Authenticating SCCP8941_8945.9-3-2-11.loads file successfully.

###############################################################

BOOT8941_8945.0-0-1-0.bin.sgn

SCCP8941_8945.9-3-2-11.bin1.sgn

SCCP8941_8945.9-3-2-11.bin2.sgn

SCCP8941_8945.9-3-2-11.bin3.sgn

SCCP8941_8945.9-3-2-11.bin4.sgn

SCCP8941_8945.9-3-2-11.bin5.sgn

SCCP8941_8945.9-3-2-11.bin6.sgn

SCCP8941_8945.9-3-2-11.bin7.sgn

SCCP8941_8945.9-3-2-11.bin8.sgn

SCCP8941_8945.9-3-2-11.bin9.sgn

###############################################################

cfgmgr_loads_task(): image_flag: 9

*** cfgmgr_loads_task 1826 callback return ***

No need to download locale resources. (English_United_States, 9.0.0.0(1), 10.0.30.11/Engl

sh_United_States, 9.0.0.0(1), 10.0.30.11)

08:42:04, 07/29/2013 : VPN Error: VPN is not Configured.

into subhandler_applications:provision_ready

ammu_get_cisco_sccp_ep_prof() video_capability=1

ammu_get_cisco_sccp_ep_prof() rtcp_enable=0

ammu_get_cisco_sccp_ep_prof() video_rtcp_enable=1

ammu_get_cisco_sccp_ep_prof() srtcp_auth_sha1_80=0

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone=0

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone_duration=65535

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone_local_vol=100

ammu_get_cisco_sccp_ep_prof() vdcfg_recording_tone_rmt_vol=50

ammu_get_cisco_sccp_ep_prof() MissedCallLoggingOption=1100

(12)25

(12)6

(12)4

(12)2

(12)7

(12)8

(12)86

(12)15

(12)16

(12)11

(12)12

(12)257

(1)103

ffs_readfile 96 fd == ERROR (/CFG/ealarm.data)

accu_sccp_guard_term_spy_register_ok() x_miss_call_dn[1]=7204 , btn_dn=7204

accu_sccp_guard_term_spy_register_ok() fk_idx[1] keep the same

accu_sccp_guard_term_spy_register_ok() x_miss_call_dn[2]=7202 , btn_dn=7202

accu_sccp_guard_term_spy_register_ok() fk_idx[2] keep the same

accu_sccp_guard_term_spy_register_ok() fk_idx[3] is reset(dn empty).

accu_sccp_guard_term_spy_register_ok() fk_idx[4] is reset(dn empty).

accu_sccp_guard_term_spy_register_ok() fk_idx[5] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[6] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[7] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[8] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[9] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[10] is reset.

accu_sccp_guard_term_spy_register_ok() fk_idx[11] is reset.

--> dot1x_activate(468) :

--> dot1x_activate(468) :

--> dot1x_activate(468) :

03:17:13:32.974 - AMMU: Checking Protocol: Device=SCCP. Config=SCCP (OK)

03:17:13:32.998 - ACCU: accu_sccp_ep_proc_tstone_config_resp_ok().

03:17:13:33.158 - ACCU: SCCP REG OK(-1, -1)

03:17:13:33.164 - ACCU: SCCP REG OK(0, -1)

03:17:13:33.200 - ACCU: SCCP REG OK(0, 1)

[MMI]Not update call ui on unregister

[MMI]Not update call ui on unregister

path = /nvdata is exist

FFS Commit to /nvdata/CFG1.

path = /nvdata is exist

FFS Commit to /nvdata/CFG2.

I hope this enough, Im not a voice guy, so please let me know if you need more information.  Thanks for taking a look.

Everyone's tags (4)
9 REPLIES
Silver

Cisco Phones and 3750V keep losing trust

How reproducible is this?  Does it happen on certain intervals such as every 60 minutes or anything like that?  Can you run a "show cdp neighbors" while it's in the broken state and see if the phone is still listed?  What's the experience on the phone?  Does it reboot?  Does it unregister?

Community Member

Re: Cisco Phones and 3750V keep losing trust

Its happening constantly,  i cant tell the exact time period as the log file on the switch isnt large enough to keep two consecutive drops.  I cant change the log file yet as I am a contractor and the employeer hasnt given me the abilities to do so on the switch.  The experience on the phone is that the call is dropped, I have had mixed reports of reboots, the device never loses power however.  It does unregister.

008955: Jul 18 08:31:57.440: %IPPHONE-6-UNREGISTER_NORMAL: ephone-80:SEP18339D15DD36 IP:10.75.210.154 Socket:51 DeviceType:Phone has unregistered normally.

008957: Jul 18 08:32:04.609: %IPPHONE-6-REGISTER_NEW: ephone-49:SEP10BD180038A9 IP:10.75.210.151 Socket:18 DeviceType:Phone has registered.

Actually about once an hour

Jul 30 09:05:38.187 EDT: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Fa4/0/37, port's configured trust state is now operational.

Jul 30 09:54:07.714 EDT: %SWITCH_QOS_TB-5-TRUST_DEVICE_LOST: cisco-phone no longer detected on port Fa4/0/37, operational port trust state is now untrusted.

Jul 30 09:54:38.356 EDT: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Fa4/0/37, port's configured trust state is now operational.

Silver

Re: Cisco Phones and 3750V keep losing trust

Are there any core dumps on the phone's webpage after this happens?

Community Member

Re: Cisco Phones and 3750V keep losing trust

Ill check and let you know.  While looking into that I just found something really odd.  When I did a show mac address table for the voice vlan, at first no mac addresses were showing up in that vlan on this switch.  I thought that was really odd, so i kept entering the command to see if that would change, within a span of 5 seconds I went from no switch ports on the switch showing mac address in vlan 769, to a bunch of them all at once.

Silver

Re: Cisco Phones and 3750V keep losing trust

That's strange.  Is it just phones on that one switch that's having problems?

Community Member

Re: Cisco Phones and 3750V keep losing trust

That is correct, just the phones.

Community Member

Hi - Did you ever get to the

Hi - Did you ever get to the bottom of this issue?? I'm experiencing exactly the same problem on a stack of 6 x 3750v2 switches and suspect it's related to this bug ->  https://tools.cisco.com/quickview/bug/CSCso40282

Before I make a start upgrading all our 3750's (300+) I wonder if you managed to fix it?

Community Member

Hi, Did you end up upgrading

Hi, Did you end up upgrading all your switches. We are experiencing the same issue.

Community Member

Hi Rajat - Yes, we upgraded

Hi Rajat - Yes, we upgraded the switches but that made no difference and the problem kept occurring.

As the problem was so disruptive and the users were losing patience with us troubleshooting we took the decision to replace all the affected 79xx series phones  with 7821 phones. This seems to of solved the problem but we've never had a satisfactory explanation as to what the problem was.

Sorry I can't be any more help than that.

1959
Views
0
Helpful
9
Replies
CreatePlease to create content