7912 cause for broadcast storm?

mesaverde · ‎01-14-2008

We have 112 Cisco phones on a CCM 3.3 Publisher and Subscriber here at Mesa Verde National Park. It has been installed since 2004 with very few VoIP related problems experienced since then. About 65 of these phones are 7912 models. We started experiencing random intermittent broadcast storms on our network last July and have not been able to trace their source. Just last week, we discovered two 7912 phones that would consistently not register with CCM. They appeared to be locked up. We discovered this because we were trying to chase a broadcast storm problem that came up when a CAT 3550 was replaced because of failure.

What I am looking for is some feedback regarding 7912 intermittent problems and failures that would cause a network storm. Is this common, not so common or does anyone just think I am crazy?

Thanks - Flint Boardman - 970-529-5060

Rob Huffman · ‎01-14-2008

Hi Flint,

You are not crazy! We have been hit by this 7912 Bug once before that caused an Arp Broadcast Storm :(

CSCee02798 Bug Details

7912 loops in ARP when duplicate IP and using DHCP.

Symptoms:

When two 7912 using DHCP to obtain their IP and somehow

got the same IP due to DHCP server internal state error,

7912 software will detect the duplicate IP condition and

attempt to recover from the situation.

We saw some potential race condition in duplicate IP address

recovery code prior to version 1.3 software that may cause

two 7912 to response to each other duplicate IP response

(which is sending out ARP of its existence on the network)

indefinitely. This condition may only be triggered by duplicate

IP in two or more 7912 that use DHCP to obtain their IPs. This

condition if occured can create ARP broadcast storm.

Workaround:

No workaround.

Fix:

The fix is in CCO image cmterm-7912g-sccp.3-3-8.exe (firmware image

with version 1.3.0 and build date 040312A).

Hope this helps!

Rob

mesaverde · ‎01-14-2008

Thanks Rob! Will the mentioned image file for the 7912 work with CCM 3.3? I only see references in a quick search regarding CCM 4.0.

mesaverde · ‎01-14-2008

Another question Rob. How did you determine it was duplicate IP problem? Our phones do get IP from DHCP and the chance of duplicate IP address issued is probably not that high right? OK, more than 1 question I reckon...

These two phones consistently start the broadcast storm as soon as they "lock up" trying to register.

Rob Huffman · ‎01-14-2008

Hi Flint,

The Firmware mentioned is still very old. You should be good to go with a new one like 8.0(4);

http://www.cisco.com/en/US/docs/voice_ip_comm/cuipph/7905g_7912g/firmware/8_0_4/english/release/notes/7912_804MR.html

When we discovered this problem we were in the middle of a CCM Upgrade that was not going well. This issue was identified via an Ethereal Trace I'm pretty sure. It's been awhile :)

Hope this helps!

Rob

mesaverde · ‎01-14-2008

OK, I bet that was fun... =8-)

Will get the new one and see what we see.

Thanks so much for your responses!

Rob Huffman · ‎01-14-2008

Hi Flint,

That Night/Day was one that convinced me to always read all associated Bugs and Known/Resolved Caveats when doing an upgrade (my bad!) But, we do "live and learn"!

When doing the Phone Firmware Upgrade, rather than using the .exe we usually use the method listed below so we can test the Firmware on a few phones before the full deployment.

Cisco IP Phone Model 7912 Alternate Firmware format

The Cisco IP Phone Model 7912 Firmware version 8.0(4) is also supplied in a zip file, as an alternative in case a Cisco CallManager is not available to run the executable installer program. The name of this file is cmterm-7912G-sccp.8-0-4.zip.

After unzipping the firmware files, manually copy the firmware to the appropriate directory on the TFTP server. Then apply the firmware to the select "test" phones via the Device>Phone>Firmware Load Information

Firmware Load Information (leave blank to use default)

Hope this helps! Good luck!

Rob

mesaverde · ‎01-14-2008

I hear you Rob. =8-) (what instructions?)

I studied it a bit and figured the zip file was the simplest way to go and manually copied it. Then tried a few phones, they seemed good, then reset them all. Now we will see.

Thanks Again!

paolo bevilacqua · ‎01-14-2008

Local Directory for CME fixed (CSCsl96231) !

Whoa, way to go cisco :)

And 5 points to Rob for being brave and trying it first!