False and Delayed SNMP Cold Start Traps

[GERARD PUOPLO]

                   I seen some behavior this past week that  I never seen before and am curious about it.  I am seeing SNMP coldstart traps that either are delayed by many hours or are false (e.g. right after receiving the coldstart trap a query to sysUptime shows the nodes been up for days).

              I seen this twice this week in a new network environment for me for two different C2900s running C2900-UNIVERSALK9-M Version 15.0(1)M3 Assuming the coldstart traps are coming from the actual source nodes,   I am curious what could be going on here.

1) One guess I have is possibly the  system clock changed could cause the SNMP agent to send a false cold start trap.  Anyone know if  that is true ?   Then my guess is in the device log I should see a system time change syslog message.

2) I recall hearing once that syslog and possible traps messages are held in configurable buffer who default value is 1 and if not sent are held and then suffer a delayed sent.    Anyone know it true for both traps and syslog ?   In the past I assumed this was simply the logging history buffer  and applicable to syslog traps only.  My assumption in the past was that last trap or last syslog message is sometimes held on reload and sent immediately after restart regardless of device connectivity to the management target.  

So in the past   I always assumed coldstart traps are never delayed for any reason and that they were pretty accurate substitutes  for system reload syslog messages.

Does anyknow know any reason for false or delayed coldstart traps on a C2900  with IOS 15.0(1) ?

Thx

[yjdabear · Accepted solution]

Is it possible for you to verify your theories in a lab environment with one or more C2900 and another model, both running IOS 15.0(1), given coldStart traps could be triggered relatively easily?

I have never seen coldStart trap delays (by itself) due to IOS bugs, but I did see several instances of every SNMP trap being delayed for longer and longer due to bug(s) in the pmd process of HPOV NNM. So I'm curious what your trap receiver is and if you've ruled out that as the culprit (which should be rather straight forward if you're seeing delays with any other non-coldStart traps, except when coldStart traps happen to be the only ones there was in your environment at the time.)

[smillerv2]

Hi Gerald,

What ended up causing your delayed alarms? We have received a reload alarm and then found the router has been up for a couple of days. I searched the Internet for similar occurances and found this discussion. We have eliminated the HPOV NNM as the cause by looking at packets coming into the NNM server from a continous trace. We see the SNMP traps coming into the NNM server at the same time the alarm showed up in HPOV. Also, all other traps are showing up from other devices without delay. So something delayed the trap from being sent from this router to the NNM server, or it was a false alarm.