HUM not collecting 3750 interface TX Stats

Answered Question
May 1st, 2009
User Badges:

When HUM is collecting stats from our 3750 switches it seems to be missing the interface TX utlisation.


All other stats (RX Util, errors, inventory) etc seem to work fine.


The switches are running

Version 12.2(35)SE5


Has anyone come accross this before and if so, any idea's how to fix.


I'm guessing IoS upgrade but just wondered if anyone else has come across this before I start to plan upgrading 120 or switches..


Thanks.

Correct Answer by Joe Clarke about 8 years 2 weeks ago

The 64-bit counter support was the main feature in the HUM 1.X.2 releases. The way you were describing the problem, I had thought you couldn't see these instances when DEFINING the poller. It wasn't clear that the missing instances were only in if*Octets in HistoGraphIt. Good, I'm glad it's working now.


These two daemons are typically down in normal operation. You should not try to start them manually. LMS is working normally.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Joe Clarke Fri, 05/01/2009 - 08:06
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

How have you configured your pollers? What version of HUM are you using?

BlueyVIII Fri, 05/01/2009 - 08:53
User Badges:

Thanks for the quick reply.


Our version of HUM is 1.1.0


The pollers are configured to poll all our 3750 every 5 minutes using the following default templates.


CPU Util

Device Availability

Interface Availiability

Interface Errors

Interface Utilisation

Memory Utilisation


Thanks for any help you can offer..

Joe Clarke Fri, 05/01/2009 - 09:01
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Check the ifOutOctets objects of the various interfaces, and make sure they are incrementing on the devices.

BlueyVIII Fri, 05/01/2009 - 09:10
User Badges:

Thanks......can you explain the best way to do this please.


I should also mention that HUM is collecting TX and RX utilisation stat from out 3550's, 4507's and 6509's with no problems..

Joe Clarke Fri, 05/01/2009 - 09:40
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

You can use the SNMP Walk tool in Device Center to get the ifOutOctets values to see if they are incrementing.

BlueyVIII Fri, 05/01/2009 - 10:00
User Badges:

Did the SNMP Walk and got the following :-


RFC1213-MIB::ifOutOctets.1 = Counter32: 0

RFC1213-MIB::ifOutOctets.10 = Counter32: 22232125

RFC1213-MIB::ifOutOctets.20 = Counter32: 50218494

RFC1213-MIB::ifOutOctets.30 = Counter32: 0

RFC1213-MIB::ifOutOctets.5179 = Counter32: 0

RFC1213-MIB::ifOutOctets.5182 = Counter32: 0

RFC1213-MIB::ifOutOctets.10001 = Counter32: 1668903728

RFC1213-MIB::ifOutOctets.10002 = Counter32: 1095919562

RFC1213-MIB::ifOutOctets.10003 = Counter32: 2075426549

RFC1213-MIB::ifOutOctets.10004 = Counter32: 1196299431

RFC1213-MIB::ifOutOctets.10005 = Counter32: 1302447247

RFC1213-MIB::ifOutOctets.10006 = Counter32: 2076502706

RFC1213-MIB::ifOutOctets.10007 = Counter32: 881610392

RFC1213-MIB::ifOutOctets.10008 = Counter32: 1458769273

RFC1213-MIB::ifOutOctets.10009 = Counter32: 0

RFC1213-MIB::ifOutOctets.10010 = Counter32: 0

RFC1213-MIB::ifOutOctets.10011 = Counter32: 1266446122

RFC1213-MIB::ifOutOctets.10012 = Counter32: 1386798059

RFC1213-MIB::ifOutOctets.10013 = Counter32: 1330969697

RFC1213-MIB::ifOutOctets.10014 = Counter32: 662143937

RFC1213-MIB::ifOutOctets.10015 = Counter32: 1423234403

RFC1213-MIB::ifOutOctets.10016 = Counter32: 565418507

RFC1213-MIB::ifOutOctets.10017 = Counter32: 1826960764

RFC1213-MIB::ifOutOctets.10018 = Counter32: 1361161655

RFC1213-MIB::ifOutOctets.10019 = Counter32: 0

RFC1213-MIB::ifOutOctets.10020 = Counter32: 0

RFC1213-MIB::ifOutOctets.10021 = Counter32: 855942010

RFC1213-MIB::ifOutOctets.10022 = Counter32: 1325196398

RFC1213-MIB::ifOutOctets.10023 = Counter32: 4742

RFC1213-MIB::ifOutOctets.10024 = Counter32: 5600

RFC1213-MIB::ifOutOctets.10101 = Counter32: 0

RFC1213-MIB::ifOutOctets.10102 = Counter32: 0

RFC1213-MIB::ifOutOctets.10501 = Counter32: 94837887

RFC1213-MIB::ifOutOctets.10502 = Counter32: 1720005462

RFC1213-MIB::ifOutOctets.10503 = Counter32: 5109

RFC1213-MIB::ifOutOctets.10504 = Counter32: 0

RFC1213-MIB::ifOutOctets.10505 = Counter32: 0

RFC1213-MIB::ifOutOctets.10506 = Counter32: 0

RFC1213-MIB::ifOutOctets.10507 = Counter32: 0

RFC1213-MIB::ifOutOctets.10508 = Counter32: 0

RFC1213-MIB::ifOutOctets.10509 = Counter32: 0

RFC1213-MIB::ifOutOctets.10510 = Counter32: 0

RFC1213-MIB::ifOutOctets.10511 = Counter32: 0

RFC1213-MIB::ifOutOctets.10512 = Counter32: 0

RFC1213-MIB::ifOutOctets.10513 = Counter32: 1515478009

RFC1213-MIB::ifOutOctets.10514 = Counter32: 1468150812

RFC1213-MIB::ifOutOctets.10515 = Counter32: 1935996052

RFC1213-MIB::ifOutOctets.10516 = Counter32: 1476773137

RFC1213-MIB::ifOutOctets.10517 = Counter32: 1819484289

RFC1213-MIB::ifOutOctets.10518 = Counter32: 1256649345

RFC1213-MIB::ifOutOctets.10519 = Counter32: 0

RFC1213-MIB::ifOutOctets.10520 = Counter32: 0

RFC1213-MIB::ifOutOctets.10521 = Counter32: 0

RFC1213-MIB::ifOutOctets.10522 = Counter32: 0

RFC1213-MIB::ifOutOctets.10523 = Counter32: 0

RFC1213-MIB::ifOutOctets.10524 = Counter32: 90612563

RFC1213-MIB::ifOutOctets.10601 = Counter32: 0

RFC1213-MIB::ifOutOctets.10602 = Counter32: 0

RFC1213-MIB::ifOutOctets.14501 = Counter32: 0


It looks like they're been recorded on the switch. Is there anyway to tell wich Counter related to which interface? I could ten check a specific port graph on HUM??


Joe Clarke Fri, 05/01/2009 - 10:17
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Static snapshots of a counter don't mean anything. You need to take multiple snapshots to see that the values are incrementing. You can use the ifDescr object to tie an ifIndex to an interface name.

BlueyVIII Fri, 05/01/2009 - 14:49
User Badges:

Left it a few hours then did the SNMP Walk again and it would appear that the stats are indeed incremeting..


RFC1213-MIB::ifOutOctets.1 = Counter32: 0

RFC1213-MIB::ifOutOctets.10 = Counter32: 22570907

RFC1213-MIB::ifOutOctets.20 = Counter32: 50520050

RFC1213-MIB::ifOutOctets.30 = Counter32: 0

RFC1213-MIB::ifOutOctets.5179 = Counter32: 0

RFC1213-MIB::ifOutOctets.5182 = Counter32: 0

RFC1213-MIB::ifOutOctets.10001 = Counter32: 1709321340

RFC1213-MIB::ifOutOctets.10002 = Counter32: 1100811950

RFC1213-MIB::ifOutOctets.10003 = Counter32: 2079949811

RFC1213-MIB::ifOutOctets.10004 = Counter32: 1201490995

RFC1213-MIB::ifOutOctets.10005 = Counter32: 1305224914

RFC1213-MIB::ifOutOctets.10006 = Counter32: 2079280792

RFC1213-MIB::ifOutOctets.10007 = Counter32: 885930521

RFC1213-MIB::ifOutOctets.10008 = Counter32: 1463408860

RFC1213-MIB::ifOutOctets.10009 = Counter32: 0

RFC1213-MIB::ifOutOctets.10010 = Counter32: 0

RFC1213-MIB::ifOutOctets.10011 = Counter32: 1266446122

RFC1213-MIB::ifOutOctets.10012 = Counter32: 1386798059

RFC1213-MIB::ifOutOctets.10013 = Counter32: 1333747496

RFC1213-MIB::ifOutOctets.10014 = Counter32: 665160389

RFC1213-MIB::ifOutOctets.10015 = Counter32: 1426076735

RFC1213-MIB::ifOutOctets.10016 = Counter32: 568166602

RFC1213-MIB::ifOutOctets.10017 = Counter32: 1831497109

RFC1213-MIB::ifOutOctets.10018 = Counter32: 1366353768

RFC1213-MIB::ifOutOctets.10019 = Counter32: 0

RFC1213-MIB::ifOutOctets.10020 = Counter32: 0

RFC1213-MIB::ifOutOctets.10021 = Counter32: 858720229

RFC1213-MIB::ifOutOctets.10022 = Counter32: 1327974355

RFC1213-MIB::ifOutOctets.10023 = Counter32: 4742

RFC1213-MIB::ifOutOctets.10024 = Counter32: 5600

RFC1213-MIB::ifOutOctets.10101 = Counter32: 0

RFC1213-MIB::ifOutOctets.10102 = Counter32: 0

RFC1213-MIB::ifOutOctets.10501 = Counter32: 95369033

RFC1213-MIB::ifOutOctets.10502 = Counter32: 1721253617

RFC1213-MIB::ifOutOctets.10503 = Counter32: 5109

RFC1213-MIB::ifOutOctets.10504 = Counter32: 0

RFC1213-MIB::ifOutOctets.10505 = Counter32: 0

RFC1213-MIB::ifOutOctets.10506 = Counter32: 0

RFC1213-MIB::ifOutOctets.10507 = Counter32: 0

RFC1213-MIB::ifOutOctets.10508 = Counter32: 0

RFC1213-MIB::ifOutOctets.10509 = Counter32: 0

RFC1213-MIB::ifOutOctets.10510 = Counter32: 0

RFC1213-MIB::ifOutOctets.10511 = Counter32: 0

RFC1213-MIB::ifOutOctets.10512 = Counter32: 0

RFC1213-MIB::ifOutOctets.10513 = Counter32: 1520427998

RFC1213-MIB::ifOutOctets.10514 = Counter32: 1470929529

RFC1213-MIB::ifOutOctets.10515 = Counter32: 1941734403

RFC1213-MIB::ifOutOctets.10516 = Counter32: 1481282682

RFC1213-MIB::ifOutOctets.10517 = Counter32: 1822122488

RFC1213-MIB::ifOutOctets.10518 = Counter32: 1260136632

RFC1213-MIB::ifOutOctets.10519 = Counter32: 0

RFC1213-MIB::ifOutOctets.10520 = Counter32: 0

RFC1213-MIB::ifOutOctets.10521 = Counter32: 0

RFC1213-MIB::ifOutOctets.10522 = Counter32: 0

RFC1213-MIB::ifOutOctets.10523 = Counter32: 0

RFC1213-MIB::ifOutOctets.10524 = Counter32: 561781067

RFC1213-MIB::ifOutOctets.10601 = Counter32: 0

RFC1213-MIB::ifOutOctets.10602 = Counter32: 0

RFC1213-MIB::ifOutOctets.14501 = Counter32: 0


Does this point to a problem with HUM rather than the switch??

Joe Clarke Fri, 05/01/2009 - 17:14
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Quite possibly. More analysis of the symptoms and logs are required. Post a screenshot illustrating exactly what the problem is.

BlueyVIII Tue, 05/05/2009 - 02:48
User Badges:

Please find attached 2 HUM Histo graphs showing TX and RX utilisation for the same port (fa1/0/1) of the same switch.


The RX Util Graph shows data, however, the TX Graph states "No Data to Generate graph"


From the SNMP Walks above we know that the stats are being gathered and incremented for this port.


This is the same situation with all (120+) of our 3750's.



Joe Clarke Tue, 05/05/2009 - 10:52
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

This could be a poller problem. Can you deleted and recreate a poller for this device (or just create a new test poller) to see if the Tx stats return?

BlueyVIII Thu, 05/07/2009 - 06:56
User Badges:

Deleted and recreated the poller which doesn't seem to have solved the problem.


However, instead of the Graph now stating "No Data to Generate Graph" the graph is just blank (see attached screenshot).


The RX graph is working as expected.



Joe Clarke Thu, 05/07/2009 - 09:33
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I haven't been able to reproduce this locally. My 3750 is reporting both Rx and Tx stats. However, I am on HUM 1.1.2. You can try upgrading from http://www.cisco.com/cgi-bin/tablebuild.pl/cw2000-hum , or contact TAC so live troubleshooting can be done on your existing installation.

BlueyVIII Thu, 05/07/2009 - 11:26
User Badges:

Thanks....can I upgrade from 1.1.0 direct to 1.1.2 or do I need to go through 1.1.1? It's unclear from the Readme file...

Joe Clarke Thu, 05/07/2009 - 11:29
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

You can upgrade directly.

BlueyVIII Fri, 05/08/2009 - 09:20
User Badges:

Upgraded to version 1.1.2 as per your link but it seems to have made the problem worse!


Now when I try and create a histograph only a small selection of interfaces are available to chose from. I done some testing and found that the ports which appear in the list are ones which are either administratively or operationally "down". The same happens for all devices, not just 3750's!


I've attache screenshots of the histograph problem and also of the poller configuration to show that the poller is configured to get utilisation data from ALL interfaces.


Any idea's? Is it possible to regress back to HUM version 1.1.1??





Joe Clarke Fri, 05/08/2009 - 10:35
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I just installed HUM 1.1.2 on a new server, and I cannot reproduce. In fact, I can't even think of why this would happen. My guess is it could have something to do with the ifSpeed of the interfaces (i.e. being set to a value less than 100 Mbps). However, I can't reproduce that, so I don't think that's the case.


If you want to troubleshoot, open a TAC service request, and they can take a look at logs and sniffer traces. To get back to HUM 1.1, you will need to uninstall HUM 1.1.2, then install HUM 1.1 from the LMS 3.1 DVD.

BlueyVIII Fri, 05/08/2009 - 15:35
User Badges:

Having looked into this further over the last few hours I can confirm that the HUM 1.1.2 upgrade has infact solved the problem.


The reason for only some interfaces appearing on the HISTOGRAPH list is because interfaces greater than 20Mbps are now shown in the 64bit Mib Variables - See attachment.


I'm now getting both TX and RX utilisation graphs for 3750's (and all other devices)


Since I upgraded however I've noticed that 2 Processes are now in a state of "Administrator has shutdown this server" (see attachment). I'm pretty sure these processes were running OK before the 1.1.2 upgrade but re-booting the server and attempting to manually start the processes has failed? You can see from the attachment that both services go back to a shutdown state within 30 seconds of me attempting to manually start them.



Do I need to be worried or is this normal operation?



Attachment: 
Correct Answer
Joe Clarke Fri, 05/08/2009 - 17:17
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

The 64-bit counter support was the main feature in the HUM 1.X.2 releases. The way you were describing the problem, I had thought you couldn't see these instances when DEFINING the poller. It wasn't clear that the missing instances were only in if*Octets in HistoGraphIt. Good, I'm glad it's working now.


These two daemons are typically down in normal operation. You should not try to start them manually. LMS is working normally.

Actions

This Discussion