SG300-28PP FindIT Network Probe causes high CPU load and https error log entries

[Hakunafrittata]

Hi guys,

first of all I want to apologize my bad English, I´m German and it´s not my mother language.

I am owner of a SG300-28PP with newest firmware and it runs smoothly in our house. Some weeks ago I thought of changing the switch into a unifi product (us-24-150W) just because its is easier to track errors or see whats going on. But, i love my cisco and it´s performance especially when moving big data from one VLAN to another. So after weeks of taking research I found the FindIT Manager and Probe. I downloaded and configured the Virtualbox versions which run on a Debian 9 PC.

As soon as I start the Probe i get high CPU load of about 60-80%. Yes, just the Probe. Same happens if I start only the Manager. I get every 5 minutes (i think so) entries in the error log. I tried everything i found worldwide. HTTPS and HTTP as service are enabled, am using SNMPv2, network is getting discovered and shown (only VLANS/"clouds" aren´t showing all subnets).

How can I get rid oh these errors please, any help would be appreciated much.

2147483634	2018-Jun-07 21:19:44	Warning	%AAA-W-REJECT: New https connection for user cisco, source 192.168.30.31 destination 192.168.30.254  REJECTED
2147483635	2018-Jun-07 21:19:44	Warning	%HTTP_HTTPS-W-WEBWARNING: viaGetSecurityHandler:credentials expected to be encrypted
2147483636	2018-Jun-07 21:19:40	Warning	%AAA-W-REJECT: New https connection for user cisco, source 192.168.30.31 destination 192.168.30.254  REJECTED, aggregated (30)
2147483637	2018-Jun-07 21:19:40	Warning	%HTTP_HTTPS-W-WEBWARNING: viaGetSecurityHandler:credentials expected to be encrypted, aggregated (30)
2147483638	2018-Jun-07 21:14:41	Warning	%AAA-W-REJECT: New https connection for user cisco, source 192.168.30.31 destination 192.168.30.254  REJECTED
2147483639	2018-Jun-07 21:14:41	Warning	%HTTP_HTTPS-W-WEBWARNING: viaGetSecurityHandler:credentials expected to be encrypted
2147483640	2018-Jun-07 21:14:19	Warning	%AAA-W-REJECT: New https connection for user cisco, source 192.168.30.31 destination 192.168.30.254  REJECTED, aggregated (26)
2147483641	2018-Jun-07 21:14:19	Warning	%HTTP_HTTPS-W-WEBWARNING: viaGetSecurityHandler:credentials expected to be encrypted, aggregated (26)
2147483642	2018-Jun-07 21:09:24	Warning	%AAA-W-REJECT: New https connection for user cisco, source 192.168.30.31 destination 192.168.30.254  REJECTED
2147483643	2018-Jun-07 21:09:24	Warning	%HTTP_HTTPS-W-WEBWARNING: viaGetSecurityHandler:credentials expected to be encrypted, aggregated (1)
2147483644	2018-Jun-07 21:08:27	Informational	%AAA-I-DISCONNECT: https connection for user cisco, source 192.168.30.32 destination 192.168.30.254 TERMINATED
2147483645	2018-Jun-07 21:07:23	Informational	%AAA-I-CONNECT: New https connection for user cisco, source 192.168.30.32 destination 192.168.30.254 ACCEPTED
2147483646	2018-Jun-07 21:07:23	Warning	%HTTP_HTTPS-W-WEBWARNING: viaGetSecurityHandler:credentials expected to be encrypted

IP .31 is the manager, 32 the probe, 30 the host, switch .254

I don´t understand where it comes from. Certificate error ? I have no own signed certificates or changed anything belonging this. In addition to this, I am getting multiple "Warning health level" notifications in FindIT and whole switch/webgui is getting slower. I wouldn´t need to change the switch if I could get this to work.

Is it possible to view in console or by snmp query the CPU temp and memory usage ? Also, is it possible to view VLAN to VLAN traffic with Findit ?

Many thanks in advance and beste Gruesse ! :-)

Vince

[David Harper · Accepted solution]

Hi Vince,

Your English is so much better than my German - I don't think you have anything to apologise for. :)

FindIT accesses the switch in two ways - using SNMP and also using a web service running on the switch.  Since you say 192.168.30.31 is the Manager and .32 is the Probe, you would not expect .31 to be trying to connect to the switch.  But you may not be aware that we include the Probe application in the Manager VM to simplify single site deployments, and from memory it is enabled by default.  So it seems likely that the AAA-W-REJECT messages are being caused by the Probe running in the Manager VM attempting to discover the switch by logging on using the factory default of cisco/cisco.  The simplest way to check this is to go to Administration > Local Probe and ensure the Local Probe is disabled.  As for the HTTPS-W-WEBWARNING messages, I'll need to doublecheck, but I'm pretty sure they are expected.  However we really should try and address the underlying cause that triggers it so that you don't see them.  I'll do some digging and get back to you on that one.

As for the high CPU, the CPU definitely will spike up when FindIT is doing discovery as we do retrieve quite a lot of data from the switch - mostly port statistics and neighbour tables.  Can I doublecheck what version of FindIT you are running and also the firmware version on the switch?  I'm guessing from your description that you have the latest version, but I'd like to confirm that.  We did make some changes recently to try and reduce some of the CPU impact and I want to make sure you are running a version that has those changes.  Also, if you are in fact running the local probe, and if the SNMP community happens to be set to public, then you will actually have two probes pulling the same data from the switch, which will also drive the CPU utilisation up.

Regarding the question about VLAN to VLAN traffic, can you be more specific about what you are trying to do?  FindIT will certainly discover across VLANs so long as it can manage at least one device that is present on both VLANs.  Is that what you are after, or are you looking for something different?

Cheers,

Dave.

[Hakunafrittata]

Good morning Dave,

 I will take it as compliment, thank you !

Great ! The high CPU load and with that, the main problem, is solved. Thanks a lot for your very quick response and the solution. This includes the https warnings, they´re gone. You were totally right with your guess (expected web warnings) and thank you for your easy-to-understand explanation.

Btw. i am using

firmware version 1.4.8.6

Boot version       1.3.5.06

Findit Manager    1.1.1.20171116

Findit Probe        1.1.1.20171115

I´m asking myself right now if it wouldn´t be better to install the Manager (2GB RAM as VM) on the headless host (Debian Stretch) but remain the Probe as VM (512MB), so save RAM.

Should i change SNMP community to something else than public ? I have an ubiquiti router which also has public as community but isn´t enabled. Would the SG300 get´s in trouble ?

About the VLAN, i really would love to see how much traffic or something similar i.ex. is getting transferred from VLAN 70 (security cams) to Synology NAS (VLAN 50), same as for other VLANS. Just to ensure the first weeks there is no wanted (big) traffic. If i could see i.ex. that my voip VLAN 80 has, let´s say 50MB but not 1GB, i would be able to see unwanted incoming/outgoing traffic.

I have enabled my first 4 ACE/ACL´s to deny i.ex. traffic from VLAN 10 (smart home/home tech stuff) to VLAN 80 (VOIP) and vice versa, just to mention it.

About the mixed up discovery, please see attached pic. Green circles are house tech (VLAN10), also IP subnet .50. is shown directly connected. I would expect for each VLAN a cloud, am i wrong ?

Mhh, about your German i would (unexpected) be positively amazed you would even speak 10 words. German as well as Italian are not really wide spreaded in the world. But feel free to ask everything you like, i will translate :-).

Best greetings from Braunschweig (Lower Saxony)

Vince

[David Harper]

Good to hear the CPU issue and the error messages are fixed.

As for the best deployment model, we don't technically support reducing the memory allocated to the Manager VM below the 4G that the VM image is configured for, though it will work up to a point, especially on a smaller network.  However, we do pre-allocate memory for performance reasons, so you could indeed have problems if you reduce it too far.

As for whether you are better off using the Probe embedded in the Manager VM or a separate Probe, actually using the embedded Probe will be more efficient as you are only running one instance of the operating system.  Effectively you will save the 512M allocated to the Probe VM.

For the SNMP community, it is always good practice from a security perspective to change it to something unique and hard to guess.  Public is the first community a hacker would test if they were able to access your network, so you should always change it if you are enabling SNMP.  The switch - and the Ubiquiti router for that matter - will not have a problem with a more secure community, so you really should change it.  When you do, you will need to go to Administration > Credentials in FindIT to enter in the new community, or FindIT will no longer be able to access the switch.

For the traffic between VLANs, you could use the Dashboard to monitor the traffic in and out of the switchport the NAS is connected to, but the will also capture traffic from the same VLAN.  Right now, I can't think of any good reason why the Dashboard should not allow you to monitor traffic on logical interfaces like VLANs as well as physical interfaces, so I'll have a talk to our engineering team and see if we can get that enabled.  But right now, we can't directly report traffic through a VLAN.

As for the clouds, they do not represent VLANs, but rather they represent a part of the network that FindIT is unable to discover properly.  FindIT knows there are all those hosts out there, and which port on the SG300 switch they are connected to, but since there are several hosts there, there must be another switch or something that FindIT cannot discover.  There are a few different reasons why this can happen, but the most common are:

• There is an unmanaged switch in that part of the network that does not support any discovery protocol or have an IP address.
• There is a managed switch, access point or other network device in that part of the network, but the device is not advertising its presence using a protocol such as LLDP, CDP, Bonjour etc, and it either does not have SNMP enabled or FindIT does not know the right credentials to access the device.
• There is only one PC actually plugged in to the network there, but the PC is running virtualisation software and has multiple virtual machines running.  It seems likely that this is the explanation for the cloud on the right hand side of the screenshot.

If you do want to see which devices are in the different VLANs, click on the Overlays button in the top right of the topology diagram, and select VLAN View from the overlay dropdown.  Then you can select the VLAN ID you are interested in and the topology diagram will show which links are in that VLAN.

Cheers,

Dave.

[Hakunafrittata]

Hi Dave,

thanks again for the background info. Well, I reduced the Manager VM to 2GB because i somewhere at your pages have read 2GB as minimum requirement. I anyway put in another 4GB to the machine so I could change it back to 4GB.

You mentioned just to use the Manager and saving the memory of the Probe. I did so. I turned off the Probe, enabled the internal Probe but, I nowhere see widgets I could add like in the Probe VM. I only see Projects, Devices in Project and Unclaimed Devices. Could you please lead me to the functions ?

Regarding SNMP, I only have 1 switch. But the Unifi controller, respectively the 2 Unifi APs, are shown as cloud, as long as at least 1 device is connected. The 3. "cloud" is for the host of the VM´s, right.

About the VLAN traffic I am looking forward to your later feedback.

Have fun and a great weekend.

Viele Grüße
Vince

[David Harper]

2G was the requirement for the original 1.0 version, but we added some functionality in 1.1 and 2G started to get a bit tight. 4G gives us plenty of headroom, and allows the Probe to run local as well.

With the local probe, it is represented as a site in the Manager just like the separate probe was. The difference is that you don't have to manually associate the local probe with the manager, so you may not know it is there. If you display the site list instead of the map, you will see it listed and you will be able to click into it and see all the network. It is visible on the map as well, but the default location is unspecified, which in practice drops the site somewhere in the middle of the pacific ocean. I'm not surprised you didn't see it.

The Unifi APs will appear as clouds when they have wireless devices associated - or more precisely as a cloud with each wireless device attached plus a host device for the AP itself.  As far as I can tell, the Unifi APs do no yet support LLDP - or at least, not enabled by default - so it is difficult for us to identify the device as an access point.  There really isn't a reliable MIB variable in SNMP that we can use to identify the type of device, so there's not a lot we can do to figure out what the device is.

I did some googling, and it seem like you may be able to run an LLDP daemon on the AP from the command line, depending on which AP and firmware version you are running.  If you do that, FindIT may be able to correctly identify the devices as access points and represent the topology more accurately, depending on what information Ubiquiti put in the LLDP advertisement.  But it would only be temproary, as as far as I can tell, the daemon will stop running if the AP reboots.

Cheers,

Dave.