All Apps and Add-ons
Highlighted

Entities not displayed in Splunk App for Infrastructure.

Path Finder

I have set up a Universal Forwarder(UF) from the script on Machine 2 but UF is not added on Splunk Enterprise(Machine 1).
I have manually added the deployment server and in this case, the UF is added on Splunk Enterprise but the entity is not displayed on Splunk App for Infrastructure for which I have waited for more than 5 mins.

Followed the below link to install SAI on Splunk Enterprise:
https://docs.splunk.com/Documentation/InfraApp/2.0.1/Install/Install

0 Karma
Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Path Finder

Does the splunkd.log from the UF say anything about whether the data is successfully sending to Machine 1?

View solution in original post

Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Path Finder

In the UF logs on Machine 2 getting the message that it's connected to Machine 1 but when I visited in Forwarder Management tab then it's not displayed there.

For the reference please refer the last few lines of UF logs after starting UF:

01-07-2020 05:35:13.024 -0500 INFO TcpOutputProc - Connected to idx=192.168.1.15:9997, pset=0, reuse=0.
01-07-2020 05:35:13.029 -0500 INFO WatchedFile - Will begin reading at offset=13776943 for file='/data/splunkforwarder/var/log/splunk/metrics.log'.
01-07-2020 05:35:13.032 -0500 INFO WatchedFile - Will begin reading at offset=978 for file='/data/splunkforwarder/var/log/splunk/conf.log'.
01-07-2020 05:35:42.667 -0500 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views

0 Karma
Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

"Add Data" script installs both collectd(Metrics) and UF(logs) for Linux machine.
Also check, "/etc/collectd/collectd.logs" for any errors.
For machine 2, what is you Linux distro like Centos, Ubuntu? and what version?

Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Path Finder

For both Machine 1 and Machine 2 I am using Ubuntu 18.04 LTS.

This file doesn't exist on Machine 2(client machine): /etc/collectd/collectd.logs

0 Karma
Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Path Finder

The specific location of the collectd.log may vary by distro, but the information should be in the collectd.log on Machine 2.

Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Path Finder

I have reinstalled it from the script and get the below error in the collectd.logs:

[2020-01-09 08:31:40] [error] processmon plugin: Error reading /proc/12820/stat
[2020-01-09 08:31:40] [notice] read-function of plugin `processmon' failed. Will suspend it for 120.000 seconds.

0 Karma
Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Path Finder

Is the data arriving at Machine 1? If you search index=_internal host=${Machine 2} or | mcatalog values(metric_name) WHERE host=${Machine 2} AND index=em_metrics, do you see data?

Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Path Finder

Yes, Gettings logs at Machine 1 but didn't get the metrics.

I am getting the output of this command:
index=_internal host=${Machine 2}

but didn't get any output of this command:
| mcatalog values(metricname) WHERE host=${Machine 2} AND index=emmetrics

0 Karma
Highlighted

Re: Entities not displayed in Splunk App for Infrastructure.

Check if collectd running or installed on monitored Machine 2..

apt-cache policy collectd
ps -ef | grep collectd

Did you get any errors when you ran the script from "Add Data" page?

Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.