topic Re: Entities not displayed in Splunk App for Infrastructure. in All Apps and Add-ons

Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Mon, 06 Jan 2020 12:06:55 GMT

I have set up a Universal Forwarder(UF) from the script on Machine 2 but UF is not added on Splunk Enterprise(Machine 1).
I have manually added the deployment server and in this case, the UF is added on Splunk Enterprise but the entity is not displayed on Splunk App for Infrastructure for which I have waited for more than 5 mins.

Followed the below link to install SAI on Splunk Enterprise:
https://docs.splunk.com/Documentation/InfraApp/2.0.1/Install/Install

Re: Entities not displayed in Splunk App for Infrastructure.

pwu_splunk — Mon, 06 Jan 2020 22:41:25 GMT

Does the splunkd.log from the UF say anything about whether the data is successfully sending to Machine 1?

Re: Entities not displayed in Splunk App for Infrastructure.

dagarwal_splunk — Mon, 06 Jan 2020 22:47:02 GMT

Can you clarify your setup more:

Machine 2 is running SAI(Splunk app for infrastructure) and SAI Add-on?
Did you use script on "Add Data" page of the app? Are you trying to monitor Machine 1? Is it Windows or Linux?

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Tue, 07 Jan 2020 07:09:16 GMT

My both machines are Linux, SAI is installed on Machine 1(Server Machine) with SAI Add-on and now I am trying to install UF on Machine 2(Client Machine) with the help of script from "Add Data" page of the App. So, I am trying to monitor Machine 2

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Tue, 07 Jan 2020 11:03:58 GMT

In the UF logs on Machine 2 getting the message that it's connected to Machine 1 but when I visited in Forwarder Management tab then it's not displayed there.

For the reference please refer the last few lines of UF logs after starting UF:

01-07-2020 05:35:13.024 -0500 INFO TcpOutputProc - Connected to idx=192.168.1.15:9997, pset=0, reuse=0.
01-07-2020 05:35:13.029 -0500 INFO WatchedFile - Will begin reading at offset=13776943 for file='/data/splunkforwarder/var/log/splunk/metrics.log'.
01-07-2020 05:35:13.032 -0500 INFO WatchedFile - Will begin reading at offset=978 for file='/data/splunkforwarder/var/log/splunk/conf.log'.
01-07-2020 05:35:42.667 -0500 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views

Re: Entities not displayed in Splunk App for Infrastructure.

dagarwal_splunk — Tue, 07 Jan 2020 17:56:55 GMT

"Add Data" script installs both collectd(Metrics) and UF(logs) for Linux machine.
Also check, "/etc/collectd/collectd.logs" for any errors.
For machine 2, what is you Linux distro like Centos, Ubuntu? and what version?

Re: Entities not displayed in Splunk App for Infrastructure.

pwu_splunk — Tue, 07 Jan 2020 18:25:23 GMT

Is the data arriving at Machine 1? If you search index=_internal host=${Machine 2} or | mcatalog values(metric_name) WHERE host=${Machine 2} AND index=em_metrics, do you see data?

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Wed, 30 Sep 2020 03:36:17 GMT

Yes, Gettings logs at Machine 1 but didn't get the metrics.

I am getting the output of this command:
index=_internal host=${Machine 2}

but didn't get any output of this command:
| mcatalog values(metric_name) WHERE host=${Machine 2} AND index=em_metrics

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Wed, 08 Jan 2020 14:42:57 GMT

For both Machine 1 and Machine 2 I am using Ubuntu 18.04 LTS.

This file doesn't exist on Machine 2(client machine): /etc/collectd/collectd.logs

Re: Entities not displayed in Splunk App for Infrastructure.

dagarwal_splunk — Wed, 08 Jan 2020 18:05:04 GMT

Check if collectd running or installed on monitored Machine 2..

apt-cache policy collectd
ps -ef | grep collectd

Did you get any errors when you ran the script from "Add Data" page?

Re: Entities not displayed in Splunk App for Infrastructure.

pwu_splunk — Thu, 09 Jan 2020 00:45:43 GMT

The specific location of the collectd.log may vary by distro, but the information should be in the collectd.log on Machine 2.

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Thu, 09 Jan 2020 08:40:29 GMT

I have reinstalled it from the script and get the below error in the collectd.logs:

[2020-01-09 08:31:40] [error] processmon plugin: Error reading /proc/12820/stat
[2020-01-09 08:31:40] [notice] read-function of plugin `processmon' failed. Will suspend it for 120.000 seconds.

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Thu, 09 Jan 2020 09:59:02 GMT

Yes, collectd is installed and running, verified from the above 2 commands.

Yes, getting error in the collectd.logs:

[2020-01-09 08:31:40] [error] processmon plugin: Error reading /proc/12820/stat
[2020-01-09 08:31:40] [notice] read-function of plugin `processmon' failed. Will suspend it for 120.000 seconds.

Re: Entities not displayed in Splunk App for Infrastructure.

dagarwal_splunk — Wed, 30 Sep 2020 03:33:34 GMT

Ignore that error. It's just that process died while it was being monitored.

To debug, let's try some steps:

Machine2: Do you see any recurring errors like "curl_easy_perform failed" in collectd.log ?
In Machine 1, Check if all the Hec tokens are enabled: Settings -> Data Inputs ->HTTP Event Collector
Machine 1: Check the Global Settings on the same page as 2. Verify "enable ssl" is checked and note down the port number.
Machine 1: Verify the HEC token you are using has default index as "em_metrics"
Now In Machine 2, check /etc/collectd/collectd.conf file. Verify that HEC token, server and port number in write_splunk stanza is correct.
If still not solved, try sending fake data from Machine 2 to Machine 1 using curl and see if you get success. Here is the curl command you need to run in Machine 2: curl -k https://Machine1:8088/services/collector -H "Authorization: Splunk hec_token_here" -d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

https://docs.splunk.com/Documentation/Splunk/7.3.3/Metrics/GetMetricsInOther#Example_of_sending_metrics_using_HEC
Update token, port and server in the command

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Wed, 30 Sep 2020 03:37:54 GMT

Got the below error in the collectd.logs when searched for "curl_easy_perform failed":

[error] write splunk plugin: curl_easy_perform failed to connect to 192.168.1.15:8088 with status 7: Couldn't connect to server

Yes, I have already enabled all the HEC tokens.
In the Global Settings, SSL is already enabled and the port number is 8088(default).
HEC token which I am using has default index as "em_metrics".
HEC token, Server IP and port number in /etc/collectd/collectd.conf file is correct.
Getting the below output of the given curl command:

{"text":"Server is busy","code":9,"invalid-event-number":0}

Re: Entities not displayed in Splunk App for Infrastructure.

dagarwal_splunk — Fri, 10 Jan 2020 13:55:51 GMT

Try this:
https://answers.splunk.com/answers/453888/after-configuring-the-http-event-collector-why-am.html

Re: Entities not displayed in Splunk App for Infrastructure.

himanshutejwani — Mon, 13 Jan 2020 07:14:31 GMT

Yeah, it's working now, I have unchecked the Use Deployment Server option from Global Settings.
Thanks, dagarwal

Re: Entities not displayed in Splunk App for Infrastructure.

dagarwal_splunk — Wed, 30 Sep 2020 03:43:02 GMT

To conclude the steps for resolving Metrics data (collectd) collection issues:
(NOTE: machine2 is the monitored Linux machine i.e running collectd and Splunk UF; machine 1 is running SAI And SAI Add-on)

Check if collectd running or installed on monitored Machine 2..
apt-cache policy collectd
ps -ef | grep collectd
Check Metrics data coming in: | mcatalog values(metric_name) WHERE host=${Machine 2} AND index=em_metrics
Machine2: Do you see any recurring errors like "curl_easy_perform failed" in collectd.log or any other error?
In Machine 1, Check if all the Hec tokens are enabled: Settings -> Data Inputs ->HTTP Event Collector
Machine 1: Check the Global Settings on the same page as 2. Verify "enable ssl" is checked, "Use Deployment Server" unchecked and note down the port number.
Machine 1: Verify the HEC token you are using has default index as "em_metrics"
Now In Machine 2, check /etc/collectd/collectd.conf file. Verify that HEC token, server and port number in write_splunk stanza is correct.
Try sending fake data from Machine 2 to Machine 1 using curl and see if you get success. Here is the curl command you need to run in Machine 2:
curl -k https://Machine1:8088/services/collector -H "Authorization: Splunk hec_token_here" -d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

You should see "Success" Message. If not, try to fix the error message that you get.
https://docs.splunk.com/Documentation/Splunk/7.3.3/Metrics/GetMetricsInOther#Example_of_sending_metrics_using_HEC
Update token, port and server in the command

Re: Entities not displayed in Splunk App for Infrastructure.

3es — Mon, 16 Mar 2020 03:50:32 GMT

I have followed the above steps and I can able to test fake data and getting success message. However still I am not able to see Machine 2 in Entities.

Re: Entities not displayed in Splunk App for Infrastructure.

dagarwal_splunk — Mon, 16 Mar 2020 16:47:18 GMT

What are the search results for Step 2?