All Apps and Add-ons

Entities not displayed in Splunk App for Infrastructure.

Path Finder

I have set up a Universal Forwarder(UF) from the script on Machine 2 but UF is not added on Splunk Enterprise(Machine 1).
I have manually added the deployment server and in this case, the UF is added on Splunk Enterprise but the entity is not displayed on Splunk App for Infrastructure for which I have waited for more than 5 mins.

Followed the below link to install SAI on Splunk Enterprise:
https://docs.splunk.com/Documentation/InfraApp/2.0.1/Install/Install

0 Karma
1 Solution

Splunk Employee
Splunk Employee

Does the splunkd.log from the UF say anything about whether the data is successfully sending to Machine 1?

View solution in original post

Splunk Employee
Splunk Employee

To conclude the steps for resolving Metrics data (collectd) collection issues:
(NOTE: machine2 is the monitored Linux machine i.e running collectd and Splunk UF; machine 1 is running SAI And SAI Add-on)

  1. Check if collectd running or installed on monitored Machine 2..
    apt-cache policy collectd
    ps -ef | grep collectd

  2. Check Metrics data coming in: | mcatalog values(metric_name) WHERE host=${Machine 2} AND index=em_metrics

  3. Machine2: Do you see any recurring errors like "curl_easy_perform failed" in collectd.log or any other error?

  4. In Machine 1, Check if all the Hec tokens are enabled: Settings -> Data Inputs ->HTTP Event Collector

  5. Machine 1: Check the Global Settings on the same page as 2. Verify "enable ssl" is checked, "Use Deployment Server" unchecked and note down the port number.

  6. Machine 1: Verify the HEC token you are using has default index as "em_metrics"

  7. Now In Machine 2, check /etc/collectd/collectd.conf file. Verify that HEC token, server and port number in write_splunk stanza is correct.

  8. Try sending fake data from Machine 2 to Machine 1 using curl and see if you get success. Here is the curl command you need to run in Machine 2:
    curl -k https://Machine1:8088/services/collector -H "Authorization: Splunk hec_token_here" -d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

You should see "Success" Message. If not, try to fix the error message that you get.
https://docs.splunk.com/Documentation/Splunk/7.3.3/Metrics/GetMetricsInOther#Example_of_sending_metr...
Update token, port and server in the command

New Member

I have followed the above steps and I can able to test fake data and getting success message. However still I am not able to see Machine 2 in Entities.

0 Karma

Splunk Employee
Splunk Employee

What are the search results for Step 2?

0 Karma

New Member

Refer below output.

| mcatalog values(metric_name) WHERE host=xxxxx AND index=em_metrics
cpu.idle
cpu.interrupt
cpu.nice
cpu.softirq
cpu.steal
cpu.system
cpu.user
cpu.wait
df.free
df.reserved
df.used
disk.io_time.io_time
disk.io_time.weighted_io_time
disk.merged.read
disk.merged.write
disk.octets.read
disk.octets.write
disk.ops.read
disk.ops.write
disk.pending_operations
disk.time.read
disk.time.write
interface.dropped.rx
interface.dropped.tx
interface.errors.rx
interface.errors.tx
interface.octets.rx
interface.octets.tx
interface.packets.rx
interface.packets.tx
load.longterm
load.midterm
load.shortterm
memory.buffered
memory.cached
memory.free
memory.slab_recl
memory.slab_unrecl
memory.used
uptime.uptime

0 Karma

Splunk Employee
Splunk Employee

you do have metrics data in Splunk.. what version of SAI do you have? Do you have any entities in SAI?

0 Karma

New Member

SAI version:
description = Splunk App for Infrastructure
version = 2.0.2
build = 10

My Splunk version: Splunk 8.0.0 (build 1357bef0a7f6)

I can see only one entity which is same server of index/SH (I have installed Splunk all in one server: machine1)

Also,

I can see the metrics data in Splunk from the server which has UF.
Sample command:
| mstats count where metric_name=cpu.idle host=* BY metric_name, host

Result: last 15 Mins
metric_name host count
cpu.idle machine1 14 (SH/Index)
cpu.idle machine2 13 (UF 8.0.1)
cpu.idle machine3 12 (7.2.4)

UF version:
Splunk Universal Forwarder 8.0.1 (build 6db836e2fb9e)
and
Splunk Universal Forwarder 7.2.4 (build 8a94541dcfac)

0 Karma

Splunk Employee
Splunk Employee

Metrics data is sent by collectd not UF for Linux machines..

So, I guess you ran the "Add Data" script on 3 machines and you only see 1 entity. But, you have metrics data from all 3 machines?

0 Karma

New Member

In Machine 1, I have installed SAI App and Addon and configured collectd (using install_agent.sh)
I ran Add Data script on machine2 and machine3.

I can see metrics data from all three machine, However we can see the entity only for machine1. Machine2 and Machine3 entities are not displaying.

0 Karma

Splunk Employee
Splunk Employee

This might be due to lag in data coming in ..

Can you try to update collectors.conf in SAI?

Change the "monitoring_calculation_window" to 180 for "os" and restart Splunk..

0 Karma

Path Finder
0 Karma

New Member

looks like its resolved the issue after the below value to 180
monitoring_calculation_window = 180.

Can you please let me know why this happened? whether changing the value on the conf file is recommended?

Also, what could be the reason if status of entity shows inactive?

0 Karma

New Member

also, found the below warning in Machine2 and Machine3 collectord.log
[warning] write splunk plugin: failed to add os version as dimension. ignoring..

Is there way to resolve?

0 Karma

Splunk Employee
Splunk Employee

Not a big issue.. SAI just adds some extra dimensions like os_version and it could not find it for your machine..

You can also add it in collectd.conf manually if you really need it.

0 Karma

New Member

Hi,

What is the procedure to troubleshoot the entity is inactive, however we can see metrics data for the entity.

0 Karma

Splunk Employee
Splunk Employee

That might be due to data lag..
Fix the lag or increase the monitoring window further..

0 Karma

Splunk Employee
Splunk Employee

SAI was looking 90sec in past to see if any new entities was found. For some reason, your Machine 2 and Machine 3 has data lag of more than 90 sec. That's why we doubled the time to 180sec. You can keep it at 180s without issues. It shows inactive when it didn't see any new data in the monitoring window..

Maybe figure why there is lag as well and fix it as well...

0 Karma

New Member

Thank you so much for your help.

One of the possible reason: These are my test servers, its not sync with NTP server. Looks like time mismatch.

0 Karma

Splunk Employee
Splunk Employee

Can you clarify your setup more:

Machine 2 is running SAI(Splunk app for infrastructure) and SAI Add-on?
Did you use script on "Add Data" page of the app? Are you trying to monitor Machine 1? Is it Windows or Linux?

Path Finder

My both machines are Linux, SAI is installed on Machine 1(Server Machine) with SAI Add-on and now I am trying to install UF on Machine 2(Client Machine) with the help of script from "Add Data" page of the App. So, I am trying to monitor Machine 2

0 Karma

Splunk Employee
Splunk Employee

Does the splunkd.log from the UF say anything about whether the data is successfully sending to Machine 1?

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!