Entities not displayed in Splunk App for Infrastructure.

[himanshutejwani]

I have set up a Universal Forwarder(UF) from the script on Machine 2 but UF is not added on Splunk Enterprise(Machine 1).
I have manually added the deployment server and in this case, the UF is added on Splunk Enterprise but the entity is not displayed on Splunk App for Infrastructure for which I have waited for more than 5 mins.

Followed the below link to install SAI on Splunk Enterprise:
https://docs.splunk.com/Documentation/InfraApp/2.0.1/Install/Install

[dagarwal_splunk]

To conclude the steps for resolving Metrics data (collectd) collection issues:
(NOTE: machine2 is the monitored Linux machine i.e running collectd and Splunk UF; machine 1 is running SAI And SAI Add-on)

1. Check if collectd running or installed on monitored Machine 2..
   apt-cache policy collectd
   ps -ef | grep collectd

2. Check Metrics data coming in: | mcatalog values(metric_name) WHERE host=${Machine 2} AND index=em_metrics

3. Machine2: Do you see any recurring errors like "curl_easy_perform failed" in collectd.log or any other error?

4. In Machine 1, Check if all the Hec tokens are enabled: Settings -> Data Inputs ->HTTP Event Collector

5. Machine 1: Check the Global Settings on the same page as 2. Verify "enable ssl" is checked, "Use Deployment Server" unchecked and note down the port number.

6. Machine 1: Verify the HEC token you are using has default index as "em_metrics"

7. Now In Machine 2, check /etc/collectd/collectd.conf file. Verify that HEC token, server and port number in write_splunk stanza is correct.

8. Try sending fake data from Machine 2 to Machine 1 using curl and see if you get success. Here is the curl command you need to run in Machine 2:
   curl -k https://Machine1:8088/services/collector -H "Authorization: Splunk hec_token_here" -d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

You should see "Success" Message. If not, try to fix the error message that you get.
https://docs.splunk.com/Documentation/Splunk/7.3.3/Metrics/GetMetricsInOther#Example_of_sending_metr...
Update token, port and server in the command

[3es]

I have followed the above steps and I can able to test fake data and getting success message. However still I am not able to see Machine 2 in Entities.

[dagarwal_splunk]

What are the search results for Step 2?

[3es]

Refer below output.

| mcatalog values(metric_name) WHERE host=xxxxx AND index=em_metrics
cpu.idle
cpu.interrupt
cpu.nice
cpu.softirq
cpu.steal
cpu.system
cpu.user
cpu.wait
df.free
df.reserved
df.used
disk.io_time.io_time
disk.io_time.weighted_io_time
disk.merged.read
disk.merged.write
disk.octets.read
disk.octets.write
disk.ops.read
disk.ops.write
disk.pending_operations
disk.time.read
disk.time.write
interface.dropped.rx
interface.dropped.tx
interface.errors.rx
interface.errors.tx
interface.octets.rx
interface.octets.tx
interface.packets.rx
interface.packets.tx
load.longterm
load.midterm
load.shortterm
memory.buffered
memory.cached
memory.free
memory.slab_recl
memory.slab_unrecl
memory.used
uptime.uptime

[dagarwal_splunk]

you do have metrics data in Splunk.. what version of SAI do you have? Do you have any entities in SAI?

[3es]

SAI version:
description = Splunk App for Infrastructure
version = 2.0.2
build = 10

My Splunk version: Splunk 8.0.0 (build 1357bef0a7f6)

I can see only one entity which is same server of index/SH (I have installed Splunk all in one server: machine1)

Also,

I can see the metrics data in Splunk from the server which has UF.
Sample command:
| mstats count where metric_name=cpu.idle host=* BY metric_name, host

Result: last 15 Mins
metric_name host count
cpu.idle machine1 14 (SH/Index)
cpu.idle machine2 13 (UF 8.0.1)
cpu.idle machine3 12 (7.2.4)

UF version:
Splunk Universal Forwarder 8.0.1 (build 6db836e2fb9e)
and
Splunk Universal Forwarder 7.2.4 (build 8a94541dcfac)

[dagarwal_splunk]

Metrics data is sent by collectd not UF for Linux machines..

So, I guess you ran the "Add Data" script on 3 machines and you only see 1 entity. But, you have metrics data from all 3 machines?

[3es]

In Machine 1, I have installed SAI App and Addon and configured collectd (using install_agent.sh)
I ran Add Data script on machine2 and machine3.

I can see metrics data from all three machine, However we can see the entity only for machine1. Machine2 and Machine3 entities are not displaying.

[dagarwal_splunk]

This might be due to lag in data coming in ..

Can you try to update collectors.conf in SAI?

Change the "monitoring_calculation_window" to 180 for "os" and restart Splunk..

[shandr]

SAI 2.1.0+ has replaced collectors.conf with entity_classes.conf

Refer to https://docs.splunk.com/Documentation/InfraApp/2.1.0/Install/NewEntitySchema

For Windows [perfmon] see https://community.splunk.com/t5/forums/replypage/board-id/apps-add-ons-all/message-id/59094

[3es]

looks like its resolved the issue after the below value to 180
monitoring_calculation_window = 180.

Can you please let me know why this happened? whether changing the value on the conf file is recommended?

Also, what could be the reason if status of entity shows inactive?

[3es]

also, found the below warning in Machine2 and Machine3 collectord.log
[warning] write splunk plugin: failed to add os version as dimension. ignoring..

Is there way to resolve?

[dagarwal_splunk]

Not a big issue.. SAI just adds some extra dimensions like os_version and it could not find it for your machine..

You can also add it in collectd.conf manually if you really need it.

[3es]

Hi,

What is the procedure to troubleshoot the entity is inactive, however we can see metrics data for the entity.

[dagarwal_splunk]

That might be due to data lag..
Fix the lag or increase the monitoring window further..

[dagarwal_splunk]

SAI was looking 90sec in past to see if any new entities was found. For some reason, your Machine 2 and Machine 3 has data lag of more than 90 sec. That's why we doubled the time to 180sec. You can keep it at 180s without issues. It shows inactive when it didn't see any new data in the monitoring window..

Maybe figure why there is lag as well and fix it as well...

[3es]

Thank you so much for your help.

One of the possible reason: These are my test servers, its not sync with NTP server. Looks like time mismatch.

[dagarwal_splunk]

Can you clarify your setup more:

Machine 2 is running SAI(Splunk app for infrastructure) and SAI Add-on?
Did you use script on "Add Data" page of the app? Are you trying to monitor Machine 1? Is it Windows or Linux?

[himanshutejwani]

My both machines are Linux, SAI is installed on Machine 1(Server Machine) with SAI Add-on and now I am trying to install UF on Machine 2(Client Machine) with the help of script from "Add Data" page of the App. So, I am trying to monitor Machine 2

[pwu_splunk · Accepted solution]

Does the splunkd.log from the UF say anything about whether the data is successfully sending to Machine 1?