All Apps and Add-ons

Splunk App for Infrastructure (SAI) missing entity

jsetzer
New Member

We have deployed SAI and the corresponding UF ps scripts to over 30 Windows systems without issue.... However we have one Windows server that is missing from the SAI Entities dashboard view. I confirmed we are collecting metrics from the system via | mstats and | mcollect on our SH and see the hostname within the content of the metric index and metrics counts. I also confirmed we are getting log/windows data from the system. I have attempted to reinstall UF and corresponding SAI app to no avail. I attempted modifying the KV store for entities and add this system in manually but that also did not work. I confirmed the instances.cfg on the client windows server differs from all other systems. Im out of ideas has anyone else run into this or can recommend something to try??

0 Karma
1 Solution

dagarwal_splunk
Splunk Employee
Splunk Employee

First try this search:
| mcatalog values("entity_type") as "entity_type" values("os") as "os" WHERE metric_name=processor.* AND index=em_metrics BY "host"

Do you see your missing server/entity ? If no, try this:
| mstats avg(_value) WHERE index=em_metrics AND metric_name=* AND host="missing_server_name" by metric_name

Replace missing_server_name in this search.

View solution in original post

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

First try this search:
| mcatalog values("entity_type") as "entity_type" values("os") as "os" WHERE metric_name=processor.* AND index=em_metrics BY "host"

Do you see your missing server/entity ? If no, try this:
| mstats avg(_value) WHERE index=em_metrics AND metric_name=* AND host="missing_server_name" by metric_name

Replace missing_server_name in this search.

0 Karma

jsetzer
New Member

|mcatalog search contained the missing server - reported entity_type and OS are identical to other servers reporting that do show up in the dashboard

| mstats search for just the missing server returned 25 metric_name results base on 300K events over a 24 hour span

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Can you check splunk log files here for errors:
$SPLUNK_HOME/var/log/splunk/em_entity_manager.log (check other log files as well if you see any errors)

ALso try to see if the missing host is present in the kv store:
| inputlookup em_entities

To run this search, go to splunk_web:8000/app/splunk_app_infrastructure/search
{i.e run search in context of SAI}

0 Karma

jsetzer
New Member

The host in question does not show up in the em_entities kvstore

Looking on the client and splunk system for errors associated with SAI and host found nothing

The only place where the host shows up was mentioned above in the metrics indexes.

0 Karma

jsetzer
New Member

Resolved ! the problem wasnt splunk but there server itself the time was not sync'd 3 min difference between it and Splunk once it was updated the machine appears in the dashboard 🙂

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Great !!!!

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...