Splunk App for Infrastructure (SAI) missing entity

[jsetzer]

We have deployed SAI and the corresponding UF ps scripts to over 30 Windows systems without issue.... However we have one Windows server that is missing from the SAI Entities dashboard view. I confirmed we are collecting metrics from the system via | mstats and | mcollect on our SH and see the hostname within the content of the metric index and metrics counts. I also confirmed we are getting log/windows data from the system. I have attempted to reinstall UF and corresponding SAI app to no avail. I attempted modifying the KV store for entities and add this system in manually but that also did not work. I confirmed the instances.cfg on the client windows server differs from all other systems. Im out of ideas has anyone else run into this or can recommend something to try??

[dagarwal_splunk · Accepted solution]

First try this search:
| mcatalog values("entity_type") as "entity_type" values("os") as "os" WHERE metric_name=processor.* AND index=em_metrics BY "host"

Do you see your missing server/entity ? If no, try this:
| mstats avg(_value) WHERE index=em_metrics AND metric_name=* AND host="missing_server_name" by metric_name

Replace missing_server_name in this search.

[jsetzer]

|mcatalog search contained the missing server - reported entity_type and OS are identical to other servers reporting that do show up in the dashboard

| mstats search for just the missing server returned 25 metric_name results base on 300K events over a 24 hour span

[dagarwal_splunk]

Can you check splunk log files here for errors:
$SPLUNK_HOME/var/log/splunk/em_entity_manager.log (check other log files as well if you see any errors)

ALso try to see if the missing host is present in the kv store:
| inputlookup em_entities

To run this search, go to splunk_web:8000/app/splunk_app_infrastructure/search
{i.e run search in context of SAI}

[jsetzer]

The host in question does not show up in the em_entities kvstore

Looking on the client and splunk system for errors associated with SAI and host found nothing

The only place where the host shows up was mentioned above in the metrics indexes.

[jsetzer]

Resolved ! the problem wasnt splunk but there server itself the time was not sync'd 3 min difference between it and Splunk once it was updated the machine appears in the dashboard 🙂

[dagarwal_splunk]

Great !!!!