All Apps and Add-ons

Is there a Reference point for Splunk ITSI Infrastructure Overview status for Entities?

vijaybaskarss
Loves-to-Learn Lots

Hello,

The Infrastructure overview in Splunk ITSI shows entities list like active, unstable, inactive and N/A. Can you help me what is reference point for all these status, in our environment it is showing many in N/A and unstable. But we are still receiving data for whichever showing N/A and unstable, also added recurring import using available modules. But still that is not reflecting as active.

Please advise.

Regards,

Vj

Labels (2)
Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

My understanding that you have entities detected in ITSI, in version 4.9.*
and some of those entities status are often flagged as "unstable" or "inactive" 

The logic to change the status of an entity if that when the scheduled import ran, it did no find data for that entity (for the health metric)  in the timerange of the search. It will flag it as "unstable" if the entity is not constantly detected, and "inactive" if this is constant.
see https://docs.splunk.com/Documentation/ITSI/4.9.3/Entity/InfraOverview#Monitor_entity_status

Out of the box, the ITSI entity import are very frequent and aggressive, this also may impact the detection
see this remark in the documentation
>Note:If you have a large number of entities, the recurring bulk import can take a longer time to complete. Tune the cron schedule of the recurring import searches to search less frequently in order to ensure your entity status updates on time.

By example for the Windows entities, the scheduled saved search doing the import is called "ITSI Import Objects - Perfmon"
It looks for key metric "metric_name=Processor.* OR metric_name=processor.*"
And it runs every minute, and look back 90seconds.

So the root cause for unstable entities may be :

  • the host matching this entity is not sending data consistently
    • To address this you want to check the data ingestion and frequency (maybe send data more frequently)
  • Or the data has lag, therefore is out of the search window.
    • To address it, measure your average lag, and decide if you can improve or adjust the search windows to account for that lag.
  • or the search used for the import has a too short timerange, that does not mach the metric collection interval.
    • To address this one, you could change the import search to run less often, but look back for longer period.

By example for the windows entities, if you collect perfmon data every 5 minutes, with an average lag of 1 minute, change the search to run every 5m and look back maybe 7 minutes to account for delay.

yannK
Splunk Employee
Splunk Employee

Addendum in 2023 :

It has been discovered an additional root cause that can lead to unstable entities.

If the entity has been discovered and updated by more then 1 import search, each import search will keep a record of the last update, and may cause the "inventory" page to consider the entities "unstable" if there too much difference between the different imports dates.

Also in the content pack for monitoring, an extra import search "normalizer" does also update the entities.

Long term solution will come in future ITSI ( after ITSI 4.16)

0 Karma

andrewtrobec
Motivator

@yannK I am also facing an identical issue with entity import in ITSI in which I am getting confirmed data every 5 minutes but for some reason the Status remains "N/A" and the Last Update remains "Invalid Date".  I have recreated the entire structure but it did not resolve, which is making things even more frustrating.

To set up I did the following:

  1. Create the Entity Type, providing only the name and saving
  2. Follow the Create Entity -> Import from Search Wizard
  3. Create the Import Object recurring search

The entities were imported the first time around through the wizard, but when the Import Object search runs it does not update the entities.  Since I currently have data coming in every 5 minutes, as overkill I scheduled the Import Objects search to run every minute and go back 24 hours in time.  There is loads of data available.  Also, when I inspect the search once it has run it gives no errors, and if I access it I can see the results!a

I don't understand why this is not working.  I have additional Entity Types set up which work without issue.  Do you have any idea?

Best regards,

Andrew

0 Karma

andrewtrobec
Motivator

@yannK After much suffering I have basically discovered that the problem is due to a "." (period) character in the Entity Title.  It gets more painful though.  In order for the Object Import search to work I have to create a separate Entity Type with the same name but with a " " (space) characted instead of a "." (period) character as well as create an Object Import search.  Once this has been done, both Entity Types get updated even though I have a single Object Import search.

Using ITSI Version:4.11.5 Build:22263.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...