All Apps and Add-ons

Is there a Reference point for Splunk ITSI Infrastructure Overview status for Entities?

vijaybaskarss
Loves-to-Learn Lots

Hello,

The Infrastructure overview in Splunk ITSI shows entities list like active, unstable, inactive and N/A. Can you help me what is reference point for all these status, in our environment it is showing many in N/A and unstable. But we are still receiving data for whichever showing N/A and unstable, also added recurring import using available modules. But still that is not reflecting as active.

Please advise.

Regards,

Vj

Labels (2)
Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

My understanding that you have entities detected in ITSI, in version 4.9.*
and some of those entities status are often flagged as "unstable" or "inactive" 

The logic to change the status of an entity if that when the scheduled import ran, it did no find data for that entity (for the health metric)  in the timerange of the search. It will flag it as "unstable" if the entity is not constantly detected, and "inactive" if this is constant.
see https://docs.splunk.com/Documentation/ITSI/4.9.3/Entity/InfraOverview#Monitor_entity_status

Out of the box, the ITSI entity import are very frequent and aggressive, this also may impact the detection
see this remark in the documentation
>Note:If you have a large number of entities, the recurring bulk import can take a longer time to complete. Tune the cron schedule of the recurring import searches to search less frequently in order to ensure your entity status updates on time.

By example for the Windows entities, the scheduled saved search doing the import is called "ITSI Import Objects - Perfmon"
It looks for key metric "metric_name=Processor.* OR metric_name=processor.*"
And it runs every minute, and look back 90seconds.

So the root cause for unstable entities may be :

  • the host matching this entity is not sending data consistently
    • To address this you want to check the data ingestion and frequency (maybe send data more frequently)
  • Or the data has lag, therefore is out of the search window.
    • To address it, measure your average lag, and decide if you can improve or adjust the search windows to account for that lag.
  • or the search used for the import has a too short timerange, that does not mach the metric collection interval.
    • To address this one, you could change the import search to run less often, but look back for longer period.

By example for the windows entities, if you collect perfmon data every 5 minutes, with an average lag of 1 minute, change the search to run every 5m and look back maybe 7 minutes to account for delay.

0 Karma

andrewtrobec
Builder

@yannK I am also facing an identical issue with entity import in ITSI in which I am getting confirmed data every 5 minutes but for some reason the Status remains "N/A" and the Last Update remains "Invalid Date".  I have recreated the entire structure but it did not resolve, which is making things even more frustrating.

To set up I did the following:

  1. Create the Entity Type, providing only the name and saving
  2. Follow the Create Entity -> Import from Search Wizard
  3. Create the Import Object recurring search

The entities were imported the first time around through the wizard, but when the Import Object search runs it does not update the entities.  Since I currently have data coming in every 5 minutes, as overkill I scheduled the Import Objects search to run every minute and go back 24 hours in time.  There is loads of data available.  Also, when I inspect the search once it has run it gives no errors, and if I access it I can see the results!a

I don't understand why this is not working.  I have additional Entity Types set up which work without issue.  Do you have any idea?

Best regards,

Andrew

0 Karma

andrewtrobec
Builder

@yannK After much suffering I have basically discovered that the problem is due to a "." (period) character in the Entity Title.  It gets more painful though.  In order for the Object Import search to work I have to create a separate Entity Type with the same name but with a " " (space) characted instead of a "." (period) character as well as create an Object Import search.  Once this has been done, both Entity Types get updated even though I have a single Object Import search.

Using ITSI Version:4.11.5 Build:22263.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...