Splunk ITSI

ITSI: How to create services from search results in N/A services?

YoungDaniel
Path Finder

Hi,
We are running ITSI on a dedicated Search head running 16 cores cpu , 23 gb RAM and about 150 gb disk. It's searching an Index Cluster serving a SH Cluster and two stand alone search heads.

We have about 400 services with 3-5 kpis each. The majority of these services have been created by import through search and mass produced. They have all worked fine. However, when we try to add more services, from a specific search, we are getting NaN for each KPI and ServiceHealthScore. We have removed backfilling on the service KPIs and set static thresholds. The skipped search rate is very low,( 1,34%, 66 searches) But the new services (138) all are missing values. When the service is opened up in deep dive and summary index turned off, we get data for the entirety of kpis. The Kpis use stats sum and count.
We turned up the
[kvstore]
max_size_per_batch_save_mb = 100

In limits.conf
Our ITSI_backfill is at 49 mb. we are not using Backfill on these services/kpis so that should not be the problem.

So I am wondering two things,
How does ITSI create the services and Kpis from search?
How do I reload a service and its kpis?

BR / Daniel

Labels (3)
0 Karma

Raja
Engager

Hello Daniel,

Can you help me with search query to create services with dependencies with service template assign.

 

0 Karma

ansif
Motivator

We had same issue ,when data fetches from summary index it shows no data but when disabling Summary Index it works.

Our problem resolved by adding more roles to admin.Try to play around roles and check if the serach works with Summary.

try to add all app_admin and user.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...