Splunk ITSI

Howto get a list of ITSI services with source of KPI base search

krusovice
Path Finder

Hello,

In my environment, I have a long list of ITSI services (created by someone else) which using default KPI base search. These default KPI base search is running every mins for 1 min data and it has causes some impact to the indexers.

Without going through the UI for ITSI services and checking the search source for each KPI, is there any query I can use to generate the list of the ITSI services + KPI base search + base search source name?

Thanks.

Tags (1)
1 Solution

jaime_ramirez
Communicator

Hi

There is this lookup included in ITSI that includes all info for each KPI defined:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name

If you want it separated:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name kpis.* as kpis_*
| eval kpis_info = mvzip(mvzip(mvzip(kpis__key, kpis_base_search, "==@@=="), kpis_search_type, "==@@=="), kpis_title, "==@@==")
| fields service_name service_id kpis_info
| mvexpand kpis_info
| eval kpis_info=split(kpis_info, "==@@==")
| eval kpis__key=mvindex(kpis_info, 0)
| eval kpis_base_search=mvindex(kpis_info, 1)
| eval kpis_search_type=mvindex(kpis_info, 2)
| eval kpis_title=mvindex(kpis_info, 3)
| fields - kpis_info

You could also try searching the splunk scheduler log or a version of the KPI Performance Panel included in the ITSI Healthcheck Dashboard:

index=_internal sourcetype=scheduler savedsearch_name="Indicator*"
            | stats count as run_count, count(eval(status!="success")) as failed_count, count(eval(suppressed!="0")) as suppressed_count,
            avg(run_time) as avg_runtime, max(run_time) as max_runtime, earliest(_time) as first, latest(_time) as last
            by savedsearch_name
            | eval KPI_search_type=if(savedsearch_name like "%Shared%", "base", "ad hoc")
            | eval runtime_headroom_pct=round((100-(max_runtime/((last-first)/(run_count-1))*100)),1)
            | eval avg_runtime=round(avg_runtime, 2)
            | eval max_runtime=round(max_runtime, 2)
            | table savedsearch_name KPI_search_type failed_count suppressed_count runtime_headroom_pct avg_runtime max_runtime run_count
            | sort +runtime_headroom_pct

Hope it helps. Cheers!!!

View solution in original post

a1bg503461
Explorer

I want to download the service_kpi_sbs_lookup.csv file and import it in another test splunk whit Lookup Editor. Does anyone know where is service_kpi_sbs_lookup.csv under linux ? Thanks

0 Karma

jaime_ramirez
Communicator

Hi

There is this lookup included in ITSI that includes all info for each KPI defined:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name

If you want it separated:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name kpis.* as kpis_*
| eval kpis_info = mvzip(mvzip(mvzip(kpis__key, kpis_base_search, "==@@=="), kpis_search_type, "==@@=="), kpis_title, "==@@==")
| fields service_name service_id kpis_info
| mvexpand kpis_info
| eval kpis_info=split(kpis_info, "==@@==")
| eval kpis__key=mvindex(kpis_info, 0)
| eval kpis_base_search=mvindex(kpis_info, 1)
| eval kpis_search_type=mvindex(kpis_info, 2)
| eval kpis_title=mvindex(kpis_info, 3)
| fields - kpis_info

You could also try searching the splunk scheduler log or a version of the KPI Performance Panel included in the ITSI Healthcheck Dashboard:

index=_internal sourcetype=scheduler savedsearch_name="Indicator*"
            | stats count as run_count, count(eval(status!="success")) as failed_count, count(eval(suppressed!="0")) as suppressed_count,
            avg(run_time) as avg_runtime, max(run_time) as max_runtime, earliest(_time) as first, latest(_time) as last
            by savedsearch_name
            | eval KPI_search_type=if(savedsearch_name like "%Shared%", "base", "ad hoc")
            | eval runtime_headroom_pct=round((100-(max_runtime/((last-first)/(run_count-1))*100)),1)
            | eval avg_runtime=round(avg_runtime, 2)
            | eval max_runtime=round(max_runtime, 2)
            | table savedsearch_name KPI_search_type failed_count suppressed_count runtime_headroom_pct avg_runtime max_runtime run_count
            | sort +runtime_headroom_pct

Hope it helps. Cheers!!!

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...