Splunk IT Service Intelligence

Howto get a list of ITSI services with source of KPI base search

Path Finder

Hello,

In my environment, I have a long list of ITSI services (created by someone else) which using default KPI base search. These default KPI base search is running every mins for 1 min data and it has causes some impact to the indexers.

Without going through the UI for ITSI services and checking the search source for each KPI, is there any query I can use to generate the list of the ITSI services + KPI base search + base search source name?

Thanks.

Tags (1)
1 Solution

Communicator

Hi

There is this lookup included in ITSI that includes all info for each KPI defined:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name

If you want it separated:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name kpis.* as kpis_*
| eval kpis_info = mvzip(mvzip(mvzip(kpis__key, kpis_base_search, "==@@=="), kpis_search_type, "==@@=="), kpis_title, "==@@==")
| fields service_name service_id kpis_info
| mvexpand kpis_info
| eval kpis_info=split(kpis_info, "==@@==")
| eval kpis__key=mvindex(kpis_info, 0)
| eval kpis_base_search=mvindex(kpis_info, 1)
| eval kpis_search_type=mvindex(kpis_info, 2)
| eval kpis_title=mvindex(kpis_info, 3)
| fields - kpis_info

You could also try searching the splunk scheduler log or a version of the KPI Performance Panel included in the ITSI Healthcheck Dashboard:

index=_internal sourcetype=scheduler savedsearch_name="Indicator*"
            | stats count as run_count, count(eval(status!="success")) as failed_count, count(eval(suppressed!="0")) as suppressed_count,
            avg(run_time) as avg_runtime, max(run_time) as max_runtime, earliest(_time) as first, latest(_time) as last
            by savedsearch_name
            | eval KPI_search_type=if(savedsearch_name like "%Shared%", "base", "ad hoc")
            | eval runtime_headroom_pct=round((100-(max_runtime/((last-first)/(run_count-1))*100)),1)
            | eval avg_runtime=round(avg_runtime, 2)
            | eval max_runtime=round(max_runtime, 2)
            | table savedsearch_name KPI_search_type failed_count suppressed_count runtime_headroom_pct avg_runtime max_runtime run_count
            | sort +runtime_headroom_pct

Hope it helps. Cheers!!!

View solution in original post

Communicator

Hi

There is this lookup included in ITSI that includes all info for each KPI defined:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name

If you want it separated:

| inputlookup service_kpi_sbs_lookup
| rename _key as service_id title as service_name kpis.* as kpis_*
| eval kpis_info = mvzip(mvzip(mvzip(kpis__key, kpis_base_search, "==@@=="), kpis_search_type, "==@@=="), kpis_title, "==@@==")
| fields service_name service_id kpis_info
| mvexpand kpis_info
| eval kpis_info=split(kpis_info, "==@@==")
| eval kpis__key=mvindex(kpis_info, 0)
| eval kpis_base_search=mvindex(kpis_info, 1)
| eval kpis_search_type=mvindex(kpis_info, 2)
| eval kpis_title=mvindex(kpis_info, 3)
| fields - kpis_info

You could also try searching the splunk scheduler log or a version of the KPI Performance Panel included in the ITSI Healthcheck Dashboard:

index=_internal sourcetype=scheduler savedsearch_name="Indicator*"
            | stats count as run_count, count(eval(status!="success")) as failed_count, count(eval(suppressed!="0")) as suppressed_count,
            avg(run_time) as avg_runtime, max(run_time) as max_runtime, earliest(_time) as first, latest(_time) as last
            by savedsearch_name
            | eval KPI_search_type=if(savedsearch_name like "%Shared%", "base", "ad hoc")
            | eval runtime_headroom_pct=round((100-(max_runtime/((last-first)/(run_count-1))*100)),1)
            | eval avg_runtime=round(avg_runtime, 2)
            | eval max_runtime=round(max_runtime, 2)
            | table savedsearch_name KPI_search_type failed_count suppressed_count runtime_headroom_pct avg_runtime max_runtime run_count
            | sort +runtime_headroom_pct

Hope it helps. Cheers!!!

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!