Splunk IT Service Intelligence

If an ITSI KPI scheduled search is skipped/continued how do I know which KPI / Service is impacted?

rphillips_splk
Splunk Employee
Splunk Employee

Seeing skipped searches in scheduler.log is a common problem faced especially if you've scheduled too many concurrent searches or in ITSI land too many KPIs. Is there an easy way to map service id, kpi id , service name and saved search name ? When you create a KPI in ITSI a scheduled search is created with a name like "Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search".

scheduler.log
04-20-2018 03:52:01.745 +0000 INFO SavedSplunker - savedsearch_id="nobody;itsi;Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", search_type="scheduled", user="nobody", app="itsi", savedsearch_name="Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", priority=default, status=success, digest_mode=1, scheduled_time=1524196320, window_time=0, dispatch_time=1524196320, run_time=0.946, result_count=2, alert_actions="indicator", sid="scheduler_nobodyitsi_RMD5e3cb0aa7dd5f97bc_at_1524196320_9870", suppressed=0, thread_id="AlertNotifierWorker-0"

If these KPI scheduled searches start skipping , my itsi_summary index will be missing data. How do I know which kpi name & service is impacted?

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...