Splunk IT Service Intelligence

If an ITSI KPI scheduled search is skipped/continued how do I know which KPI / Service is impacted?

rphillips_splk
Splunk Employee
Splunk Employee

Seeing skipped searches in scheduler.log is a common problem faced especially if you've scheduled too many concurrent searches or in ITSI land too many KPIs. Is there an easy way to map service id, kpi id , service name and saved search name ? When you create a KPI in ITSI a scheduled search is created with a name like "Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search".

scheduler.log
04-20-2018 03:52:01.745 +0000 INFO SavedSplunker - savedsearch_id="nobody;itsi;Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", search_type="scheduled", user="nobody", app="itsi", savedsearch_name="Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", priority=default, status=success, digest_mode=1, scheduled_time=1524196320, window_time=0, dispatch_time=1524196320, run_time=0.946, result_count=2, alert_actions="indicator", sid="scheduler_nobodyitsi_RMD5e3cb0aa7dd5f97bc_at_1524196320_9870", suppressed=0, thread_id="AlertNotifierWorker-0"

If these KPI scheduled searches start skipping , my itsi_summary index will be missing data. How do I know which kpi name & service is impacted?

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...