Splunk IT Service Intelligence

If an ITSI KPI scheduled search is skipped/continued how do I know which KPI / Service is impacted?

rphillips_splk
Splunk Employee
Splunk Employee

Seeing skipped searches in scheduler.log is a common problem faced especially if you've scheduled too many concurrent searches or in ITSI land too many KPIs. Is there an easy way to map service id, kpi id , service name and saved search name ? When you create a KPI in ITSI a scheduled search is created with a name like "Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search".

scheduler.log
04-20-2018 03:52:01.745 +0000 INFO SavedSplunker - savedsearch_id="nobody;itsi;Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", search_type="scheduled", user="nobody", app="itsi", savedsearch_name="Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", priority=default, status=success, digest_mode=1, scheduled_time=1524196320, window_time=0, dispatch_time=1524196320, run_time=0.946, result_count=2, alert_actions="indicator", sid="scheduler_nobodyitsi_RMD5e3cb0aa7dd5f97bc_at_1524196320_9870", suppressed=0, thread_id="AlertNotifierWorker-0"

If these KPI scheduled searches start skipping , my itsi_summary index will be missing data. How do I know which kpi name & service is impacted?

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...