Splunk IT Service Intelligence

If an ITSI KPI scheduled search is skipped/continued how do I know which KPI / Service is impacted?

rphillips_splk
Splunk Employee
Splunk Employee

Seeing skipped searches in scheduler.log is a common problem faced especially if you've scheduled too many concurrent searches or in ITSI land too many KPIs. Is there an easy way to map service id, kpi id , service name and saved search name ? When you create a KPI in ITSI a scheduled search is created with a name like "Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search".

scheduler.log
04-20-2018 03:52:01.745 +0000 INFO SavedSplunker - savedsearch_id="nobody;itsi;Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", search_type="scheduled", user="nobody", app="itsi", savedsearch_name="Indicator - Shared - 5ad7aa9759479309e84b84a2 - ITSI Search", priority=default, status=success, digest_mode=1, scheduled_time=1524196320, window_time=0, dispatch_time=1524196320, run_time=0.946, result_count=2, alert_actions="indicator", sid="scheduler_nobodyitsi_RMD5e3cb0aa7dd5f97bc_at_1524196320_9870", suppressed=0, thread_id="AlertNotifierWorker-0"

If these KPI scheduled searches start skipping , my itsi_summary index will be missing data. How do I know which kpi name & service is impacted?

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

this search (tested in ITSI 3.0.0):
| inputlookup service_kpi_lookup | rename _key as serviceid title as service_name | eval kpi_info = mvzip('kpis._key', 'kpis.title', "==@@==") | fields kpi_info service_name serviceid | mvexpand kpi_info | rex field=kpi_info "(?<kpiid>.+)==@@==(?<kpi_name>.+)" | fields - kpi_info | join kpiid [search index=itsi_summary | dedup search_name |rename search_name as savedsearch_name] | table service_name serviceid kpi_name kpiid savedsearch_name

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...