Splunk IT Service Intelligence

Why is the itsi_event_grouping scheduled search is always being skipped?

rphillips_splk
Splunk Employee
Splunk Employee

I have a SHC on 6.4.1 and always see the itsi_event_grouping scheduled search skipped in scheduler.log.

/etc/apps/SA-ITOA/default/savedsearches.conf
Search to group events ###
[itsi_event_grouping]
cron_schedule = * * * * *
disabled = 0
dispatch.earliest_time = rt
dispatch.indexedRealtime = 1
dispatch.latest_time = rt
enableSched = 1
search = itsi_event_management_index | where isnull(itsi_is_edited) | spath | fields - _raw | itsirulesengine | where 1=2

Tags (3)

Jarohnimo
Builder

Hi rPhillips,

Where did you get that information? Seeing how when we search google/ splunk on ITSI Event Grouping there is literally NOTHING. I would like to just turn this off if I can't figure out what the benefit is?

This message is being fired off on the indexer (not the WFE) so Im not sure if i need to have this enabled. Please let me know

0 Karma

Jarohnimo
Builder

I have this exact same problem. Only occurs on the indexer, the search head seems to work fine. The indexer has alot of issues (it was the previous ITSI box) but now that i have a Dedicated sh it runs from there...

Dirty move, i simply deleted the services once i migrated the content to a new box. wasn't sure if that was the "proper" way to ridding the content but i got a feeling i need to turn something off on the indexer

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

The itsi_event_grouping search is a real-time search. Because it runs forever subsequent search instances that are spawned by the cron are skipped. There can only be one occurence of the real-time search running at any given time. This is expected and is not an issue because the search is actually running as you can see in resource_usage.log:

$SPLUNK_HOME/var/log/introspection
tail -f resource_usage.log | grep rt_

In 6.4.5+ and 6.5.1+ the skipped message 'reason' has been improved to clarify this and modified to: "The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached"

You will now see an event like this in scheduler.log when a real-time search is skipped:

11-09-2016 05:49:03.443 -0800 INFO SavedSplunker - savedsearch_id="nobody;search;search1", user="nobody", app="search", savedsearch_name="search1", status=skipped, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1478699340, window_time=0

Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...