Splunk IT Service Intelligence

Why is the itsi_event_grouping scheduled search is always being skipped?

Splunk Employee
Splunk Employee

I have a SHC on 6.4.1 and always see the itsi_event_grouping scheduled search skipped in scheduler.log.

Search to group events ###
cron_schedule = * * * * *
disabled = 0
dispatch.earliest_time = rt
dispatch.indexedRealtime = 1
dispatch.latest_time = rt
enableSched = 1
search = itsi_event_management_index | where isnull(itsi_is_edited) | spath | fields - _raw | itsirulesengine | where 1=2

Tags (3)


Hi rPhillips,

Where did you get that information? Seeing how when we search google/ splunk on ITSI Event Grouping there is literally NOTHING. I would like to just turn this off if I can't figure out what the benefit is?

This message is being fired off on the indexer (not the WFE) so Im not sure if i need to have this enabled. Please let me know

0 Karma


I have this exact same problem. Only occurs on the indexer, the search head seems to work fine. The indexer has alot of issues (it was the previous ITSI box) but now that i have a Dedicated sh it runs from there...

Dirty move, i simply deleted the services once i migrated the content to a new box. wasn't sure if that was the "proper" way to ridding the content but i got a feeling i need to turn something off on the indexer

0 Karma

Splunk Employee
Splunk Employee

The itsi_event_grouping search is a real-time search. Because it runs forever subsequent search instances that are spawned by the cron are skipped. There can only be one occurence of the real-time search running at any given time. This is expected and is not an issue because the search is actually running as you can see in resource_usage.log:

tail -f resource_usage.log | grep rt_

In 6.4.5+ and 6.5.1+ the skipped message 'reason' has been improved to clarify this and modified to: "The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached"

You will now see an event like this in scheduler.log when a real-time search is skipped:

11-09-2016 05:49:03.443 -0800 INFO SavedSplunker - savedsearch_id="nobody;search;search1", user="nobody", app="search", savedsearch_name="search1", status=skipped, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1478699340, window_time=0

Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...