Why is the itsi_event_grouping scheduled search is always being skipped?

[rphillips_splk]

I have a SHC on 6.4.1 and always see the itsi_event_grouping scheduled search skipped in scheduler.log.

/etc/apps/SA-ITOA/default/savedsearches.conf
Search to group events ###
[itsi_event_grouping]
cron_schedule = * * * * *
disabled = 0
dispatch.earliest_time = rt
dispatch.indexedRealtime = 1
dispatch.latest_time = rt
enableSched = 1
search = itsi_event_management_index | where isnull(itsi_is_edited) | spath | fields - _raw | itsirulesengine | where 1=2

[Jarohnimo]

Hi rPhillips,

Where did you get that information? Seeing how when we search google/ splunk on ITSI Event Grouping there is literally NOTHING. I would like to just turn this off if I can't figure out what the benefit is?

This message is being fired off on the indexer (not the WFE) so Im not sure if i need to have this enabled. Please let me know

[Jarohnimo]

I have this exact same problem. Only occurs on the indexer, the search head seems to work fine. The indexer has alot of issues (it was the previous ITSI box) but now that i have a Dedicated sh it runs from there...

Dirty move, i simply deleted the services once i migrated the content to a new box. wasn't sure if that was the "proper" way to ridding the content but i got a feeling i need to turn something off on the indexer

[rphillips_splk]

The itsi_event_grouping search is a real-time search. Because it runs forever subsequent search instances that are spawned by the cron are skipped. There can only be one occurence of the real-time search running at any given time. This is expected and is not an issue because the search is actually running as you can see in resource_usage.log:

$SPLUNK_HOME/var/log/introspection
tail -f resource_usage.log | grep rt_

In 6.4.5+ and 6.5.1+ the skipped message 'reason' has been improved to clarify this and modified to: "The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached"

You will now see an event like this in scheduler.log when a real-time search is skipped:

11-09-2016 05:49:03.443 -0800 INFO SavedSplunker - savedsearch_id="nobody;search;search1", user="nobody", app="search", savedsearch_name="search1", status=skipped, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1478699340, window_time=0