Splunk IT Service Intelligence

ITSI Skipped Events Percentage and Agg Policies

Path Finder


We have a correlation search that produces a couple of thousand events every 5 minutes. At the same time we are seeing the "Skipped Events Percentage" in the "Event Analytics Monitoring" dashboard go to 100%. In addition, I see the KV stores for itsi_notable_group_user and itsi_notable_group_system hit 50,000 (which I subsequently upped in itsi_notable_event_retention.conf to 150000).

For some reason episodes are not reliably getting generated.

Two questions -
1. how do we troubleshoot the skipped events percentage issue, which presumably is causing the lack of episodes (though I can't seem to find documentation discussing how this all works).
2. Should we change our correlation search to not include normal severity events? Currently, the normal severity events are be produced so that we can change episodes to "info" when a "normal" event comes in. Welcome recommendations on a better practice than this!

Thanks in advance.

Labels (1)
0 Karma

Splunk Employee
Splunk Employee

So first thing to check is that the agg policies you have to group are not excluding events. Skipped events is relevant to grouped events only so when you see skipped events, are they events that are not being grouped by the agg policies.  You can test this by making sure your default policy is grouping by 'source' only.  Also can you tell me if you have tsix enabled?  There is a known issue with large number of NE's being created if you have that enabled.  Troubleshooting, turn off corr searches except for one, and one agg policy.  See if events get grouped and skipped searches reduce.  If so, then you need to analyze what your agg policy is doing that is excluding those events.  Basically it's the process of elimination and remembering skipped events means, skipped grouping, not skipped creating NE's.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...