Splunk IT Service Intelligence

ITSI Skipped Events Percentage and Agg Policies

sail4lot
Path Finder

Hi-

We have a correlation search that produces a couple of thousand events every 5 minutes. At the same time we are seeing the "Skipped Events Percentage" in the "Event Analytics Monitoring" dashboard go to 100%. In addition, I see the KV stores for itsi_notable_group_user and itsi_notable_group_system hit 50,000 (which I subsequently upped in itsi_notable_event_retention.conf to 150000).

For some reason episodes are not reliably getting generated.

Two questions -
1. how do we troubleshoot the skipped events percentage issue, which presumably is causing the lack of episodes (though I can't seem to find documentation discussing how this all works).
2. Should we change our correlation search to not include normal severity events? Currently, the normal severity events are be produced so that we can change episodes to "info" when a "normal" event comes in. Welcome recommendations on a better practice than this!

Thanks in advance.

Labels (1)
0 Karma

eduncan
Splunk Employee
Splunk Employee

So first thing to check is that the agg policies you have to group are not excluding events. Skipped events is relevant to grouped events only so when you see skipped events, are they events that are not being grouped by the agg policies.  You can test this by making sure your default policy is grouping by 'source' only.  Also can you tell me if you have tsix enabled?  There is a known issue with large number of NE's being created if you have that enabled.  Troubleshooting, turn off corr searches except for one, and one agg policy.  See if events get grouped and skipped searches reduce.  If so, then you need to analyze what your agg policy is doing that is excluding those events.  Basically it's the process of elimination and remembering skipped events means, skipped grouping, not skipped creating NE's.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...