Splunk IT Service Intelligence

I see the same notable event grouped several time, on different Episodes in ITSI

yannK
Splunk Employee
Splunk Employee

I have a correlation search creating notable events.
In the index=itsi_tracked_alerts, I see one event for a given event_id.

But on the Episode review, I see the event being member of several Episodes
index=itsi_grouped_alerts , comparing event_id and itsi_group_id

This is happening randomly.

I see the dashboard on the ITSI healthcheck, that show me the multiple grouping.
What can cause that?

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

Double grouping of notable events is a known bug, linked to splunkcore bugs on older versions.

  • Double rules engine java process : SPL-155648
    You can see that on the process list you have several java process with rules_engine in the arguments.
    Read this https://docs.splunk.com/Documentation/ITSI/4.3.0/ReleaseNotes/Knownissues
    to fix: you need to be on a valid version of splunk (not 7.2.0 to 7.2.3) and upgrade to ITSI 4.3.1+ or add the workaround manually.

  • SHCluster, SPL-169046 multiple search jobs for indextime time realtime scheduled search
    to confirm, look in the job inspector, how many "itsi_event_grouping" searches are running, if you see it on more than one search-head, at a time, it can be this bug
    Look for SPL-169046, fixed since splunk core 7.2.8
    see https://docs.splunk.com/Documentation/Splunk/7.2.8/ReleaseNotes/Fixedissues

View solution in original post

0 Karma

eduncan
Splunk Employee
Splunk Employee

This may not be a bug.  Remember that NE's can make it into multiple episodes on purpose.  If a NE is related to more than one agg policy, it will be grouped with that policy as well.  Make sure that is not the case before thinking it is a bug.

0 Karma

yannK
Splunk Employee
Splunk Employee

Double grouping of notable events is a known bug, linked to splunkcore bugs on older versions.

  • Double rules engine java process : SPL-155648
    You can see that on the process list you have several java process with rules_engine in the arguments.
    Read this https://docs.splunk.com/Documentation/ITSI/4.3.0/ReleaseNotes/Knownissues
    to fix: you need to be on a valid version of splunk (not 7.2.0 to 7.2.3) and upgrade to ITSI 4.3.1+ or add the workaround manually.

  • SHCluster, SPL-169046 multiple search jobs for indextime time realtime scheduled search
    to confirm, look in the job inspector, how many "itsi_event_grouping" searches are running, if you see it on more than one search-head, at a time, it can be this bug
    Look for SPL-169046, fixed since splunk core 7.2.8
    see https://docs.splunk.com/Documentation/Splunk/7.2.8/ReleaseNotes/Fixedissues

View solution in original post

0 Karma

yannK
Splunk Employee
Splunk Employee

Addendum : Keep in mind that other issues can lead to multiple grouping of the same notables

- Rules engine backfill issues -> upgrade to recent versions of ITSI (4.4.4 or more)

- "Tsidx reduction" core bug on the indexers, causing old notables to be rediscovered over and over by the realtime searches. -> see workaround ITSI-4606 https://docs.splunk.com/Documentation/ITSI/4.4.0/ReleaseNotes/Knownissues

>Workaround:
This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes.

0 Karma

yannK
Splunk Employee
Splunk Employee
 
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!