Splunk ITSI

I see the same notable event grouped several time, on different Episodes in ITSI

yannK
Splunk Employee
Splunk Employee

I have a correlation search creating notable events.
In the index=itsi_tracked_alerts, I see one event for a given event_id.

But on the Episode review, I see the event being member of several Episodes
index=itsi_grouped_alerts , comparing event_id and itsi_group_id

This is happening randomly.

I see the dashboard on the ITSI healthcheck, that show me the multiple grouping.
What can cause that?

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

Double grouping of notable events is a known bug, linked to splunkcore bugs on older versions.

  • Double rules engine java process : SPL-155648
    You can see that on the process list you have several java process with rules_engine in the arguments.
    Read this https://docs.splunk.com/Documentation/ITSI/4.3.0/ReleaseNotes/Knownissues
    to fix: you need to be on a valid version of splunk (not 7.2.0 to 7.2.3) and upgrade to ITSI 4.3.1+ or add the workaround manually.

  • SHCluster, SPL-169046 multiple search jobs for indextime time realtime scheduled search
    to confirm, look in the job inspector, how many "itsi_event_grouping" searches are running, if you see it on more than one search-head, at a time, it can be this bug
    Look for SPL-169046, fixed since splunk core 7.2.8
    see https://docs.splunk.com/Documentation/Splunk/7.2.8/ReleaseNotes/Fixedissues

View solution in original post

0 Karma

eduncan
Splunk Employee
Splunk Employee

This may not be a bug.  Remember that NE's can make it into multiple episodes on purpose.  If a NE is related to more than one agg policy, it will be grouped with that policy as well.  Make sure that is not the case before thinking it is a bug.

0 Karma

yannK
Splunk Employee
Splunk Employee

Double grouping of notable events is a known bug, linked to splunkcore bugs on older versions.

  • Double rules engine java process : SPL-155648
    You can see that on the process list you have several java process with rules_engine in the arguments.
    Read this https://docs.splunk.com/Documentation/ITSI/4.3.0/ReleaseNotes/Knownissues
    to fix: you need to be on a valid version of splunk (not 7.2.0 to 7.2.3) and upgrade to ITSI 4.3.1+ or add the workaround manually.

  • SHCluster, SPL-169046 multiple search jobs for indextime time realtime scheduled search
    to confirm, look in the job inspector, how many "itsi_event_grouping" searches are running, if you see it on more than one search-head, at a time, it can be this bug
    Look for SPL-169046, fixed since splunk core 7.2.8
    see https://docs.splunk.com/Documentation/Splunk/7.2.8/ReleaseNotes/Fixedissues

0 Karma

yannK
Splunk Employee
Splunk Employee

Addendum : Keep in mind that other issues can lead to multiple grouping of the same notables

- Rules engine backfill issues -> upgrade to recent versions of ITSI (4.4.4 or more)

- "Tsidx reduction" core bug on the indexers, causing old notables to be rediscovered over and over by the realtime searches. -> see workaround ITSI-4606 https://docs.splunk.com/Documentation/ITSI/4.4.0/ReleaseNotes/Knownissues

>Workaround:
This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes.

0 Karma

yannK
Splunk Employee
Splunk Employee
 
0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...