I see the same notable event grouped several time, on different Episodes in ITSI

[yannK]

I have a correlation search creating notable events.
In the index=itsi_tracked_alerts, I see one event for a given event_id.

But on the Episode review, I see the event being member of several Episodes
index=itsi_grouped_alerts , comparing event_id and itsi_group_id

This is happening randomly.

I see the dashboard on the ITSI healthcheck, that show me the multiple grouping.
What can cause that?

[eduncan]

This may not be a bug.  Remember that NE's can make it into multiple episodes on purpose.  If a NE is related to more than one agg policy, it will be grouped with that policy as well.  Make sure that is not the case before thinking it is a bug.

[yannK · Accepted solution]

Double grouping of notable events is a known bug, linked to splunkcore bugs on older versions.

• Double rules engine java process : SPL-155648
  You can see that on the process list you have several java process with rules_engine in the arguments.
  Read this https://docs.splunk.com/Documentation/ITSI/4.3.0/ReleaseNotes/Knownissues
  to fix: you need to be on a valid version of splunk (not 7.2.0 to 7.2.3) and upgrade to ITSI 4.3.1+ or add the workaround manually.

• SHCluster, SPL-169046 multiple search jobs for indextime time realtime scheduled search
  to confirm, look in the job inspector, how many "itsi_event_grouping" searches are running, if you see it on more than one search-head, at a time, it can be this bug
  Look for SPL-169046, fixed since splunk core 7.2.8
  see https://docs.splunk.com/Documentation/Splunk/7.2.8/ReleaseNotes/Fixedissues

[yannK]

Addendum : Keep in mind that other issues can lead to multiple grouping of the same notables

- Rules engine backfill issues -> upgrade to recent versions of ITSI (4.4.4 or more)

- "Tsidx reduction" core bug on the indexers, causing old notables to be rediscovered over and over by the realtime searches. -> see workaround ITSI-4606 https://docs.splunk.com/Documentation/ITSI/4.4.0/ReleaseNotes/Knownissues

>Workaround:
This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes.