Splunk Enterprise Security: Is there a workaround for duplicate events being reported by ES Search Head?

[mipeters_splunk]

We have Splunk Enterprise Security (ES) Search Head (SH) which is reporting duplicate events even though those events are only on our Indexer (IDX) cluster once.

We think that the issue is due to a mismatch of Splunk versions. ie the ES SH is on 6.4 and the IDX cluster is on 6.2.

When we run the identical search from our other SH clusters the results are fine, obviously the other SH clusters are on the same version of Splunk as our IDX cluster.

We will be upgrading the IDX cluster in due course, but would like to know:

is there a temporary workaround that could fix this without having to downgrade our ES SH ?

Our SOC already divide any statistical search results by two... not a great work around.

A google of answers or docs did not come up with anything useful.

[mipeters_splunk · Accepted solution]

The answer in the end is to upgrade.

With the version mismatch between the SH and the IDXc makes the SH return duplicate events.

We were not able to find a work around.

---

An upgrade from 6.2 on the IDXc to 6.5.x and the ES SH to 6.5 fixed the issue.

[mipeters_splunk]

@adonio the ES search head is stand alone. We have three other Search heads these are not experiencing the issue. Hope that explains the architecture a little better.

Also unfortunately we have all notables turned off at the moment as we use ES in a "special way".

[adonio]

when you say "duplicate events" what exactly do you mean? seams like you checked _raw and there are not duplicate events on the indexer tier which leads me to think that you mean "alerts" or "correlated searches results" (no notables as you mentioned in comment). can you share a search and a duplicate event?

[adonio]

you mentioned SH clusters, do you have couple of clusters or 1 SH cluster?
maybe you have multiple results cause the SH aren't in sync and the search is executed twice?
can you elaborate on the architecture a little bit?
can you search the notable index and see how many results are for any correlation search?
hope it helps