topic Re: Same search returns different results each time it is run. in Splunk Search

Same search returns different results each time it is run?

mmarinov — Thu, 16 Mar 2023 15:21:39 GMT

I have this weird issue where the same exact search, run for a same exact period returns different number of events each time it is run.
Thus, rendering all attempts for accurate reporting obsolete.
It doesn't matter the type of search, for instance, if it has some statistics or it's just plain search - same searches return different results.
We've checked all the usual stuff - event sampling is turned off, indexing time is OK and it's not lagging, so no skewing of the results can come from this.
Searches are run directly against indexes, no data models are involved and search logs for the searches are identical for the runs compared to each other.
What we discovered for sure is, that this issue affects only indexes that are stored in an S3 Storage. Locally kept indexes are fine and do not have this issue.
The S3 storage was tested, it is configured correctly, there are no network disruptions, there are no errors in the logs concerning it, there's nothing that could hint a problem.
Yet, the problem remains.

Any idea what may be causing this?

Attaching a screenshot just for visualization, and here's the search for which it was made:

index="qualys" sourcetype="qualys:hostDetection" PATCHABLE="YES" NETBIOS="*"

Re: Same search returns different results each time it is run.

sindhi — Thu, 21 Apr 2022 23:39:19 GMT

Hi mmarinov,

I am facing same issue. Did you find anything to resolve it. Thanks in Advance

Re: Same search returns different results each time it is run.

PickleRick — Fri, 22 Apr 2022 11:30:56 GMT

First thing to check in such case would be to see the job inspect window for any differences. And see the job log for any warnings/errors.

Re: Same search returns different results each time it is run.

isoutamo — Fri, 22 Apr 2022 12:51:21 GMT

have you both S3 as a SmartStore in splunk or some other S3-storage? Is this on AWS S3 or some other S3 implementation e.g. in OnPrem?

r. Ismo

Re: Same search returns different results each time it is run.

mmarinov — Wed, 27 Apr 2022 08:34:22 GMT

Hi Sindhi,

No resolution as of now, unfortunately.

Re: Same search returns different results each time it is run.

mmarinov — Wed, 27 Apr 2022 08:37:01 GMT

S3 SmartStore is used and the Splunk machines are on AWS EC2 instances.

Re: Same search returns different results each time it is run.

mmarinov — Wed, 27 Apr 2022 08:39:08 GMT

As stated in the original post, the job inspector returns basically identical information for the runs.
The only thing that is different are the execution times, which obviously cannot be identical.

Re: Same search returns different results each time it is run.

isoutamo — Wed, 27 Apr 2022 10:12:01 GMT

Have you looked from MC how cacheing is working or are there continuously need to get those from S3 instead of use cached version?

Can you also give some specification what you environment is looking and which kind of queries there are running (is those limited e.g. last 7d or all time etc.)

r. Ismo

Re: Same search returns different results each time it is run.

mmarinov — Wed, 27 Apr 2022 11:24:29 GMT

The deployment is as follows:
1. Indexer cluster with 3 indexers

2. Cluster master node which is also a DMC

3. Search head with Enterprise Security

4. Deployment server

5. Heavy forwarder

6. Numerous UFs.
The deployment is on AWS EC2 instances.

The type of query run makes no difference as stated in the original post.
I've tested now with the one from the original post, and monitored the S3 cache, check the screenshot.

Re: Same search returns different results each time it is run.

PickleRick — Wed, 27 Apr 2022 11:56:12 GMT

Just to be on the safe side.

Does the number increase or is it "randomly fluctuating"?

Did you try limiting by _index_earliest and _index_latest and see if the number of results is constant?

Re: Same search returns different results each time it is run.

mmarinov — Wed, 27 Apr 2022 12:07:32 GMT

It fluctuates.
Limiting with _index_earliest and _index_latest has no effect, the number of events still fluctuates.

Re: Same search returns different results each time it is run.

Gabriel — Thu, 16 Mar 2023 09:23:37 GMT

I do realize this is an old post, however, I had the same issue of slight fluctuations in search results. During the course of examining this issue, I stumbled upon this yet unanswered question.

In my search query, I use the perc() function. Its documentation says the following:

The perc and upperperc functions give approximate values for the integer percentile requested. The approximation algorithm that is used, which is based on dynamic compression of a radix tree, provides a strict bound of the actual value for any percentile.

This means, sometimes it might be perc50.3, in the next run perc49.6, etc. In my case, this was the cause for the fluctuations I observed. Once I swapped it for a function like avg(), search results were steady.