Splunk Search

Same search returns different results each time it is run.

mmarinov
Explorer

I have this weird issue where the same exact search, run for a same exact period returns different number of events each time it is run.
Thus, rendering all attempts for accurate reporting obsolete.
It doesn't matter the type of search, for instance, if it has some statistics or it's just plain search - same searches return different results.
We've checked all the usual stuff - event sampling is turned off, indexing time is OK and it's not lagging, so no skewing of the results can come from this.
Searches are run directly against indexes, no data models are involved and search logs for the searches are identical for the runs compared to each other.
What we discovered for sure is, that this issue affects only indexes that are stored in an S3 Storage. Locally kept indexes are fine and do not have this issue.
The S3 storage was tested, it is configured correctly, there are no network disruptions, there are no errors in the logs concerning it, there's nothing that could hint a problem.
Yet, the problem remains.

Any idea what may be causing this?

Attaching a screenshot just for visualization, and here's the search for which it was made:

 

index="qualys" sourcetype="qualys:hostDetection" PATCHABLE="YES" NETBIOS="*"​

 

 

Labels (1)
Tags (2)
0 Karma

sindhi
Loves-to-Learn Lots

Hi mmarinov,

I am facing same issue. Did you find anything to resolve it. Thanks in Advance

0 Karma

mmarinov
Explorer

Hi Sindhi,

No resolution as of now, unfortunately.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

have you both S3 as a SmartStore in splunk or some other S3-storage? Is this on AWS S3 or some other S3 implementation e.g. in OnPrem?

r. Ismo

0 Karma

mmarinov
Explorer

S3 SmartStore is used and the Splunk machines are on AWS EC2 instances.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Have you looked from MC how cacheing is working or are there continuously need to get those from S3 instead of use cached version?

Can you also give some specification what you environment is looking and which kind of queries there are running (is those limited e.g. last 7d or all time etc.)

r. Ismo

0 Karma

mmarinov
Explorer

The deployment is as follows:
1. Indexer cluster with 3 indexers

2. Cluster master node which is also a DMC

3. Search head with Enterprise Security

4. Deployment server

5. Heavy forwarder

6. Numerous UFs.
The deployment is on AWS EC2 instances.

The type of query run makes no difference as stated in the original post.
I've tested now with the one from the original post, and monitored the S3 cache, check the screenshot.
 

0 Karma

PickleRick
Ultra Champion

First thing to check in such case would be to see the job inspect window for any differences. And see the job log for any warnings/errors.

0 Karma

mmarinov
Explorer

As stated in the original post, the job inspector returns basically identical information for the runs.
The only thing that is different are the execution times, which obviously cannot be identical. 

0 Karma

PickleRick
Ultra Champion

Just to be on the safe side.

Does the number increase or is it "randomly fluctuating"?

Did you try limiting by _index_earliest and _index_latest and see if the number of results is constant?

 

0 Karma

mmarinov
Explorer

It fluctuates.
Limiting with _index_earliest and _index_latest has no effect, the number of events still fluctuates.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>