topic Why does Splunk search result count diminishes during search? in Splunk Search

Why does Splunk search result count diminishes during search?

JLTsx — Wed, 19 Apr 2023 21:31:03 GMT

Hey,

When running a query the results found are diminishing over time. Pagination is not of incluence ( tried 10, 50, 100 ) and it seems to be somewhere in the index this behavior is triggered. Meaning, if i search all events for today the count goes up to a count of events which then stalls for about 10 seconds to then continue with the result count to then start diminishing.

When setting the time to a few hours back this behavior also happens. Since the search takes quite long to hit this "point of return" i assume the two time frames overlap and the same events are causing this diminishing of results to happen.

Suggesting there is something wrong in the index ?

Br,

JLT

Re: Splunk search result count diminishes during search

ITWhisperer — Tue, 18 Apr 2023 17:27:30 GMT

Please share the search causing the issue

Re: Splunk search result count diminishes during search

woodcock — Tue, 18 Apr 2023 17:35:51 GMT

You didn't even give us the search SPL. I am sure that the problem is you are hitting limits but there is now way to help because you told us nothing useful.

Re: Why does Splunk search result count diminishes during search?

JLTsx — Mon, 24 Apr 2023 14:59:26 GMT

Thanks for sharing. I'm not in the habit of sharing queries. This one in particular was written by a 3rd party so I'm not sure i can legally share it in whole.

The mention of limits led me to test with 8 hour windows instead of 24 hours and this did result in queries which no longer failed to return the results without removing results.

The query looks like this

index=ind | where (cidrmatch(....,src ) OR cidrmatch(...,src)) OR (cidrmatch(...,dest) OR cidrmatcht(...,dest)) AND src!="<IP>" | stats dc(dest) as dcounter, values(dest) as dip by src,dest_port
| where dcounter >= 500
| eval nowtime = strftime(now(),"%d/%b/%Y:%H:%M:%S")
| eval tip = mvindex(dip,0)
| eval alerting = "scan" + dcounter + "port" + dest_port
| table nowtime dip tip alerting

While writing alternates to this query it was found using long "| stats" does not matter too much but adding some keys such as user when use "... as <name> by <key1>,<key2>,<key>" does cause the same issue

I've encountered this a few years ago but was not allowed to keep the notes on how to fix

Re: Why does Splunk search result count diminishes during search?

ITWhisperer — Mon, 24 Apr 2023 15:07:26 GMT

You could run your search over shorter periods of time and save the results to a summary index. Then you can search the summary index over a number of sets of results and combine them as if they were run over the original index for the whole time.

Re: Why does Splunk search result count diminishes during search?

JLTsx — Mon, 24 Apr 2023 16:45:15 GMT

Thanks, i fixed it by not using eval, just stats and a where clauses, no limits are hit now. The query also appears to run (much?) faster.

Re: Why does Splunk search result count diminishes during search?

JLTsx — Wed, 20 Sep 2023 15:09:02 GMT

this issue was worked around by rewriting the query