topic Re: Why does my simple search return a variable number of events with each run? in Splunk Search

Why does my simple search return a variable number of events with each run?

MonkeyK — Tue, 29 Sep 2020 17:37:46 GMT

I have a simple search against my firewall logs. the search looks like
index=firewall session_id=1234 src_ip=10.10.0.10 dest_ip=200.100.200.100 dest_port=22
While the numbers are not correct, I do know that on 1/2/18 my firewall has 3 events for the parameters named:
-a start event
-an end event
-and a threat event

I have been trying to run this search with a time window of YTD, but get back 0-3 events with each run

the search summary notes 3 events as:
3 events (1/1/18 12:00:00.000 AM to 1/11/18 2:31:10.000 PM)
and the events tab shows
Events (3)

But my results have varying numbers of events in them: any number of events between 0-3

What is going on here? and how do I go about getting it fixed?

Re: Why does my simple search return a variable number of events with each run?

micahkemp — Thu, 11 Jan 2018 21:09:44 GMT

Does this search take a long time to run?

Re: Why does my simple search return a variable number of events with each run?

micahkemp — Thu, 11 Jan 2018 21:12:15 GMT

Consider this previous answers post to see if it sounds like what you're seeing. There is a known issue that is resolved in 6.6.4 regarding keepalives between the search head and indexers.

Look in your serach.log for the searches that are failing and look for line(s) like:

Timeliner - Ignored 2 events because they were after the commit time (0)

Re: Why does my simple search return a variable number of events with each run?

MonkeyK — Thu, 11 Jan 2018 21:17:44 GMT

On the order of 40-50 seconds

Re: Why does my simple search return a variable number of events with each run?

MonkeyK — Thu, 11 Jan 2018 21:45:18 GMT

that explains it. I found the Timeliner line in my search too.
And the linked links explains that that the temporary fix is to add "|sort _time" to the query

Re: Why does my simple search return a variable number of events with each run?

micahkemp — Thu, 11 Jan 2018 21:45:54 GMT

Are you seeing those messages in search.log?

Re: Why does my simple search return a variable number of events with each run?

MonkeyK — Thu, 11 Jan 2018 22:21:17 GMT

I do see the messages in the simple query.

I only found the issue because of a problem in a query that had a subsearch. In the case of the query with a subsearch, the timeliner messages do not show up.

To be honest, I am not sure how I will do "|sort _time" in my subsearch: I do not want _time to be a search criteria.

Re: Why does my simple search return a variable number of events with each run?

micahkemp — Thu, 11 Jan 2018 22:27:21 GMT

| sort _time isn't really a solution. It's more of a method of showing that the issue is present and helps explain why.

The real solution is to upgrade to 6.6.4 as soon as you can. This isn't the only issue resolved in 6.6.4 that I've run into.

Re: Why does my simple search return a variable number of events with each run?

MonkeyK — Thu, 11 Jan 2018 23:45:41 GMT

I see. thank you for helping me to understand the situation.