Splunk Search

Why does my simple search return a variable number of events with each run?

MonkeyK
Builder

I have a simple search against my firewall logs. the search looks like
index=firewall session_id=1234 src_ip=10.10.0.10 dest_ip=200.100.200.100 dest_port=22
While the numbers are not correct, I do know that on 1/2/18 my firewall has 3 events for the parameters named:
-a start event
-an end event
-and a threat event

I have been trying to run this search with a time window of YTD, but get back 0-3 events with each run

the search summary notes 3 events as:
3 events (1/1/18 12:00:00.000 AM to 1/11/18 2:31:10.000 PM)
and the events tab shows
Events (3)

But my results have varying numbers of events in them: any number of events between 0-3

What is going on here? and how do I go about getting it fixed?

Tags (2)
0 Karma
1 Solution

micahkemp
Champion

Consider this previous answers post to see if it sounds like what you're seeing. There is a known issue that is resolved in 6.6.4 regarding keepalives between the search head and indexers.

Look in your serach.log for the searches that are failing and look for line(s) like:

Timeliner - Ignored 2 events because they were after the commit time (0)

View solution in original post

0 Karma

micahkemp
Champion

Consider this previous answers post to see if it sounds like what you're seeing. There is a known issue that is resolved in 6.6.4 regarding keepalives between the search head and indexers.

Look in your serach.log for the searches that are failing and look for line(s) like:

Timeliner - Ignored 2 events because they were after the commit time (0)
0 Karma

MonkeyK
Builder

that explains it. I found the Timeliner line in my search too.
And the linked links explains that that the temporary fix is to add "|sort _time" to the query

0 Karma

micahkemp
Champion

Are you seeing those messages in search.log?

0 Karma

MonkeyK
Builder

I do see the messages in the simple query.

I only found the issue because of a problem in a query that had a subsearch. In the case of the query with a subsearch, the timeliner messages do not show up.

To be honest, I am not sure how I will do "|sort _time" in my subsearch: I do not want _time to be a search criteria.

0 Karma

micahkemp
Champion

| sort _time isn't really a solution. It's more of a method of showing that the issue is present and helps explain why.

The real solution is to upgrade to 6.6.4 as soon as you can. This isn't the only issue resolved in 6.6.4 that I've run into.

0 Karma

MonkeyK
Builder

I see. thank you for helping me to understand the situation.

0 Karma

micahkemp
Champion

Does this search take a long time to run?

0 Karma

MonkeyK
Builder

On the order of 40-50 seconds

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...