topic Re: Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0? in Splunk Search

Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

shengcow — Fri, 24 Apr 2015 16:25:49 GMT

I have a little confusion about how time stamp actually works. I want to do a very simple query to combine the result for two searches into one single table. I am using a simple OR to achieve that. A simplified version of my query will be:

(index=firewall1 dst=8.8.8.8) OR (index=firewall2 starttime="03/23/2015:12:13:45" endtime="03/23/2015:13:13:55" dst=8.8.8.8)

I know this is a very inefficient search and I am supposed to use subsearches. However, I find that the Splunk server I am using does not have enough memory for the millions of results returned by a subsearch in this case.

The tricky part is that, the time modifier should only be applied to the second index. When running the search, I find the results for the first search are missing. I know it's missing because I ran the first search alone and there are a lot of results, yet, nothing shows up for the search after the OR clause. If I remove the time modifier in the second part, I get all the expected results.

What is creepy is that this does not happen for all the IPs, only for some of the IP addresses, I got all the results I want and for some I got nothing for the first search. The time modifier cannot be global because I can sometimes see results from the first search that does not fall in the time range. Same thing happened with earliest/latest keyword and I double checked I don't have any format issue with IPs.

The version of my Splunk is 5.0. Can anyone help me explain the logic behind that? Or help me come up with another solution without using any subsearches?

Re: Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

stephane_cyrill — Fri, 24 Apr 2015 16:50:56 GMT

Hi I'm not sur i understand you very well.but this is a way to do what you want without using OR.

1-
|multisearch [search index=firewall1
dst=8.8.8.8 ]
[search index=firewall2
starttime="03/23/2015:12
:13:45"
endtime="03/23/2015:13:
13:55" dst=8.8.8.8]

2-you can pipe all an continue your search with: table,chart,..........

Re: Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

shengcow — Fri, 24 Apr 2015 17:02:09 GMT

Thanks for the quick answer. As I metioned in my question, I can not use any subsearches because of the massive number results for both queries and the limited memory my Splunk server. And it will be nice if you can explain my observation first.

Re: Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

stephane_cyrill — Fri, 24 Apr 2015 17:27:52 GMT

You can also control the subsearch with
settings in limits.conf for the runtime and
maximum number of results returned:
[subsearch]
maxout =
Maximum number of results to return
from a subsearch.
This value cannot be greater than or
equal to 10500.
Defaults to 10000.
maxtime =
Maximum number of seconds to run a
subsearch before finalizing
Defaults to 60.
ttl =
Time to cache a given subsearch's
results, in seconds.
Do not set this below 120 seconds.
Defaults to 300.

docs.splunk.com/Documentation/Splunk/6.2.2/Search/Aboutsubsearches

Re: Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

shengcow — Fri, 24 Apr 2015 18:00:12 GMT

Let me explain more. Our team do not control the Splunk server and because of the limited memory, we are not allowed to change the subsearch configuration. It's a shared server and I should not consume all the resources. It will be great if we can focus on the problem itself. I am sorry for my current situation and limitations.

Re: Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

chimell — Mon, 27 Apr 2015 22:10:54 GMT

Hi shengcow
Use simply the following search code :

|set union[ search index=firewall1 dst=8.8.8.8 ][search index=firewall2 starttime="03/23/2015:12:13:45" endtime="03/23/2015:13:13:55" dst=8.8.8.8]

Re: Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

shengcow — Mon, 27 Apr 2015 22:16:17 GMT

set union/intersect will not return the result for sub-searches if they have millions of results(over the sub-search limitation), it will just return empty. And there is usually no notification on that. It is not working in my case here since there are not enough memory on Splunk server to increase the sub-search limitation.