topic Re: Search query taking ages to finish in Splunk 7.0.1 in Splunk Search

Search query taking ages to finish in Splunk 7.0.1

AKG1_old1 — Wed, 17 Jan 2018 13:29:29 GMT

Hello,

I have upgraded Splunk Enterprise to 7.0.1. One of the search query is taking ages to finish it. Same query finished quickly in Splunk 6.x.

Splunk 6.6.1 = 5 secs
Splunk 7.0.1 = 26 mins (still running)

Does anyone have encounter such situation or have idea for this behaviour in Splunk 7.

Re: Search query taking ages to finish in Splunk 7.0.1

mayurr98 — Wed, 17 Jan 2018 13:45:47 GMT

In the first search its written before 17/1/2018 from how long do you have data in your system?

Re: Search query taking ages to finish in Splunk 7.0.1

AKG1_old1 — Wed, 17 Jan 2018 13:58:47 GMT

file is created on 17/01/2018 06:27:12. So its having data for few hours.

same file is taking by both Splunk version. I am having feeling that as time is specified as "All Time" , Splunk 7 is not specifically looking for specified file. It's trying to find out data from the begnning of Splunk Time (1st Jan 1970).

Search is still running.

Re: Search query taking ages to finish in Splunk 7.0.1

elliotproebstel — Wed, 17 Jan 2018 14:20:34 GMT

That's correct. There are a small handful of fields that are extracted at index-time, and unless you've done some intentional work to change that on your system, then the field service_name will not be one of them. So now Splunk is searching back through all indexed data looking for any possible matches. Your best bets to make this search run more efficiently are to narrow the time window and specify the index(es) in which you'd like to search. Here's a good guide to provide more of an overview:
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Search/Writebettersearches#Tips_for_tuning_your_searches

Re: Search query taking ages to finish in Splunk 7.0.1

sarfarajsayyad — Mon, 05 Mar 2018 06:19:08 GMT

Same behavior I see on my setup. My query was running fine on 6.5.2 (within 2 mins) and now it is taking more time (90 mins) on 7.0.2. Is there any change happened on 7.0.x? I don't see any error msg on splunkd log too. There is no clue.

Re: Search query taking ages to finish in Splunk 7.0.1

niketn — Mon, 05 Mar 2018 07:00:45 GMT

@agoyal, you second search should be part of your base search

<yourBaseSearch> "*concurrent mode failure*"
| stats count as FAIL

Have you tried it this way?

Re: Search query taking ages to finish in Splunk 7.0.1

niketn — Mon, 05 Mar 2018 07:01:19 GMT

@sarfarajsayyad , would it be possible for you to share the query?

Re: Search query taking ages to finish in Splunk 7.0.1

sarfarajsayyad — Mon, 05 Mar 2018 08:35:51 GMT

That query is having some customer specific info. Cant share here.
My point is, the same query is running fine on 6.5.2 and not on 7.0.2. Believe me, I have installed 7.0.2 Splunk enterprise on a new machine without data. Still, it's taking more time. Looks like its nothing related to data. Something is changed in 7.0.x.

JFYI - In my query, I have 20+ joined on various indexes/lookup.

Re: Search query taking ages to finish in Splunk 7.0.1

sarfarajsayyad — Mon, 05 Mar 2018 09:31:54 GMT

I see lot of below messages

02-22-2018 04:06:41.512 INFO SearchPipeline - Command='eval' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='inputlookup' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='inputlookup' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='eval' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='search' doesnt have raw field
02-22-2018 04:06:41.512 INFO SortOperator - maxmem = 209715200

Re: Search query taking ages to finish in Splunk 7.0.1

sarfarajsayyad — Mon, 05 Mar 2018 10:20:24 GMT

If I use "|noop search_optimization=false" at the end of my query its giving me result very fast. After cross-checking I got the same result.

Is there any impact of "|noop search_optimization=false" ?

Re: Search query taking ages to finish in Splunk 7.0.1

niketn — Mon, 05 Mar 2018 10:28:20 GMT

@sarfarajsayyad if you have valid Splunk Entitlement, reach out to Splunk Support.

Re: Search query taking ages to finish in Splunk 7.0.1

amaind1990 — Fri, 11 May 2018 09:42:29 GMT

@sarfarajsayyad @agoyal I am also getting same issue. Were you able to resolve the issue? If yes, what is the solution? Is this issue is specific to new version 7.1.0?

Re: Search query taking ages to finish in Splunk 7.0.1

sarfarajsayyad — Tue, 22 May 2018 05:25:52 GMT

In my case, I have added "| noop search_optimization=false" at the last line which makes my query fast. You can give a try.

Re: Search query taking ages to finish in Splunk 7.0.1

sarfarajsayyad — Tue, 22 May 2018 06:08:57 GMT

In my case I have added "| noop search_optimization=false" at the end of the line to resolve the issue. It worked for me, you can give a try.