Splunk Search
Highlighted

Search query taking ages to finish in Splunk 7.0.1

Builder

Hello,

I have upgraded Splunk Enterprise to 7.0.1. One of the search query is taking ages to finish it. Same query finished quickly in Splunk 6.x.

Splunk 6.6.1 = 5 secs
Splunk 7.0.1 = 26 mins (still running)

Does anyone have encounter such situation or have idea for this behaviour in Splunk 7.

alt text

Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

SplunkTrust
SplunkTrust

In the first search its written before 17/1/2018 from how long do you have data in your system?

0 Karma
Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

Builder

file is created on 17/01/2018 06:27:12. So its having data for few hours.

same file is taking by both Splunk version. I am having feeling that as time is specified as "All Time" , Splunk 7 is not specifically looking for specified file. It's trying to find out data from the begnning of Splunk Time (1st Jan 1970).

Search is still running.

Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

That's correct. There are a small handful of fields that are extracted at index-time, and unless you've done some intentional work to change that on your system, then the field service_name will not be one of them. So now Splunk is searching back through all indexed data looking for any possible matches. Your best bets to make this search run more efficiently are to narrow the time window and specify the index(es) in which you'd like to search. Here's a good guide to provide more of an overview:
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Search/Writebettersearches#Tips_for_tuning_yo...

0 Karma
Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

New Member

Same behavior I see on my setup. My query was running fine on 6.5.2 (within 2 mins) and now it is taking more time (90 mins) on 7.0.2. Is there any change happened on 7.0.x? I don't see any error msg on splunkd log too. There is no clue.

0 Karma
Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

Legend

@sarfarajsayyad , would it be possible for you to share the query?




| eval message="Happy Splunking!!!"


0 Karma
Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

New Member

That query is having some customer specific info. Cant share here.
My point is, the same query is running fine on 6.5.2 and not on 7.0.2. Believe me, I have installed 7.0.2 Splunk enterprise on a new machine without data. Still, it's taking more time. Looks like its nothing related to data. Something is changed in 7.0.x.

JFYI - In my query, I have 20+ joined on various indexes/lookup.

0 Karma
Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

New Member

I see lot of below messages

02-22-2018 04:06:41.512 INFO SearchPipeline - Command='eval' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='inputlookup' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='inputlookup' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='eval' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='search' doesnt have raw field
02-22-2018 04:06:41.512 INFO SortOperator - maxmem = 209715200

0 Karma
Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

New Member

If I use "|noop search_optimization=false" at the end of my query its giving me result very fast. After cross-checking I got the same result.

Is there any impact of "|noop search_optimization=false" ?

0 Karma
Highlighted

Re: Search query taking ages to finish in Splunk 7.0.1

Legend

@sarfarajsayyad if you have valid Splunk Entitlement, reach out to Splunk Support.




| eval message="Happy Splunking!!!"


0 Karma