Splunk Search
Highlighted

How can I optimize the performance of my search?

Path Finder

I am doing a simple search:

index=pqr host=xyz* NOT TYPE="*ABCDE*" | fields X, Y | timechart limit=0 span=10m count, avg(X) by Y

on a two week period.
The search has been running for more than two hours now and is still only at about 60%. Is there a way to make it run faster?

Also sometimes I get

'search has been remotely
cancelled or expired'

What does that mean? how do I prevent a search from getting cancelled?

0 Karma
Highlighted

Re: How can I optimize the performance of my search?

Path Finder

According to this link: http://docs.splunk.com/Documentation/Splunk/6.3.0/Search/Writebettersearches
you should avoid using NOT wherever possible

Highlighted

Re: How can I optimize the performance of my search?

SplunkTrust
SplunkTrust

Also, do not use * searches. Since Splunk will have to read in your example, all events containing host fields from disk and compare if the values starts with xyz.

Take a good lock at the slide here http://conf.splunk.com/session/2015/conf2015_JHarty_DuncanTurnbull_Splunk_UsingSplunkSearchLanguage_... to learn more about Search Efficiency Optimisation.

cheers, MuS

Highlighted

Re: How can I optimize the performance of my search?

SplunkTrust
SplunkTrust

Give this a try

index=pqr host=xyz* TYPE!="*ABCDE*" | table _time, X, Y | timechart limit=0 span=10m count, avg(X) by Y

Also, if the data span is to a limited sourcetypes, please add them as well as filter.

Highlighted

Re: How can I optimize the performance of my search?

Path Finder

when i tried above, its running but all fields are staying blank. So i replaced 'table _time' with 'fields' and it's working better.
Is there anyway I can measure/compare the performance of the two search queries in terms of total approx. run time?

0 Karma
Highlighted

Re: How can I optimize the performance of my search?

SplunkTrust
SplunkTrust

YOu can check the job inspector (there is a job menu just below the search bar towards the right side) which shows the amount of time taken by the search.

0 Karma
Highlighted

Re: How can I optimize the performance of my search?

SplunkTrust
SplunkTrust

This looks like a good case for a summary index. Populate the index every 10 minutes with your stats, and then you search the summaries when you want to build a trend.

0 Karma
Highlighted

Re: How can I optimize the performance of my search?

Path Finder

hi muebel, could you elaborate some. i'm pretty new to splunk .

0 Karma
Highlighted

Re: How can I optimize the performance of my search?

SplunkTrust
SplunkTrust

Some reading about summary indexing available here.
http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/Usesummaryindexing

0 Karma
Highlighted

Re: How can I optimize the performance of my search?

Esteemed Legend

To answer your question directly, you can prevent it from happening by EITHER speeding up your search OR clicking on the Jobs menu click Send Job to Background and your job will NOT timeout and you can opt to receive an email when the job completes.