topic Re: How to improve the performance of our Splunk search query? in Splunk Search

How to improve the performance of our Splunk search query?

dhavamanis — Mon, 28 Sep 2020 18:07:49 GMT

We have indexed access logs into index="mpsapp", When we do a stats search or filter any records for these data for a particular month, it's extremely slow (took more than 1.5 hours for first query). Can you please tell us how to optimize this query?

Sample Splunk Queries :

index="mpsapp" | stats count by response_code (filtered time range only for October month, we have extracted the filed response_code in search node)
source="mps" date_year=2014 date_month=october response_code!=200
date_year=2014 date_month=november site="fandango-web" path="%2F"

Total Events for the month of October is : 355,925,951 events (10/1/14 12:00:00.000 AM to 11/1/14 12:00:00.000 AM)
Splunk Version : 6.1.4

Is there any configuration level optimization required to speed-up the query response? Please share your suggestions.

Re: How to improve the performance of our Splunk search query?

aljohnson_splun — Thu, 13 Nov 2014 19:24:35 GMT

Do you ever use the fields command to only retrieve the relevant fields ?

Re: How to improve the performance of our Splunk search query?

dhavamanis — Thu, 13 Nov 2014 19:27:11 GMT

we haven't tried any fields command. Can you please give me some sample for fields command.

Re: How to improve the performance of our Splunk search query?

MuS — Thu, 13 Nov 2014 21:04:02 GMT

Hi dhavamanis,

try to use index and any of the metadata http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/Metadata fields ( host, source or sourcetype) in your searches. Try to filter the fields as tight as possible in your base search or use fields http://docs.splunk.com/Documentation/Splunk/6.2.0/SearchReference/Fields#Examples as @aljohnson_splunk mentioned.

Do not use NOT searches, rather search for events that you want and need.

Also, take a look at this great Answer of @jrodman about How do optimizations for field-based searches work http://answers.splunk.com/answers/172275/how-do-optimizations-for-field-based-searches-work.html#answer-172230

hope this helps to understand the possibilities to speed up your searches...

cheers, MuS

Re: How to improve the performance of our Splunk search query?

aljohnson_splun — Thu, 13 Nov 2014 21:08:04 GMT

 sourcetype=access_combined

Job Inspector: This search has completed and has returned 1,000 results by scanning 177,830 events in 14.393 seconds.

sourcetype=access_combined | fields clientip bytes action

Job Inpector: This search has completed and has returned 1,000 results by scanning 177,835 events in 3.508 seconds.

It's similar to using fast mode. Are you using fast mode?
Set search mode to adjust your search experience

Performance recommendations
How search types affect performance

Re: How to improve the performance of our Splunk search query?

dhavamanis — Wed, 10 Dec 2014 21:11:42 GMT

Thanks!. We have accelerated the report and added to Dashboard. Using the summary index and its fetching results very fast.

Re: How to improve the performance of our Splunk search query?

puneethgowda — Wed, 26 Apr 2017 10:16:11 GMT

We have done the following things after doing R & D.

1.Changed date range from real time to today.
2.Set dashboard refresh time to every 5 minutes.
3.Summary indexing
4.Report acceleration
5.Scheduled this search every 5 minutes so it will save in the cache.
6.Search query optimization.
7.Auto restart splunk daily at 2:00 AM UTC so that memory will be released.
8.Set high priority to this dashboard.
7.Set high priority to this scheduled search.
8.Run stats tables first then start charts.
9.Changed the delimer of raw data from text files method to new way which will reduce the time while converting raw data to fields of delimiting proccess.
10.Reduce the number of indexes and source type

After all this my dashboards loading time reduced from 3 minutes to less than 10 seconds.

Super fast

Re: How to improve the performance of our Splunk search query?

preactivity — Sat, 31 Aug 2019 13:24:48 GMT

You can improve the performance by 10 X times by using Splunk meta data fields. I can help you in that please contact me in fiverr or Email (hurdlej1@gmail.com)

https://www.fiverr.com/s2/affc9b7a8a
https://www.fiverr.com/s2/608e8ed73f?utm_source=CopyLink_Mobile